Improving Bilinear RNN with Closed-loop Control
Authors: Jiaxi Hu
Yongqi Pan
Jusen Du
Disen Lan
Xiaqiang Tang
Qingsong Wen
Yuxuan Liang
Weigao Sun
Venue: NeurIPS 2025 spotlight ()
Keywords: Sequence Modeling
Primary area: deep_learning

Abstract: Recent efficient sequence modeling methods, such as Gated DeltaNet, TTT, and RWKV-7, have achieved performance improvements by supervising the recurrent memory management through the Delta learning rule. Unlike previous state-space models (e.g., Mamba) and gated linear attentions (e.g., GLA), these models introduce interactions between the recurrent state and the key vector, resulting in a bilinear recursive structure. In this paper, we first introduce the concept of Bilinear RNNs with a comprehensive analysis on the advantages and limitations of these models. Then based on the closed-loop control theory, we propose a novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections. We also implement a hardware-efficient chunk-wise parallel kernel in Triton and train models with 340M/1.3B parameters on a large-scale corpus. Comba demonstrates its superior performance and computation efficiency on both language modeling and vision tasks.

Improving Bilinear RNNs with Closed-loop Control
Jiaxi Hu'
Yongqi Pan
Jusen Du
Disen Lan² Xiaqiang Tang' Weigao Sun
Qingsong Wen
Yuxuan Liang
Thee Hong Kong University of Science and Technology (Guangzhou)
Shanghaai AI Laboratory
SSquirel Ai Learning, USA
Abstract
Recent efficient sequence modeling methods such as Gated DeltaNet, TTT, and RWKV-7 have achieved performance improvements by supervising the recurrent memory management through Delta learning rule. Unlike previous state-space mod- els (e.g., Mamba) and gated linear attentions (e.g., GLA), these models introduce interactions between the recurrent state and the key vector, structurally resembling bilinear systems. In this paper, we first introduce the concept of Bilinear RNNs with comprehensive analysis the advantages and limitations of these models. Then, based on closed-loop control theory, we propose novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections. We also implement a hardware-efficiinn chunk-wis parallel kernel in Triton and train models with 340M/I .3B parameters on large-scale corpus. Comba demonstrates superior performance and computation efficiency in both language and vision modeling.
httpss//gttaab..om/Ileeroo// hrt.aaaallrrraaaaaaaaaaaaaaaaaaa ainah--innaar-attetttee
"Learning without thinking misleads."
1 Introduction
-Confucius, 479 BC
Autoregressive Transformers [95] have become a foundation of modern AI, primarily due to the efficient parallel computation made possible by rofimax-based self-attention. This mechanism enables effective memory scaling by directly appending key and value vectors into the KV cache, which contributes to strong performance However, this design also comes with challenges, including quadratic time complexity and unbounded memory growth during inference [59], which limits the model's scalability for long- sequence tasks. To address these, numerous improvements have been introduced, including sliding window attention [8, 25), sparse attention techniques [64, 1 111, 103], and efficient KV cache management [59].
tasks such as in-context learning and long-context retrieval.
Meanwhile, efficient sequence-mise mixxng approaches such as gated linear attention [112, 79, 22, 107, 105, 5] and selective state space models (SSMs) [38, 26, 84] offer compelling alternative. These models aim to establish a linear key-value associative memory [105, 34] register with constant states and dedeeden) gating (a,, B3,, as presented by S, St + B1o) u. The inherent recurrent structure of these models enables them to maintain constant memory overhead and ((1) time complexity during inference. Despite these models originating from distinct theoretical frameworks, for example, early linear attentions [53, 82) like Linformer [98] attempt to reformulate quadratic attention computation
O o(Q) (o(K))V) with kernel mapping while original SSMs [39, 42] like S4 [41] intent to parameterize a continuous dynamical system to discrete form, recent literature [22, 7, 52, 6] have unified these models to the concept of Linear RNNs.
Corresponding authors wewtliinn@outllo...oo sunveigao@outlook.com @outlook.com). Work done during Yongqi Pan's internship at Hong Kong University of Science and Technology (Guangzhou).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table I: Memorizing Mechanismes in state-of-the-art Sequence Modeling Methods. Softmax Attention ensures precise memory storage, while sliding window attention constrains storage space. Linear R.NNs (lines 3-6) use data-(in)dependar gate for unsupervised memory management, with Mamba2 approximating the forget gate to 1, forcing the model to forget. Bilinear RNNs (lines 7-9) is no longer linear key-value memory register S.+1 (a. ())@()(()) (S., eT))" which supervise the management process based on the Delta learning rule effectively equivalent to minimizing a Stochastic Gradient Descent Vs || Sc well" When ignoring layer normalization and residual components, TTT-Linear can also be categorized as bilinear RNN. As modern Nonlinear RNNs, MIRAS and its variants (c.g., TTT-MLP, Titans [7]. Lattice [52]) have stronger expressiveness because of the nonlinearity 9. high-order optimizations, and MLP-based deep memory, but are limited by the chunk-wise parallelism ($3). Our proposed Comba further improves Bilinear RNNs by closed-loop control.
Model
Memorizing with Gate
Optimization Objective C.
Seftmax Attention: Conductivist based infinite key-value value associative memory regisiery
SA (95]
S. = S 1. append(ks, v.)
2111 expl q/ k.) [ will (97) - as exple] k:) ||0 w.ll"
SWA [8]
S. = 1- v2) drop(k, Mr R)-M)
Linear RNNs: Inductivist bused finite veyae associative memory registers
LA [98)
S. = S, + vckk
(S. vi)
GLA [107]
9, = S (dlgg(e)(( + vekk
- (s vz) + ++|l| /diug(1 (1 ax
HGRN2 [79]
S S.didia((a)) (1(1 - O.)T
(5. - a.), vi) + /diag(1 - a.
Mamba2 [26]
S. S + v.k!
-4 (St v.) + 3 21ll
S. S.
Bilineor RNNs: Moving beyond linear key-value memory registers with memory correction
G-DeltaNet [105)
S = S 10," (T: B.k(A]] ) + Pevek]
Eaip 1 Sike
+ 1l a: 1
RWKV7 [73]
S (dx) kck,k]) wick
1 th
S.k.
< /diap( ar.).8
Comba (ours)
S: - ar 55 k.
B. vole
3. 1|vr
S.
|| 1 S. 11= (g., cki
(Modern) Nonlinear RNNs: Stronger expressiveness and memory capacity, but limited in wise parnlllllim.
TTT-MLP [90]
S.(.) = S:-a() E A Vs
w.ll
A. l|u, (k.)))
MIRAS [6] S. = 0 A (ls( kz) m
& l|v g(v) As + 1 /ding( a.)8.
denotes memory, while RNNs, such as LSTM [46] and GRU [21] are omitted on the table. Softmax attention can also be interpreted as an LI loss with a kemnel function [114].
key-value pairs, oz' denoces close to 88 _I represent smaller/igger factors. Early Nonlincur
The data-dependent gating in Linear RNNs provides a dynamic memory management similar to the adaptive, weighted information fusion in softmax attention. This allows such models selectively update and retain relevant information, enabling Linear RNNs to serve as practical replacements for Transformers in many downstream tasks [48, 62, 49]. However, this mechanism remains heuristic; that is, the model lacks a criterion for determining which memories to forget, and all kcy-value associations are forgotten uniformly, rendering the process less targeted and efficient [105].
To address this, recent models such as DeltaNet [108, 105], RWKV-7 [73], and TTT [90] have ad- vanced state transition to generalized Householder transformations [51], enhancing model's learning capacity and enabling supervised memory control via the Delta learning rule [101]. These architec- tural shifts foster richer interactions between the internal state S and the input information k., moving beyond simple linear key-value memory registery and resembling bilinear dynamics. Consequently, we refer to these models as Bilinear RNNs in this paper. Further improvements [7, 52, 6] have built upon TTT by introducing higher-orde order nonlinear optimization or MLP-basce deep memory, giving rise to modern Nonlinear RNNs, which enhance expressiveness but sacrifice the ability to perform chunk-wisg wise parallelism over the sequence. In summary, research in this field is still in its early stages, and there remain open challenges in achieving a good balance between model expressiveness and hardware- efficient implementation during pretraining.
In this paper, we make the following contributions:
We summarize the progress of efficient sequence modeling methods in Table 1, and highlight the core design principles behind recent advances within the concept of Bilinear RNNs. (Section 2) Inspired by closed-loop control theory, we propose novel Bilinear RNN architecture named Comba. Unlike previous models, Comba features a cellar-plus-low and applies feedback control to the query vector during the output. We further develop hardware-
(SPLR) state transition
2

---

friendly implementation of Comba using chunk-wise parallelism in Triton [93], which achieves a 40% speed improvement in forward propagation compared to Gated-DeltaNee (Section 3)
We pretrain models with 340M and 1.3B parameters and evaluate their performance across a range of tasks, including language modeling and vision tasks. Extensive ablation studies conducted to assess the impact of key components of Comba. (Section 4)
2
Preliminary & Related Works
2.1 Linear RNNs
Unlike autoregressive Transformers that store all contextual information in KV cache, linear RNNs compress highly abstract knowledge into fixed-size state for generalization, structurally resembling energy-based models [56]. i.e., Hopfield networks [65, 31] and neural Hebbian learning systems [18]. Early models like Linformer [98], S4 [41] and RetNet [91] lack effective, data-dependent memory control, resulting in inferior performance to softmax attention. Later models like Mamba [38] and GLA [107] address this by introducing dynamic, projection-base improvements. Formally, these models are linear register systems with key-value associative memory, where the memory is written by the forgetting/input gates (a,, (3) and retrieved via query-based read:
gating, yielding substantial
S: = (at. @.)
(Write),
o S.q. (Read).
(I)
Due to differences in theoretical foundations and development trajectories, linear RNNs have evolved into two main implementation paradigms: (i) linear attentions (LAs) [112, 79, 22, 107, 105, 5] and (ii) state space models (SSMs) [72, 38, 39]. we summarize their key distinctions in Table 2:
i) State size: SSMs like Mamba2 [26] adopt a fixed state expand dimension of 128, resulting in state size of 256D. Whereas in LAs, the state size is determined by the dimen- sion of key/value heads (typically 64). In practice, Mamba2 offers a higher capacity [20], which underpins its advantages in retrieval tasks and hybrid architectures [88, 89].
Table 2: Comparison hetween SSM and LA families with the head number H.
Component SSMs (Mamba2)
LAs
Input value
14. E R220/1
1h E RR///M
State expand
B; C I
k R R
Output
C, R228
q. R
ii) Parameter composition: SSMs like Mamba2 resembles a multi- attention mechanism [26], where the input pro-
State size
2, : 256 x D
S.
dbodu N
ML.
x
jection B (k) and output projection C' (q) are shared across all value heads u (v). Additionally, state space models omit the feedforward network and instead double both the input dimension and the number of layers to
Mode
Multi-value
Multi-head
increase model capacity. In this paper, we empirically follow the linear attention design but keep the head dimension at 256 to match the state size of Mamba2. A detailed architectural ablation is in $4.2.
Chunk-wise Parallel Although Linear RNNs achieve favorable pretraining time complexity of OLLD)), they often slower than softmax attention with (L) complexity on shorter sequences. This is mainly due to the fact that current hardware is highly optimized for matmul operations, which limits the efficiency of linear recurrence, necessitating additional training optimizations. S4 [41] introduces a complex diagonal-plus-lonn rlus-low-rank design using the Cauchy kernel, which is later simplified in DSS [43] and S4D [40] through complex and real diagonal approximations to employ Vandermonde-basse
convolutions. Other models like S5 [84] and Mamba (38] leverage the Blelloch sean algorithm [15] to cache intermediate results and speed up recurrent computation. Recent methods inspired by FlashAttention [25], including Lightning-AIn
[77, 78, 58), GLA [107], and Mamba2 [26). introduce inter-chunk recurrence combined with intra-chunk parallelism to fully utilize matrix compute throughput. A basic formulation using chunk size C can be expressed as:
S|1+1] = S +
K
01 S + (Q18) KT + Maskj/ Vig E CxD (2)
M
2.2 Bilinear RNNs and Beyond
In a neural memory perspective [34], effective memory management remains a central challenge. Unlike Hebbian learning rule [18], which relies on reinforcement-baane memory updates, Delta learning nule [75, 101] focuses on supervised memory control and has been extensively explored in various works, such as fast weight programs [50, 81] and Meta-learning [68, 67].
3

---

[108, 105] employs it as memory correction wi = v St- and based on Eq. 2, introduces an efficient chunk- -wiee parallel algorithm for hardware-efficien training in sequence modeling:
= ( v.)k! = SI k,) + Sruck
O, S.q.-
These models are no longer a linear key-value memory register as in Eq. I; instead, the interaction between the state S and the input k introduces bilinear term Sk, this resembles affine bilinear system' [17, 113, 99, 70). Accordingly, models that are similar to the updating rule in Eq. can be
classified as Bilinear RNNs, and their key strengths are summarized as follows:
Supervised Memory Management As shown in Fig. 1, the Householder transform (I - k, generated by the Delta rule defines a mirror transform, effectively reflecting stored memo- ries across a hyper-plane orthogonal to kiz- Given memory S E DxD with at most D orthogonal memory patterns, when the sequence length >> D, to avoid memory conflicts, this re- flection attenuates components of stored memories { in non-orthogonal directions based on factor 3t. This mechanism implicitly enforces orthogonal memory management, enabling the model to preserve D distinguishable historical memories over time. In this paper, Comba modifies the state transition to (at 3,kkT) which is a scalar- scalar-plus- plus-low low-rank form (SPLR), enabling more flexible supervision. To some extent, this process can be seen Schmidt orthogonalization [57] or a rotation operation [86] on the KV cache.
kt
t-1
B ==
B =1
B = 2
Figure I: Householder transform mirror transform with factor B.
Richer Expressive Power Linear RNNs generally approximate the dense state transition matrix [66, 46] with a diagonal matrix or scalar [40. 26], significantly reducing computational overhead but at the cost of expressiveness [1]. While the additional low-rank terms of the state transition in Eq. 3 improve the model's 's expressiveness while preserving tractability for efficient parallelization.
Building on this, Gated-Deltale [105] introduces a global scalar forgetting gate on the state, and
Delta-Product [83] explore multi-step Householder transform I1; (I kt., kk,5), enabling smooth interpolation between purely diagonal and fully dense transitions. RWKV7 [73] improves the IPLR form to diagonal-plus-low-rad
(DPLR) form, which aligns with insights from HiPPO theory [39, 42], which shows that all orthogonal polynomial projection matrices can be decomposed into DPLR components. Our proposed Comba adopts scalar-plus
(SPLR) form. Empirically, we find that the scalar is sufficiently expressive (similar to the empirical simplification from Mambal to Mamba2) and offers significant pretraining acceleration over RWKV-7.
From another perspective, models such TTT [90] separate the model weights into inner and outer components, updating the inner memory directly via stochastic gradient descent (SGD) [16].
B Σ
S. = S.Sn R-
B.Vs [[LayerNorm(S.-n nk:) 0:112 O2 =:
(4)
where the state S can be parameterized by a matrix or a two-layer MIP to increase memory capacity. When mini-batch B = and ignoring normalization operation, this reduces to the update rule in Eq. 3, and when adopting MLP-based deep memory, the model transitions into a modern form of Nonlinear RNN. Subsequently, Titans [7] introduces data-dependent state decay and momentum, while models like MIRAS [6] and Lattice [52] enhance S using nonlinear higher-order optimization, e.g.. sign function or high-order derivatives. However, there is no free lunch: these models rely on batch gradient descent approach to compensate for the sequence-leve parallelism barrier introduced by their structural complexity, but empirical results show that a mini-batch of (similar to Eq. 3, which is a first-order approach) remains optimal [90], especially in language modeling. Comba retains the bilinear form and offers an efficient chunk-wise parallel optimization.
3 Bilinear RNNs in Closed-loop Control
Unlike perspectives from neural memory optimization, this work revisits Bilinear RNNs through the lens of control theory [23, 24, 87]. As shown in Table 3, linear RNNs such as Mamba2 and
Bilinear systems are linear with respect to state and input individually, but nonlinear overall due to the product term (e.g.. Sk). They regarded as special class of nonlinear systems that preserve controllability.

---

Table Update rules in control/neural memory perspective, with feedback P() and scalar factor d.
Option
Open-loop Control (Mamha2)
Close-loop Control (Comba)
Gated Delta Rule
Input/ / Memorize
S. a.Si + B.0t kg
S: = 0.. B,0r ky
S. = S1 + BBer te ki
Feedback Reflect
N/A
t t: P.(S--1)
vt 11 S1-
Output / Recollect
S,q:
S.q. - (S.)
O, S.q.
GLAA are generally viewed, some extent, as open-loop control systems, where the output does not provide feedback to influence the control behavior. In contrast, another class of systems. known as closed-loop control systems [45], incorporates negative feedback enhance the adaptability of the system and allows it to handle more complex dynamic tasks. According to Wikipedia, a closed-loop controller should use feedback control states or outputs of a dynamical system. So in this paper, Comba adopts a two-stage feedback strategy: the input information u, is first corrected via state-based feedback P:(.). and the output is similarly refined using the same feedback mechanism. Compared to directly output correction methods, e.g., use q. to compute vft this approach is generally considered more robust and better suited for parallel computation (as it only modifies qe at the output stage without involving it in state updates). From this perspective, models such as TTT, DeltaNet, and RWKV-7 incorporate only the first-stage state feedback correction.
Feedback Parameterization Similar to the optimizer perspective, the feedback P.(.) in closed- loop control can be either linear and first-order, or nonlinear and higher-order, However, two major challenges arise: (i) adopting nonlinear or high-order optimization techniques, as in TTT, Lattice, or MIRAS, will hinder chunk-wise parallelism; and (ii) recurrent models suffer from the well-known issue of exponential gradient explosion [74] during training. If we follow DeltaNet using first-order feedback but initialize new vector [106] to interact with the state, it becomes difficult to ensure that the spectral radius of the state transition matrix remains below one. Therefore, considering these factors, Comba follows previous models using the & vector to interact with the state S, while introducing special treatment of feedback strength detailed in the following sections. For standard Comba with forget gate Ct,, state feedback factor 31, input gate Br, and output feedback factor d:
S St-1 k, +B.
E dedxx
= S:(q - -dks)
E Rdv
(5)
State correction
Output enrrection
Compared to previous Bilinear RNNs, Comba exhibits the following structural differences:
Scalar Plus Low-Rank (SPLR) In practice, the scalar gating will be limited to the interval (0, 1). and ||; = (L.2 Norm). DeltaNet formulates the state transition in IPLR structure, while RWKV-7 improves expressiveness via DPLR extension with LoRA-style [47] diagonal ini- tialization, i.c., diag(w) wx) k, However, our results indicate that the low-rank form will impair model capacity. Moreover, recent efforts [12, 55, 11, 69] aim to distill Transformers into recurrent structures to leverage prior knowledge of large-scale pretrained Transformers, where key design principle is to minimize additional parameters. To this end, Comba adopts an SPLR structure that only introduces data-dependent scalar, achieving superior empirical performance compared to RWKV-7 (as imple- mented in FLA [109]), along with a 2 x pretraining acceleration (the memory usage is only half). Recent works [83, 36] have also improved IPLR structures by extending their eigenvalues into the negative domain, e.g.. I - k, to enhance the model's 's state-tracking capability. Notably, SPLR naturally admits negative eigenvalues. For fair comparison, in Table 4, we introduce a variant named Comba-iplr. However, such models with global state decay tend to overfit, and the SPLR structure remains optimal and is more structurally aligned with control theory.
Table 4: State Transition for Comba Variants.
Version
State transition
Figenvalues
Comba-iplr ax 23rkkk])
(-1,1)
Comba-splr (ax - B.k:k])
(-179, 1)
Output Correction Comba introduces additional output feedback, from an optimization perspec- tive, this is equivalent to incorporating a similarity optimization objective (g:, dkk;) with factor d. In neural memory perspective, & ensures that memory is stored as clearly as possible, enabling precise querying by q. This optimization objective directly facilitates this process, and significantly reduces the model's perplexity, thereby enhancing performance ($4.2). Empirically, initializing d
5

---

to 0.02 improves performance for some smaller models (e.g., 340M), enabling gradual learning of the similarity between q and k. For larger models (e.g., 1.3B, 2.7B). initializing d to 1 leads to the greatest performance*. While prior research has largely focused on improving gating mechanisms or optimizing state updates via key-value operations, few models have explicitly modified the output pathway (i.e., the query @ State). We think there appears to be potential connection between Comba and MesaNet [96], as MesaNet also performs query correction in the output stage by solving a closed- form recursive least squares problem [10], resulting in q: = (H, + A)- qt. Notably, the correction in Comba can be implemented with single line of code q: dk, and is applicable to nearly all RNN variants. In App. A.2, we also provide an alternative explanation, namely that is equivalent to the residual matrix D in Mamba [38].
Table 5: Various initialization examples and numerical range of the existing recurrent model gates.
Model
Forget Gate a::
Range
Input Gate Br
Range
GLA [107]
sigmoid Wi Wam.)=1
(0, 1)
N/A softplus (Wa at + c)
Mamba2 [26]
(=p(foofss ( (Wa +
~
~ 0
MetaLA [22]
sigmoid((....]
(0, I)
1 sigmoid (W..:))
(0. 1)
Comba (ours)
exp sofsollllus (Wav c))
Agmoid(Wsar;
B. < E (0, 1)
3.1
Forcing Forgetting for Long-range Modeling
Extensive work [61, 92, 76] has shown that positional encoding is crucial for language models to generalize from short pretraining contexts to unconstrained inference lengths. While gated linear recurrent structures themselves can be viewed as the continuous accumulation of bias [92], enabling these models to extrapolate effectively by learning how to forget. Recent studies [20, 110, 9] suggest that pretraining lengths (c.g., 2K or 4K tokens) are insufficient to fill the model's s state capacity, thus making the forget gate close to 1, forcing the model to learn how forget. As shown in Table 5, evaluate representative gating initialization methods. For models like GLA [107]. the forget gate is initialized to (0, 1) and retains all incremental memories. For MetaLA [22], HGRN2 [79], and GSA [112), the sum of the forget and input gates is constrained to 1, achieving a relative balance in system memory capacity. Experimental results in $4.2 indicate that breaking this balance is necessary. To this end, Comba follows Mamba2's design by constraining the forget gate de to close to 1, while separately initializing the input gate Be (0, 1) [108]. Additionally, feedback correction is @ Br, where b is computed by Sigmoid function to be constrained in interval (0, 1) to ensure that the feedback strength is weaker than the incremental information.
set the strength of state
3.2 Comba with Chunk Parallel
Based on Eq. 5, Comba can be implemented recursively, enabling constant memory usage and O(1) time complexity during the inference stage, where the Python-style pseudocode is provided in App. B.I. However, the naive recursive implementation in PyTorch lacks sufficient matrix multiplication and thread parallelism [25], resulting unacceptable overhead to pretraining. Referring to DeltaNet [105], we optimize Comba through chunk-wise parallelism. App. B.3 presents an alternative form to fuse feedback decay factor into k as p = bk for a flexible invocation.
In the following illustration, [gg E Rxxd for O E (Q, V,O W] defines the chunkwise matrices that stack the q:. kr, Ut, Os, Ur we vectors. Additionally, we set ! i
CC+
D% [4]
Diag(Dl)") = Diag By partially expanding the recurrence to a chunk-wise formulation for Eq. 5, we have:
(i pj..................... ................................ 1 and A.P E RCxC is a matrix with element 1: 1:j
[t /a
ii M
5 = 5%
of T II - M
(6)
1=
Dig ("pseuda" memory dezay)
("pscudo" Incremental memory)
We find that d = yields better performance in most cases, which is the default configuration for Comba.

---

Ousputs
Comba
Hybrid Comba
Linear
XL
Closed-loop RNN
MLP
RMSNom
XN
Comba
MLP
Chank & Recurentt Comba
SWA
Pr
(b) Mamba like Comba
Suftplan
XL
MLP
Sigusii
Comba
Comba
Conv
Conv
Conv
Comba
Linear
Linear
Lincar
Figure 2: Comba Families. The Mamba-like architecture omits MLP layers, uses multi-value attention, and doubles the model depth. For the hybrid model, we incorporate sliding window attention in flexible proportions to boost the model's recall ability. The window size is set to the context length, equivalent to softmax attention.
( preventing parallelization in the sequence level. Then, we employ the WY representation [13] to eliminate these terms:
Eq. involves matrix-matre products at each time step, i.e., Si D
Dij
lir
wit = [3/e
(1
1:r-111
Σ wit
(attr
Hig
Σ
"i0 Bi
p/u
u
(o
(8)
To maximize hardware efficiency, we apply the UT transform [51] Eq. 7-8 reduce non-mannee non-matmul FLOPSs, which is crucial to enable hetter hardware utilization during training:
My Diag
0-((C-1
K())
Up MMy
Vin
(9)
My lower
(Diag
(A
(1/5
K KT K1
(10)
The inverse of lower triangular matrix can be efficiently computed through iterative row-wise approach by forward substitution in Gaussian elimination [37] and maintain data in float32. Notably, Comba computes the inverse matrix once in Eq. 10 (twice in the original mainly attributed to the introduction of a new form of mathematical induction deriving the WY representation in Eq. 7, leading to
This is
concise formulation and resulting in speedup".
Finally, we can formulate Eq. in a matrix form to perform chunk- parallel training:
S |++1|
1:C aft
Sig
(Uly)
W WISTT Diag (atte)
Ky
(11)
O [4] Diag
a
1-C
i
S + Or Kiy o All))
W1n W1n
(12)
inner
intra chunk
"pseudo"
where the query matrix Q is also influenced by the feedback control and can be precomputed by: Diag dg +C K( [ in chunk-wise at minimal cost.
Q (c) Q
'During the forward pass, Comba alleviates the main bottleneck of inverse matrix computation with (d') the original Gated-DeltaNet, yielding 40% speedup. In the backward pass, performance gains diminish due reuse of cached inverse matrices M. However, in large-scale models where recomputation required, Comba expected to offer significant performance advantages. Moreover, this structured modification applies to almost all models with Delta rule. Recently, inspired by Comba, -eltt-ee has also been upgraded to single inversion in flash- -ll--iim linear-at attention, resulting in significant speedup.
7

---

3.3 Neural Architecture
As shown in Fig. 2, we present the architecture for Comba families, where {a, c, d} are trainable scalars (experimental results indicate that data dependency is not required). Following prior work [22. 105, 38, 32]. we introduce short convolutions to qk to incorporate token shift to improve the model's retrieval capacity. We also retain feature map operations [98] and utilize the SiL.U function to approximate the exponential kernel in the softmax attention. To further stabilize training, we apply 1.2 normalization to qk and employ a Sigmoid-basee gating mechanism. Additionally, we explore hybrid architecture [89, 105, 80] by integrating Comba layers directly with softmax attention.
Experiments
Setting In this paper, all models are pretrained based on flas--Inear-attentiio [109] reposi- tory and utilize NVIDIA A800-80G GPUs. The 340M Comba pretraining requires 8x /0 GPU hours, while the 1.3B Comba requires 32x48 GPU hours. We employ the AdamW optimizer [63] with a 3e-4 learning rate, cosine schedule, 0.01 weight decay, and 1.0 gradient clipping. Random seed is 42.
4.1 Operator Efficiency Analysis
As shown in Fig. 3, we compared the speeds of various operators in both forward and backward processes. The recurrent Comba in Py Torch [71] incurs significant com- putational overhead, limiting its scalability for large-scale pretraining. Flash-attention [25] achieves the fastest speed for shorter sequences (e.g., 1024), but its quadratic plexity results in decreasing efficiency as sequence length increases. Among four modern RNN operators, Comba shows nearly 40% speed improvement in the forward pro- cess over Gated-DeltaNed due more efficient formula structure. GLA suffers from slower operator speed due to the use of diagonal gating matrices. Although RetNet achieves the fastest speed, it falls behind other models in performance due the lack of data-dependent memory management (Table 6). Overall, Comba shows potential as a foundational framework.
400-
Forword
fonni_ootssffem--e -lwdh
chunk, chuni, blolbddrbbe)
Computation Time (ms)
chunk, retemiienn/d wnd bwc)
300-
Resh-attrrlles---eee botll
chunt, gine wil-bbe))
250
comba, naie
100-
50
10"
30*
Context Length
Figure 3: Operator speed evaluated the Len--eestin--Becheeeee Be-chmarkk [93] (fwd
and bwd) in single A800-80G GPU.
4.2 Language Modeling & Architecture Ablation
Commonsense Reasoning Ability As shown in the left half of Table 6, (i) most recursive models outperform transformers commonsense reasoning tasks, due to their recursive structure that resembles a chain of thought [100]. (ii) The SPLR structure outperforms both the IPLR and DPLR structures in two model sizes and achieves the highest computational efficiency. (iii) Output correction, i.e. (g, dk), significantly reduces perplexity, enhancing memory utilization during question answering and improving performance across various metrics. (iv) We find that the Mamba architecture design is suboptimal. MLP, as special (non- (non-Hebbian) key-value memory [35, 34], complements the key-value associative memory in the state, which is particularly important for tasks such as inference. (v) Although Mamba's 's multi-valus attention model aids efficient key-value memory storage [59], sacrifices performance compared to standard multi-head attention. (vi) We discover that the initialization of d should be chosen differently for model scales, e.g., d 0.02 for 340M models and 1 for 1. 1.3B models, as smaller models are more prone to incorrect gradient descent directions.
Figure 4 shows the training loss curves of various architectures, and Comba, particularly with output correction, demonstrates lower loss and greater expressive power. However, in our experiments, we found that the IPLR structure typically yields lower loss. This may be because (i) as shown in Table 4, the range for the SPLR structure eigenvalues is slightly smaller than that IPLR, likely due to the special initialization of a. (ii) We observe that in visual modeling, the IPLR version of the model often experiences a rapid loss decrease followed by an increase. Combining these findings with the results in Table 6. we speculate that the IPLR structure is prone to overfitting.
Recall Ability & Hybrid Architecture As shown in the right half of Table 6, (i) the overall results align with the trends observed commonsense reasoning tasks. (ii) Recursive models have tradition- ally struggled with limited recall due to their finite state space, unlike the transformer's 's unlimited
8

---

Table 6: Zero-shot performance of 340M and 1.3B models trained on SlimPajama [85] datasets. The com- monsense Reasoning task is evaluated by Lan-vallattonn-arreeeee [33] and the recall-intensive task follows prefix- wwweeeraatteneeeeeeeeeeeeeeeeeee
[3] with 2K input tokens. Some of the baseline results are from [108] and [29].
Lamb.f
Wiki.
ARC.
ARC. Hella, Lamb. PIQA
Wino.
Avg.
PDA
SWDE SQD.
NQ
TQA. Drop
Avg.
Model& & Scale
pply
ppl.
acc
340M params with 1SB training tokens and 0.5M butchosize tokens
Trans+**
76.46
28.39
44.91
25.94
34,95
26,90
64.31
51.07
41.35
46.14
25.87
33.22
18,94
45.97
19.94
31.68
GLAA
72.41
28.44
45.30
23.13
34.71
26.14
64.58
51.64
40.92
11.26
16.78
27.85
12.77
43.80
17.68
Mamba*
64.75
28.39
46.30
23.60
35.40
26.72
65,00
50.10
41.80
7.14
12.96
24.35
9.47
41.84
17.11
18.81
RWKV7
45.00
25.74
49.03
25.09
36.63
29.01
65.45
51.54
42.79
29.34
29.15
31.81
18.21
49.17
20.56
29.71
G-DeltaNet
45.46
26.47
46.04
23.55
37.28
29.59
66.05
50.75
42.21
20.53
23.34
28.55
14.98
44.91
16.48
Comba-iplr
35.37
24.31
48.15
23.04
38.01
31.71
65,83
51.62
43.06
27.98
27.66
28.92
17.96
47.75
18.35
28.10
Comba-spir
39.91
24.15
48.56
24.32
38.18
30.98
66.73
51.41
43.36
38.51
27.61
30.07
16.38
48.60
21.22
30.40
w/o. (q. (R)
44.91
49
47.94
22.78
37.93
28.96
66.43
50.67
42.45
26.33
28.02
30.03
15.64
48.93
20.36
28.21
w/o. 8 = b
40.17
24.56
48.37
24.36
38.02
31.18
65.53
51.36
43.13
35.66
27.70
29.31
16.72
48.10
20.84
29.72
wlo.
42,05
25.11
48.33
22.94
37.28
30.44
66.38
50.75
42.68
34.31
25.60
29.44
16,24
46.89
19.74
28.70
w/o, output gate
40.16
24.71
48.29
23.16
37.32
29.48
66.70
50.86
42.64
31.79
23.62
29.63
18.53
48.52
20.84
28.82
Initial dm1
39.37
24.27
47.98
22.87
38.36
30.66
66.65
53,04
43.26
30.24
26.24
28.82
15.68
48.04
18.64
27.94
W. mamba-like
43.20
25.45
46.04
22.95
36.96
30.76
64.36
48.78
41.64
30.64
25.31
28.44
17.33
46.68
18.82
27.87
params with 1008 troining tokens and batchsize tokens
Transt+*
19.29
17.61
55.01
28.07
49.21
40.95
70.08
56.27
49.93
44.32
32.43
42.59
24.49
58.47
21.56
37.31
RetNer*
21.97
18.18
$7.49
26.88
48.09
37.75
69.37
53.28
48.81
13.62
22.59
33:46
15.43
53.79
19.79
26.45
GLA
19.66
17.61
55.18
27.56
48.89
40.03
69.86
53.91
49.24
27.61
30.93
35.04
22.27
56.28
19.45
31.93
Mamba*
19.01
17.12
56.22
28.01
50.01
42.05
70.36
54.49
50.19
13.90
25:40
33.20
18.50
53.50
21.70
27.70
G- DeDeltaNet
18.80
17.14
56.82
27.39
49.77
39.94
71.76
51.78
49.58
30.25
27.65
34.06
23.22
58.23
20.36
32.29
Comba iplr
13.58
16.51
57.11
27.99
$1.34
44.40
71.16
52.64
50.77
28.96
34.83
57.03
21.03
Comba-spir
13.39
16.19
58.54
27.90
52.64
44.21
72.03
55.33
51.78
41.69
35.33
36.14
23.69
58.53
22.85
36.37
w. Initial d=1
12.68
16.01
58.42
27.73
53,02
44.94
71.76
55.56
51.91
42.14
38.24
35,47
25.28
59.30
21.31
36.96
w/o. (q.8) k)
15.64
16.94
55.39
26.02
50.30
44.65
68.82
53.12
49.72
36.97
33.55
33.96
23.66 58.12 20.65
34.49
1. For certain reasons, conduct our evaluation the lambda-standard datasct rather than on lambda-openai used in models such as Gated- Deltanet. Consequently, the PPL metric may not be directly comparable with those reported in prior work.
Table 7: Recall-intensive tasks in [3] for hybrid architectures, where the quadratic attention com- ponent is implemented using FlashAttention [25]. The input length is truncated to 2K.
340M Models with Thkens
Sequence: 512, KV Pairs: 64
GuA
Bearanng Line
Accuracy (%)
75
65A
50
GLA
Manrba
FDA SWDE SQD. NQ TQA. Drop Avg.
3.5
25
Hlyaa
Buthet
Trans++
340M params with 158 tokens
46.14
25.87 33,22 18:94
45.97
19.94 31.68
233
GLA-H
58.76
39.46 31.47 16.98
17.68
200
64
128
256
512
GSA-H
62.13
45.36 31.17
20.62
43.78
18.78 36.97
Tranning 0lggaing-323
Model dimension
RWKV7-11
43.60
32,71 33.15
18.02 50.00
19.74 32.87
Figure 4: Training loss on 8x
Figure 5:
Results on synthetic MQAR task with settings in [2].
GDN-H
52.13
38.80
29.90
15.20
35.55
17.73 31,55
A800 GPUs with logging 32.
Combe-H
60.47 37.84 31.70 16.48 47.15
20.11 35.63
KV cache. However, Comba exhibits recall performance very close to that of transformers. (iii) Ablation studies reveal that output correction significantly contributes to Comba's 's recall performance, optimizing the similarity of qk will enhance the model's ability to retrieve precise memories, thereby improving recall performance. (iv) In Fig. 5, we test Comba's 's synthetic recall ability on the MQAR task. The Bilinear RNNs, both Gated-DeltaNet and Comba, demonstrate perfect recall ability.
Inspired by previous hybrid architectures [105, 89, 14, 28, 60, 19]. we empirically replace the second- to-last layer every eight layers of Comba with softmax attention to boost the model's 's recall ability. In Table 7, (i) we find that nearly all hybrid architectures outperform transformers even with a few quadratic layers. (ii) The best overall performance comes from the GSA-H [1 [2], as GSA itself is type of lntra-layer hybrid architecture. Computationally. it can be expressed as two GLA operations followed by a softmax operation to address attention sparsity issues [92], thus improving recall ability. In the future, we plan to combine GSA with Comba for a flexible hybrid.

---

Table 8: Performance on LongBench [4] tasks with 10K length hased on In-evvaluatioon-arrees [33].
Single QA
Mulii QA
Summarization
Few-shot
Code
NQA.
QQA.
MQA.
HQA.
2WM.
MSQ
GvR.
QMS.
MNs.
TRE.
TQA.
SAM.
LCC.
RBP.
AVG.
Transformer++
17.03
11.96
11.65
7.40
15.14
2.40
15.48
9.20
10.72
RetNet
25.34
24.36
21.30
23.50
21.22
14.03
11.49
12.15
7.00
22.94
6.75
17.34
19.32
17.92
GLA
26.80
25.55
23.33
24.23
26.52
23.26
11.94
7.19
9.03
9.54
27.63
5.30
20.80
21.32
18.75
Gated-DeltaNe
27.62
27.55
23.34
25.19
26.63
20.26
12.33
7.24
10.51
6.83
28.42
6.07
20.37
18.12
18.61
Comba-spir
27.73
28.56
25.78
27.49
29.55
23.34
11.61
6.20
9.47
6.58
29.63
7.11
18.04
17:07
19.16
w/o.
27.53
27.12
24.55
27.30
28.73
23.12
12.42
7.01
9.04
6.86
28.14
6.88
17.65
17.49
18.85
Table 9: Performance the ImageNet-1K K [27] classifi- cation, compared Vision Mamba [1 [1] (linear), DeiT [94] (quadratic), and Agent Attention [44] (sparse).
Table 10: Performance on the object tracking datasets such as GOTIOK [54] and LaSOT [30], compared to baselines including Vision Mamba (linear), Agent At- tention (sparse), and Mixformer [104] (quadratic).
Model
Res.
Params.
FLOPs
Top-1
DeiT-T
2244224
5.7(MB)
1.2(G)
72.2%
Model
GOT1OK
LaSOT
AgentT
2244224
6.0
1.2
74.9(+2.7)
AD
SRa.s
SRo.75
Suc.
N-Pre.
Pre.
Vim-T
2244224
7.0
1.5
76.1(+3.9)
0.704
0.796
0.675
0.690
0.785
0.749
Comba-T
2244224
5.8
1.1
76.3(+4.1)
Agent-A
0.695
0.787
0.662
0.644
0.731
0.689
DeiT-S
224+224
22.1
4.6
79.8
Agent-S
2244224
22.7
4.6
80.5(+0.7)
mamba vision
0.700
0.789
0.673
0.677
0.771
0.730
Vim-S
2244224
26.0
5.1
80.3(+0.5)
Comba-splr
0.715
0.804
0.686
0.693
0.789
0.751
Comba-S
224*224
22.6
4.4
80.5(+0.7)
Comha iplha-ipll
0.718
0.809
0.688
0.694
0.786
0.755
Long-context Modeling Ability As shown in Table 8, Comba generally outperforms the other architectures, with a significant lead in QA and Few-shot tasks. However, lags behind Gated- DeltaNet in summarization and code tasks, and the underlying reasons warrant further exploration in the future. Additionally, when the special initialization method in $4 is removed, the model's long-sequence modeling capability decreases, which supports the effectiveness of our approach.
4.3 Vision Modeling
Classification As shown in Table 9, Comba achieves SOTA efficiency- trade-offs across all model scales. For tiny variants. Comba-T improves Top-1 accuracy by 4.1% over DeiT-T with similar parameter count and 8.3% fewer FLOPs. Notably, Comba-T outperforms both Agent-T and Vim-T in accuracy despite requiring fewer computational resources. At small scales, Comba-S matches Agent-S in accuracy while reducing FLOPs by 4.3% and using fewer parameters than Vim-S.
Object tracking To further validate Comba's 's cross-domain capabilities, we extend experiments to object tracking tasks GOT-10k and LaSOT datasets. Unlike static image classification, tracking demands efficient temporal modeling and robustness to appearance variations. As shown in Table 10, Comba variants consistently outperform standard attention mechanisms, with the highest AO (0.718) and SRo.75 (0.688), exceeding Softmax Attention by 1.4% and 1.3%. Besides, Comba (splr) closely matches Softmax Attention without added computational cost. These results underscore Comba's ability to capture long-range dependencies, such as occlusion recovery and motion continuity.
5 Conclusion & Future Work
This paper provides comprehensive summary of the development of recursive models in efficient sequence modeling methods and highlights the reasons behind the success of the latest generation of Bilinear RNNs. Drawing on closed-loop control theory, we propose Comba, a new architecture that incorporates both state feedback and output correction, based on SPLR state transformations. We also implement a chunk-wise parallel operator using Triton. Extensive experimental results demonstrate the practical advantages of Comba. However, this paper also has several limitations. For instance, due to limited computational resources, the experimental scale was not extended to larger models, such as the 2.7B model (which typically requires 32x x /20 GPU hours). Additionally. since models like Titans, Lattice. and MIRAS have not yet been open-sourced, direct comparisons with these models are difficult. In the future, we will focus on addressing the chunk-wise parallel optimization of these models and explore the integration of GSA with the Comba architecture in an elegant hybrid.
10

---

Acknowledgments and Disclosure of Funding
Here, would like to express my gratitude to those who helped me when first entered this field, as well as the lene-i--aattattiiooo community for their valuable feedback on this paper.
This work is mainly supported by the Guangdong Basic and Applied Basic Research Foundation (No. 2025A151501 190011999 1994). This work is also supported by the National Natural Science Foundation of China (No. 62402414) 14). Tencent (CCF-Tencen Open Fund, Tencent Rhino-Bird Focused Research Program), Didi (CCF-DiDi GAIA Collaborative Research Funds), Guangzhou Municipal Science and Technology Project (No. 2023A03J001 1), Huawei Industrial Funds, and the Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A0310628).
References
[1] Ido Amos, Jonathan Berant, and Ankit Gupta. Never train from seratch: Fair comparison of long-sequence models requires data-driven priors. arXiv preprint arXiv:2310.02988 2023,
[2] Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. arXiv preprint arXiv:2312.04921, 2023.
[3] Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher Ré. Just read twice: closing the recall gap for recurrent language models. arXiv preprint arw.v:2407..555550 2024.
[4] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[5] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prud- nikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04511 2024.
[6] Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It's all connected: journey through test-time memorization, attentional bias, retention, and online optimization. arXiv preprint arXiv:2504. 131733 2025.
[7] Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint MrXiv:2501.0066 2024.
[8] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- transformer.
arviv preprint arXiv:2004.05150 2020.
[9] Assaf Ken-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential mamba. arXiv preprint arXiv:2406.1 1442288 2024.
[10] Jacob Benesty, Constantin Paleologu, Tomas Gänsler, and Silviu Ciochină. Recursive least- squares algorithms. In A perspective stereophonic acoustic echo cancellation, pages 63-69.
Springer, 101
[11] Aviv Bick, Tobias Katsch, Nimit Sohoni, Arjun Desai, and Albert Gu. Llamba: Scaling distilled recurrent models for efficient language processing. arXiv preprint utXiv:2502.14458 2025.
[12] Aviv Bick, Kevin Li, Eric Xing. Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratie knowledge to subquadratic models. Advances in Neural Information Processing Systems, 1131178--118 2024.

---

[13] Christian Bischof and Charles Van Loan. The wy representation for products of householder matrices. SIAM Journal on Scientific and Statistical Computing, 8(1):s2-s13, 1987.
[14] Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, et al. Nemotron-h: A family of accurate and efficient hybrid mamba- transformer models. arXiv preprint XXiv:2504.03624, 2025.
[15] Guy Blelloch. Prefix sums and their applications. 1990.
[16] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings COOMPSTTT 2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 177-186. Springer, 2010.
[17] Carlo Bruni, Gianni DiPillo, and Giorgio Koch. Bilinear systems: An appealing class of" nearly linear" systems in theory and applications. IEEE Transactions on automatic control, 19(4):334 348, 1974.
[18] Snehashish Chakraverty, Deepti Moyi Sahoo, Nisha Rani Mahato, Snehashish Chakraverty, Deepti Moyi Sahoo, and Nisha Rani Mahato. Hebbian learning rule. Concepts of Soft Computing: Fuzzy and ANN with Programming, pages 175-182, 2019.
[19] Xiuwei Chen, Sihao Lin, Xiao Dong, Zisheng Chen, Meng Cao, Jianhua Han, Hang Xu, and Xiaodan Liang. Transmamba: Fast universal architecture adaption from transformers to mamba. arXiv preprint 1rXiv:2502.15133 2025.
[20] Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Stuffed mamba: State collapse and state capacity of ran-based long-context modeling. arXiv preprint mrXiv24410.01155 2024.
[21] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- decoder for statistical machine translation. arXiv preprint arXiv: 1406: 1078, 2014.
[22] Yuhong Chou, Man Yao, Kexin Wang, Yuqi Pan, Rui-Jie Zhu, Jibin Wu, Yiran Zhong, Yu Qiao, Bo Xu, and Guoqi Li. Metala: Unified optimal linear approximation softmax attention map. Advances in Neural Information Processing Systems, 37:71034-7100 2024.
[23] James J Collins and Carlo J De Luca. Open-loop and closed-loop control of posture: random- walk analysis of center-of-pressure trajectories. Experimental brain research, 95:308-318, 1993.
[24] JJ Collins, CJ De Luca, A Burrows, and LA Lipsitz. Age-related changes in open-loop and closed-loop postural control mechanisms. Experimental brain research, 104:480-499 1995.
[25] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances neural information processing systems, 35:16344-16359, 2022.
[26] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint utXiv:2455.21061 2024.
[27] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Icee, 2009.
[28] Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sumil Mahabalesh- warkar, Shih- Liu, Matthijs Van Keirshilek, Min-Hung Chen, Yoshi Suhara, al. Hymba: A hybrid-head architecture for small language models. arXiv preprint arXiv:2411.13661
2024.
[29] Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, and Yu Cheng. Mom: Linear sequence modeling with mixture-of-memories arXiv preprint arXiv:2502. 1368.5550.13885, 2025.
12

---

[30] Lin L. T. Fan H., Bai H. X. and et al. Lasot: A high-quality scale single object tracking benchmark. International Journal of Computer Vision, 129:439 461, 2021.
[31] Nabil H Farhat, Demetri Psaltis, Aluizio Prata, and Eung Pack. Optical implementation of the hopfield model. Applied optics, 24(10):1469 14110)11699-47,, 1985.
[32] Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.1400 14052, 2022.
[33] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[34] Samuel J Gershman, Ila Fiete, and Kazuki Irie. Key-value memory in the brain. arXiv preprint 2Xiv:2501.02250 2025.
[35] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer fred-rd layers are key-value memories. arXiv preprint arXiv:2012.14913 2020.
[36] Riccardo Grazzi, Julien Siems, Arber Zela, Jörg KH Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear rmns through negative eigenvalues. arXiv preprint arXiv:2411.12533 2024.
[37] Joseph Grear. Mathematicians of gaussian elimination. Notices of the AMS, 586))782--79 2011.
[38] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
[39] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474-1453 1487, 2020.
[40] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971-35985 2022.
[41] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint (tt::2111.000990 2021.
[42] Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. How to train your hippo: State space models with generalized orthogonal basis projections. arXiv preprint arXiv:2206.12033 2022.
[43] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982- 22994, 2022.
[44] Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear atten- tion. In Computer Vision - ECCV 2024: 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part I., page 124-140. Springer, 2024.
[45] Häkan Hjalmarsson. From experiment design to closed-loop control. Automatica, 41(3)3393- 438, 2005.
[46] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 159::17351780, 1997.
[47] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
13

---

[48] Jiaxi Hu, Disen Lan, Ziyu Zhou, Qingsong Wen, and Yuxuan Liang. Time-ssm: Simplifying and unifying state space models for time series forecasting. arXiv preprint arXiv:2405.16312. 2024.
[49] Zheyuan Hu, Nazanin Ahmadi Daryakenari, Qianli Shen, Kenji Kawaguchi, and George Em Kariiadakis. State-space models are accurate and efficient neural operators for dynamical systems. arxiv preprint arXiv:2409.033: 03230, 2024.
[50] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond lin- ear transformers with recurrent fast weight programmers. Advances in neural information processing systems, 34:7703-7717, 2021.
[51] Thierry Joffrain, Tze Meng Low, Enrique S Quintana-Orti, Robert van de Geijn, and Field G Van Zee. Accumulating householder transformations, revisited. ACM Transactions Mathematical Software (TOMS), 32(2):169-17)) 2006.
[52] Mahdi Karami and Vahab Mirrokni. Lattice: Learning to efficiently compress the memory. arXiv preprint arXiv:2504.0566 05646, 2025.
[53] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference machine learning, pages 5156-5165. PMLR, 2020.
[54] X. Zhao I.. Huang and K. Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1562-1577, 2021.
[55] Disen Lan, Weigao Sun. Jiaxi Hu, Jusen Du, and Yu Cheng. Liger: Linearizing large language models gated recurrent structures. arXiv preprint arXiv:2503.01496 2025.
[56] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy-based learning. Predicting structured data, ((0), 2006.
[57] Steven J Leon, Åke Björck, and Walter Gander. Gram-schmidt orthogonalization: 100 years and more. Numerical Linear Algebra with Applications, 20(3):492-533 2013.
[58] Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313 2025.
[59] Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. A survey on large language model acceleration based on kv cache management. arXiv preprint arXiv:2412. 19442, 2024.
[60] Yixing Li, Ruobing Xie, Zhen Yang. Xingwu Sun, Shuaipeng Li, Weidong Han, Zhanhui Kang, Yu Cheng, Chengzhong Xu, Di Wang, et al. Transmamba: Flexibly switching between transformer and mamba. arXiv preprint arXiv:2503.2406;; 2025.
[61] Zhixuan Lin, Evgenii Nikishin, Xu Owen He, and Aaron Courville. Forgetting transformer: Softmax attention with a forget gate. arXiv preprint ttXiv:2503.021330 2025.
[62] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. Advances in neural information processing systems, 37:103031 2024.
[63] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
[64] Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture block attention for long-context 1lms. arXiv preprint arXiv:2502.13 2025.
[65] ROBERTJ McEliece, Edwarde Posner, EUGENER Rodemich, and SANTOSHS Venkatesh. The capacity of the hopfield associative memory. IEEE transactions on Information Theory, 33(4):461-44) 1987.
14

---

[66] Larry R Medsker, Lakhmi Jain, et al. Recurrent neural networks. Design and Applications, 5(64-67):2, 2001.
[67] Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalearned neural memory. Advances in Neural Information Processing Systems. 32, 2019.
[68] Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 1, page 397, 2017.
[69] Daniele Paliotta. Junxiong Wang, Matteo Pagliardini, Kevin Y Li, Aviv Bick, Zico Kolter, Albert Gu, François Fleuret, and Tri Dao. Thinking slow, fast: Scaling inference compute with distilled reasoners. arXiv preprint arXiv:2502.20339, 2025.
[70] Panos M Pardalos and Vitaliy A Yatsenko. Optimization and control of bilinear systems: theory, algorithms, and applications, volume 11, Springer Science & Business Media, 2010.
[71] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[72] Badri Narayana Patro and Vijay Srinivas Agneeswaran. Mamba-360: Survey of state space models transformer alternative for long sequence modelling: Methods, applications, and challenges. arXiv preprint arXiv:2404. 12, 2024.
[73] Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Haowen Hou, Janna Lu, William Merrill, Guangyu Song. Kaifeng Tan, Saiteja Utpala, al. Rwkv-7" goose" with expressive dynamic state evolution. arXiv preprint arXiv:2503, 14456, 2025.
[74] George Philipp. Dawn Song, and Jaime G Carbonell. The exploding gradient problem demystified-ddffmition, prevalence, impact, origin, tradeoffs, and solutions. arXiv preprint arXiv:1712.05577 2017.
[75] DL. Prados and SC Kak. Neural network capacity using delta rule. Electronics Letters, 25(3):197-199, 1989.
[76] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409, 2021.
[77] Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. arxiv preprint arXiv:2401.04658 2024.
[78] Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. arXiv preprint 11Vv22405.17811 2024.
[79] Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong, Hgrn2: Gated linear mns with state expansion. arXiv preprint arXiv:2404.07904, 2024.
[80] Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.07522. 2024.
[81] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International conference machine learning, pages 9355-9366. PMLR, 2021.
[82] Zhuoran Shen, Mingyuan Zhang. Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531-3539, 2021.
[83] Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. Deltaproduct: Improving state-tracking in linear mns via householder products. arXiv preprint arXiv:2502. 10299 2025.
15

---

[84] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint 1.XXx:2208.049333 2022.
[85] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. sl
https: / /wwwwwwww// www.braarbbr..com www.braa..ee/bllrr...eeeeeee net/blog/ ama,
2023.
[86] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568:127063. 2024.
[87] Jingrui Sun, Xun Li, and Jiongmin Yong. Open-loop and closed-loop solvabilities for stochas- tic linear quadratic optimal control problems. SIAM Journal on Control and Optimization, 5465):2274-2308, 2016.
[88] Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, and Yu Cheng. Linear-moe: Linear sequence modeling meets mixture-of-experts. arXiv preprint arXiv:2503.05447, 2025.
[89] Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, et al. Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent. arXiv preprint arXiv:241 1.02265, 2024.
[90] Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang. Sanmi Koyejo, et al. Learning to (learn at test time): Runs with expressive hidden states. arXiv preprint #tXiv:2407.04622 2024.
[91] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint MtXiv:230.008621. 2023.
[92] Shawn Tan, Yikang Shen, Songlin Yang, Aaron Courville, and Rameswar Panda. Stick- breaking attention. arXiv preprint arXiv:2410. 17980, 2024.
[93] Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10-19, 2019.
[94] Hugo Touvron, Matthieu Cord. Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers distillation through attention. In Proceedings of the 38th International Conference Machine Learning, pages 10347-10357.
PMLR, 2021.
[95] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[96] Johannes Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Max- imilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, et al. Mesanet: Sequence modeling by locally optimal test-time training. arXiv preprint arXiv:2506.05233, 2025.
[97] Ke Alexander Wang, Jiaxin Shi, and Emily B Fox. Test-time regression: a unifying framework for designing sequence models with associative memory. arXiv preprint arXiv:2501.12352 2025.
[98] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04766, 2020.
[99] Xinyue Wang, Junxia Ma, and Weili Xiong. Expectationn-mxxiizaatiii algorithm for bilinear state-space models with time-varying delays under non gon-saassiann noise. International Journal of Adaptive Control and Signal Processing, 27(10))270--7224 2023.
16

---

[100] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. of-thhughtt prompting elicits reasoning in large language models. Advances in neural information processing systems, 45:24824-244831 2022.
[101] Bernard Widrow and Marcian E Hoff. Adaptive switching circuits. In Neurocomputing: foundations of research, pages 123-133 134. 1988.
[102] Darrell Williamssn. Observation of bilinear systems with application to biological control. Automatica, 13(3):243-255) 1977.
[103] Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring, arXiv preprint arXiv:2503, 16428, 2025.
[104] L. Wang Y. Cui. C. Jiang and G. Wu. Mixformer: End-to-end tracking with iterative mixed at- tention. 2022 TEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13598-13660 13608. CVPR Organization, 2022.
[105] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464 2024.
[106] Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda, and Yoon Kim. Path attention: Position encoding via accumulating householder transformations. arXiv preprint arXiv:2505. 16381, 2025.
[107] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficien training. arXiv preprint arXiv:2312.06635, 2023.
[108] Songlin Yang, Bailin Wang, Yu Zhang. Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06488 2024.
[109] Songlin Yang and Yu Zhang. Fla: A triton- triton-based library for hardware-cfffiicen implementations of linear attention mechanism, January 2024.
[110] Zhifan Ye, Kejing Xia, Yonggan Fu, Xin Dong, Jihoon Hong. Xiangehi Yuan, Shizhe Diao, Jan Kautz, Pavlo Molchanov, and Yingyan Celine Lin. Longmamba: Enhancing mamba's long con- text capabiliite via training-free receptive field enlargement. arXiv preprint arXiv:2504.1600 16053, 2025.
[111] Jingyang Yuan, Huazuo Gao. Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie. YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-alige aliiged and natively trainable sparse attention. arXiv preprint 1XXv:2502.11108)
2025.
[112] Yu Zhang, Songlin Yang, Rui-Jie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear- time sequence modeling. Advances in Neural Information Processing Systems, 37:1 16870-1168888 2024.
[113] Yingbo Zhao and Jorge Cortés. Gramian-based reachability metrics for bilinear networks. IEEE Transactions on Control of Network Systems, 4(3):620-611, 2016.
[114] Shu Zhong, Mingyu Xu, Tenglong Ao, and Guang Shi. Understanding transformer from the perspective of associative memory, arXiv preprint arXiv:2505. 19488, 2025.
[115] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: efficient visual representation learning with bidirectional state space model. In Proceedings of the 41st International Conference on Machine Learning, pages 62429-62445 JMLR.org, 2024.
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
As shown in the abstract and the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations in the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
We provide the proof/derivation in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We state the experiment setting before the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
We provide the code in an anonymous link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niiss wwtpcccppppcceeeet.ccccccccccltt ggwudde//oodeeeee
wwsisssssiiiiyycy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We state the experiment setting before the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We conducted the evaluation using open-source tools that inherently include statistical significance, though we chose not to present these results in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the experimental details and operator speed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: pattpa//nneurigee cc httpp///lcctttiiiciiiieeeel.....
Answer: [Yes]
Justification:
We follow the Code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We state our work has the potential to become a foundation structure for Nonlinear RNN.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We use open-source datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited corresponding works in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code in the anonymous link has detailed readme.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /tapps/////urps...//p http://aurrrs...eeeeeeeeeeeeeeee htt..///oooereeeccc///////// for what should or should be described.
24

---

Supplementary Material
Improving Bilinear RNNs with Closed-loop Control
TABLE OF CONTENTS
A Additional Background
25
A.1 Bilinear Systems
25
A.2 Comba in a State Space Model Perspective
25
B Operator Implementation Derivation
25
Recurrent Implementation for Comba
25
B.2 WY Representation & UT Transform
26
B.3
Comba-SPLR-pk
26
A
Additional Background
A.1 Bilinear Systems
Formally, TTT, Geted-DeltaNet, RWKV-7, and Comba should be classified bilinear systems [17, 113, 99, 102, 70]. These systems are linear with respect to both S and K, V, but due to the interaction between S and K, the overall system is nonlinear. Typically considered special type of nonlinear system, bilinear systems possess more expressive power than linear systems while remaining more controllable than strictly nonlinear systems, such chaotic systems. They widely applied in the study of physical systems and biological population dynamics [102, 70].
A.2
Comba State Space Model Perspective
Table 11: Update rules in control & neural memory perspective, with feedback P(-) and scalar factor
Oplion
Open-loop Control (Mambu2)
Close-lopp Control (Combu)
Gated Delta Rule
Input/ Memorize
exp( Ar Aja A Bou. exp( (osp(A.A)ar (A:A)a + B.u. n0
S. 0. + B.0t k
Feedback / Reflect
P. (
S.
Output Recollect
= C, + Du,
C.a: D(u: P(().))
S.q.
In Table 11, we provide an explanation of Comba from the perspective of state-space models. Here, 3. corresponds to A,, which is Euler discretization. In state-space models, there typically residual term Du, which we integrate into the residual connection in our franework to omit it.
B Operator Implementation Derivation
B.1 Recurrent Implementation for Comba
We provide the recurrent Comba in Algorithm 1.
def Recurrent comba (q, k, alpha, beta, b, d):
B, т, H, D shape
q_new =q 2er.sseeos...eer torch. zeros (b, h, d, d)
Output correction
o, S
for i in range (T):
.. k, aalhaa beta [: 1], k[;, i], alphal:, i],
betal:, 1]
25

---

-v_new _beta [..., None] (v[:, i] (S * _k...., Nonel)
(-2))
At [..., None] _k. unsqueeze (-1) _v_new unsqueeze
(-2)
ol: i] torch einsum bhd, bhdm >> >>bmm -9, S)
return
Listing 1: Recurrent Comba-pk in Pytorch Minech-iike/ae Pseudo-code for Inference
B.2
WY Representation UT Transform
Detailed derivation can found the Appendix of DeltaNet [108, 105].
B.3
Comba-SPLRRpk
We also provide an alternative implementation, where bk is integrated into p to make it more close to control theory, these two are equivalent.
def Recurrent Recurrent_comba (q, k, v, P. At, dt, D) b, t, h, d q. shape q-new q D None] p Output correction S = torch zeros_ like(()) torch zeros h, d, d)
for i range (t):
9,, _,, _P. At, dt q_new [:, 11, k[:, i1, p(:, i], At (t,
J. dt i]
v_new dt ...., None] (v[:, i] - (S * -P p[..., Nonel) sum
(-2))
At (..., None]
_k. unsqueeze (-1) _v_eew unsqueeze
(-2)
ol:, i] torch einsum (* bhd bhdn -> bhm .9, S)
return
Listing 2: Recurrent Comba-pk in Pytorch-like Pseudo-code for Inference
By partially expanding the recurrence for Eq. 5, we have
ii (a
1 = St
S
Pie
+
B Pi
Ki J=C+1
II
(al - Smpia
("pseudo" memory docay)
Hi
("pseudo" Incremental memuryy
Then, employ the WY representation [13]:
Dig
why
[1
1:-
Pij Σ Euiy (o ((fT-(A(Tp)s))
H
Σ
uig
uiy
(a (or (a kipppi))
To maximize hardware efficiency, we apply the UT transform [51] to reduce non-matin non-matmul FLOPs. which is crucial to enable better hardware utilization during training.
W1g Mn Diag [B]
(1)
(-(c-1))
Pic.
U1
M Diag
(B Vi
MØ T + lower (Diag (0) (4
(i-1)/
K
The inverse of lower triangular matrix can be efficiently computed through an iterative row-wise approach by forward substitution in Gaussian elimination [37] and maintain data in float32.
26

---

Then we have the following vector form:
1/ $ Pij
= [1
S
+
प (0())))) kijl
% ii a
S P4
)) (a (i
Equivalently, in matrix form:
UU11 Wwy ) Diag (ff Ki
5/+1| Oid] S/g
O₁ Diag (ali s
Qu Kig @ All
(Un - Win
inner chunk
chunk
"pscudo" vvllue term
where the query matrix Q is also influenced by the closed-loop control and can be precomputed by: Q Q Diag(d)) ( P(t
27


Reviews:
[{"review_id": "TBJFGcXmRW", "reviewer": "Reviewer_cuva", "summary": "The paper introduces Comba as a new model to the series of sub-quadratic architectures like Mamba-2 \\[1] and Gated-DeltaNet \\[2]. It makes the following contributions:\n\n1. It improves the Gated DeltaNet recurrence,\n\n   $$\n    \\mathbf{S_t}  = \\mathbf{S_{t-1}} {\\alpha_t} (\\mathbf{I} - {\\beta_t} \\mathbf{k_t}\\mathbf{k_t}^T) + \\beta_t \\mathbf{v_t}\\mathbf{k_t}^T\n   $$\n\n   to\n\n   $$\n     \\mathbf{S_t}  = \\mathbf{S_{t-1}} (\\alpha_t \\mathbf{I} - \\tilde{\\beta}_t \\mathbf{k_t}\\mathbf{k_t}^T) + \\beta_t \\mathbf{v_t}\\mathbf{k_t}^T\n   $$\n\n   untying the two $\\beta$s and relocating $\\alpha$ (motivated by control theory).\n\n2. It uses UT transforms for its kernel, making it 40 % more efficient than the WY transform.\n\n3. It shows improvements in performance on language modeling, recall tasks, Long Bench, and ImageNet classification.\n\n\n-----\n\n[1]: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. Tri Dao, Albert Gu\n\n[2]: Gated Delta Networks: Improving Mamba2 with Delta Rule. Songlin Yang, Jan Kautz, Ali Hatamizadeh", "questions": "Please see weaknesses", "limitations": "Please see weaknesses", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. The paper is well written and the proposed change is straightforward to follow.\n2. The PPL improvements on Wiki and LAMBADA over Gated DeltaNet are substantial.\n3. The kernel achieves a 40 % speedup compared to the WY transform.\n\n### Weaknesses\n\n1. The switch from WY to UT transform for Delta Rule–based methods appears to have been introduced in the DeltaNet paper \\[3]. Could the authors clarify how Comba’s algorithm differs from \\[3] beyond the expected updates to the $W$ and $T$ matrices?\n\n2. In Table 6 (main LM results), I think important baselines seem to be under-tuned/missing:\n\n   * **Gated DeltaNet**:\n\n     * DeltaNet \\[3] reports Wiki/LAMBADA PPLs of $(28.24\\,37.37)$ and $(16.87\\,12.21)$ for the 340M/1.3B models on SlimPajama (15 B/100 B tokens).\n     * The authors report $(26.47\\,45.46)$ and $(17.14\\,18.80)$ for Gated DeltaNet. These results are worse than DeltaNet itself, even though Gated DeltaNet is known to more performant than DeltaNet.\n     * Comba achieves $(24.15\\,39.91)$ and $(16.01\\,12.68)$.\n      \n     Under this light, the results become much more mixed than they seem to be. Could the authors please comment on this discrepancy\n\n   * **Mamba-2 vs. Mamba**:\n     I recommend using Mamba-2 \\[1] rather than the original Mamba \\[4] in Table 6, as Mamba-2 is the more recent and better-performing variant.\n\n   * **Transformer++**:\n     I understand that the authors have used a previous paper for Transformer++ results, but I would recommend training this model in-house. The reported PPL numbers strongly suggest that this baseline has been poorly tuned. For reference, please see that the gap is much less pronounced in the numbers reported in Table 2 in MesaNet \\[5]\n\n\nTo sum up, I believe Gated DeltaNet, Mamba-2, and Transformer++ are the most important baselines, and they appear under-tuned/missing. I would recommend better-tuning these baselines to more reliably understand Comba’s improvements.\n\n----\n\n[3]: Parallelizing Linear Transformers with the Delta Rule over Sequence Length. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, Yoon Kim\n\n[4]: Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Albert Gu, Tri Dao\n\n[5]: MesaNet: Sequence Modeling by Locally Optimal Test-Time Training. Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Agüera y Arcas, João Sacramento", "was_revised": true, "final_justification": "The authors have addressed my concerns on the both the novelty in the use of one inversion v/s two inversions in Gated Deltanet as well as the differences in perplexity reported compared to previous papers. I have raised my score to an accept.", "rebuttal": "Thank you very much for your detailed review. Your concerns align closely with the clarifications we have incorporated into the revised version. As NeurIPS does not allow document revisions, we will first restate each point and then provide our response.\n\n\n# Q1: The difference between Comba and Gated-DeltaNet.\n\nComba offers three key contributions compared to Gated-DeltaNet, which we will analyze point by point. Additionally, during the review period, we conducted further experiments, and we would like to share some new findings with you alongside our responses.\n\n\n## A1.1: Kernel Speedup via Comba's Specific Chunkwise Parallelism\n\nWe believe the primary contribution of Comba lies in its direct acceleration of DeltaNet-style operators. Specifically, Comba achieves up to a **46% speedup** compared to Gated-DeltaNet in forward propagation when `head_dim = 128`, which is of high practical value. Comba has already been open-sourced and supports verification. \n\nThis performance gain primarily originates from the structural difference in how Comba constructs the **WY representation** (Equation 7), which leads to the **elimination of one matrix inversion** in the UT transformation process (Equation 10). Specifically:\n\n**Gated-DeltaNet**: $M_1 = \\mathbf{I}-\\text{lower}(\\text{Diag}(\\beta)\\mathbf{K}\\mathbf{K}^T)^{-1}, ~~~~M_2 = \\mathbf{I}-\\text{lower}(\\text{Diag}(\\beta)(\\mathbf{\\Gamma}\\odot\\mathbf{K}\\mathbf{K}^T))^{-1},~$\n\nwhere $\\mathbf{\\Gamma}_{ij} = \\prod_1^j \\alpha_t / \\prod_1^i \\alpha_t$.\n\n**Comba**：$M = \\mathbf{I}-\\text{lower}(\\text{Diag}(\\beta)(\\mathbf{\\Gamma}\\odot\\mathbf{K}\\mathbf{K}^T))^{-1},~$\n\nwhere $\\mathbf{\\Gamma}_{ij}=\\prod_0^{j-1} \\alpha_t/\\prod_1^i \\alpha_t$  **(The numerator contains a shift)**.\n\n\nMatrix inversion is a major performance bottleneck for kernel operators. In models like Comba and Gated-DeltaNet, the matrices to be inverted are masked with lower-triangular patterns, breaking symmetry and preventing the use of the Woodbury identity to reduce complexity to $\\mathcal{O}(n^2)$. As a result, inversion typically relies on Gaussian elimination with $\\mathcal{O}(n^3)$ complexity. By eliminating one matrix inversion, Comba achieves substantial speed gains.\n\nImportantly, this benefit can be extended to other DeltaNet-style models. Recently, inspired by Comba, the implementation of the open-sourced Gated-DeltaNet was updated: it now first derives a Comba-like formulation and then computes using the Comba operator. **Comba is, to our knowledge, the first model to introduce this operator structure**. This contribution was not emphasized in our original submission but will be further clarified in the revised version.\n\n## A1.2: Superior SPLR Structure in Comba\n\nAs described in lines 167–181 of the paper, Comba employs an SPLR structure $(\\alpha_t-b*\\mathbf{k}_t\\mathbf{k}_t^{T})$ with eigenvalues in $( -1, 1 )$, which offers strictly greater expressiveness than the $( 0, 1 )$ range used in Gated-DeltaNet $\\alpha_t(\\mathbf{I}-\\mathbf{k}_t\\mathbf{k}_t^{T})$. Detailed experiments in Table 3 further demonstrate that among the three latest generation (Delta-based) RNNs that support efficient parallelism (Comba, Gated-DeltaNet, and RWKV-v7), Comba's SPLR structure outperforms the IPLR structure of Gated-DeltaNet and the DPLR structure of RWKV-v7. \n\nMoreover, as shown in Table 3, the update rule used in Gated-DeltaNet appears to be incorrect, as it cannot be explicitly written in the standard Stochastic Gradient Descent (SGD) form. In contrast, Comba's formulation adheres to this structure, making it theoretically sound. This issue has also been noted in several recent blogs (in accordance with NeurIPS rebuttal guidelines, we can not include external links):\n\n**Gated-DeltaNet**: $S_t = S_{t-1} - \\beta_t \\nabla_S\\|\\| \\frac{1}{ \\alpha_t } v_t - \\alpha_t S_{t-1} k_t \\|\\|^2$\n\n**Comba**: $S_t = S_{t-1} - \\beta_t \\nabla_S\\|\\| v_t - S_{t-1}k_t \\|\\|^2$\n\n## A1.3: Significant Impact of Comba's Output Correction $\\mathbf{o}_t=\\mathbf{S}_t(\\mathbf{q}_t-d\\mathbf{k}_t)$\n\nDuring the review period, we conducted a more comprehensive investigation into Comba’s output correction mechanism (`q_t - d*k_t`) and observed unexpectedly strong improvements. Integrating this mechanism into both the Gated-DeltaNet and GLA models led to notable loss reductions. (1.3B/100B Gated-DeltaNet dropped from 2.213 to 2.175, and 1.3B/100B GLA dropped from 2.232 to 2.204). The significant improvement of Comba over Gated-DeltaNet can also be largely attributed to the output correction mechanism.\n\nWhile prior RNN research has largely focused on improving gating mechanisms or optimizing state updates via key-value operations, few models have explicitly modified the output pathway (i.e., the query @ State). We think there appears to be a potential connection between Comba and MesaNet, as MesaNet also performs query correction in the output stage by solving a closed-form recursive least squares problem, resulting in $q_t=(H_t + \\Lambda)^{-1}q_t$. Notably, the correction in Comba can be implemented **with a single line of code** `q_t - d*k_t` and is applicable to nearly all RNN variants.\n\n\n# Q2: The evaluation result.\n\nThank you again for your careful and rigorous review. Some experimental details may have caused confusion, and we would like to address each of your concerns regarding evaluation metrics in detail. All experiments are reproducible using the **open-sourced Comba model** and the provided **experiment settings** in the paper.\n\n\n## A2.1: PPL Gap on Wiki/LAMBADA Compared to DeltaNet\nAs shown in Table 6, the Wiki PPL scores in our paper are relatively close to the published DeltaNet paper. We assume your main concern lies with the LAMBADA results.\n\nThis discrepancy stems from the fact that the `lm-evaluation-harness` library includes two versions of the LAMBADA dataset: `LAMBADA_openai` and `LAMBADA_standard`. DeltaNet uses `LAMBADA_openai`, while Comba (due to IP restrictions to HuggingFace) and another paper in this field [1] adopt `LAMBADA_standard`. This difference in dataset versions leads to noticeable variation in PPL scores. Thank you for pointing this out. We will clarify which version of the LAMBADA dataset was used in the revised paper.\n\nTo address your concern more thoroughly, as shown below, we re-evaluated all models using the `LAMBADA_openai` dataset. The results align with the conclusion reported in our original paper. We think that `LAMBADA_openai` may be a truncated version of `LAMBADA_standard`. Typically, when the input length during evaluation exceeds the pretraining context length, it can lead to a significant increase in PPL. In our experiments with `LAMBADA_standard` using `lm-evaluation-harness`, we found that setting `max_length=2048` during evaluation noticeably reduces the PPL. This further supports our assumption and explains the observed differences.\n\n| PPL in Lambda_openai | 340M/15B |1.3B/100B \n|-|-|-|\n| Transformer++  | 37.78 |  18.37  |\n| Gated-DeltaNet | 34.22  |   17.21   |\n| Comba | **31.17**|  **13.06**  |\n\n[1] Mom: Linear sequence modeling with mixture-of-memories\n\n## Q2.2 Evaluation for Mamba2.\n\nWe appreciate your comment. As prior works [2,3,4] have already demonstrated that Gated-DeltaNet outperforms Mamba2 in fair settings, we chose Gated-DeltaNet as our primary comparison target. We have now added Mamba2 evaluation results for 340M and 1.3B models:\n\n| PPL in Wiki | 340M/15B |1.3B/100B |\n|-|-|-|\n| Mamba2  | 26.94 | 17.31 |\n| Gated-DeltaNet  | 26.47 | 17.14 |\n| Comba | **24.15** | **16.01** |\n\n| Acc. in Commonsense Reasoning | 340M/15B |1.3B/100B |\n|-|-|-|\n| Mamba2  | 42.14 |  49.26 |\n| Gated-DeltaNet  | 42.21 |  49.58  |\n| Comba | **43.36** | **51.91** |\n\nIt is clear that Comba > Gated-DeltaNet > Mamba2 across scales. This is likely because Mamba2 still follows the Hebbian learning rule (L1 Loss for state updates), whereas Gated-DeltaNet and Comba rely on the Delta rule (L2 Loss for state updates), which is stronger. We appreciate your suggestion and will include Mamba2 results in the revised paper. And in Table 6, we have already shown that applying a Mamba2-like architecture to Comba leads to a significant performance drop.\n\n[2] Gated delta networks: Improving mamba2 with delta rule\n\n[3] MesaNet: Sequence Modeling by Locally Optimal Test-Time Training\n\n[4] Log-Linear Attention\n\n\n## Q2.3 Bad PPL for Transformer++.\n\nThank you for your question. The reason behind the higher PPL for Transformer++ at the 340M scale is again related to the use of `LAMBADA_standard` instead of `LAMBADA_openai`. The results in Answer 2.1 confirm that the models in our paper were not poorly tuned, and the PPL difference arises solely from the evaluation dataset choice. Moreover, all models in the original Comba paper used `LAMBADA_standard`, so the reported comparisons remain fair.\n\n> We will incorporate all the details into the revised version of the manuscript. Considering that **other reviewers have generally regarded this paper as a good submission**, we would be sincerely grateful if you would consider raising the score, provided your concerns have been addressed. Should you have any further questions, we would be more than happy to respond."}, {"review_id": "cZi645wab0", "reviewer": "Reviewer_fcuH", "summary": "This paper presents Comba, a nonlinear RNN model with SPLR state transition and output correction to improve the model performance. Using chunkwise parallelization, Comba can be efficiently trained, with 40% speedup compared to Gated DeltaNet. The effectiveness of two proposed techniques is confirmed by experimental results on language and vision tasks.", "questions": "Questions:\n\n1. What is the required memory size for different sequence lengths during training? How does it compare with the Transformer and other modern RNN variants?\n\n2. Eq. 3 seems to only hold for $\\alpha=1$. Is this intended?\n\n3. In Table 2, the input dimension of Mamba2 is said to be $2D/H$. Should it be $2D$?\n\n4. Appendix A.1 is empty. Is this intended?\n\n5. Based on Table 1, the difference between the state transition equations of Gated DeltaNet and Comba seems to be the parametrization of $\\alpha_t$ and $\\beta_t$. In this case, why is Comba sufficiently faster than Gated DeltaNet? Could you apply the method in Sec. 3.2 to Gated DeltaNet similarly so that only one matrix inverse is computed?", "limitations": "Yes: the authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Sec. 2 and Table. 1 provide an informative overview of the development of linear and nonlinear RNNs and the motivations behind these designs.\n\n2. The effectiveness of SPLR and output correction are verified through extensive ablation studies; several variants of Comba are explored to justify the design decisions.\n\n3. Open access to the code is provided.\n\nWeaknesses:\n\n1. It is claimed in the title and Sec. 3 that the design of Comba is based on closed-loop control. However, it is quite difficult to identify the key components (i.e., controller, plant, error detector, feedback element, etc.) of a closed-loop control system from the equations in Table 3 and Eq. 5. In addition, it is difficult to see how closed-loop control helps to improve model performance, besides the general claim that negative feedback improves system adaptability (Line 148). I believe providing a block diagram (as commonly seen in the control literature) for each column of Table 3 would provide better insight to Comba.\n\n2. The output correction seems to be a feed-forward instead of a feedback path, because the output $\\boldsymbol{o}_t$ is still a linear transforma of the state $\\boldsymbol{S}_t$, just replacing the projection vector from $\\boldsymbol{q}_t$ to $\\boldsymbol{q}_t-d\\boldsymbol{k}_t$, and both $\\boldsymbol{q}_t$ & $\\boldsymbol{k}_t$ depend on the instantaneous input (maybe I misunderstood what ‘feedback’ exactly means in Comba – see W.1). In this case, it might be incorrect to claim that TTT/DeltaNet/RWKV-7 are not strictly ‘closed-loop systems’ (Line 155) and that Comba has two-stage feedback (Line 149).\n\n3. The initial value of $d$ (for output correction) seems to affect the model performance substantially, and the optimal choice of the initial value seems to heavily depend on the model size (50x difference in $d$ for 3x difference in model size). Investigating the model performance for other values of $d$ (between 0.02 and 1) would help practitioners in choosing this hyperparameter.\n\\end{enumerate}", "was_revised": true, "final_justification": "This paper introduces Comba, a bilinear RNN model with state and output correction to improve the model performance upon other recent RNN models, such as Mamba 2 and Gated-DeltaNet. Using chunkwise parallelization, Comba can be efficiently trained, with 40% speedup compared to Gated DeltaNet. The effectiveness of the proposed techniques is confirmed by experimental results on language and vision tasks.\n\nDuring the rebuttal period, the authors have addressed my initial concerns and questions, such as the usage of the term ‘closed-loop control’, the initial value of $d$, and the training memory costs. Given that the topic is highly relevant to the community and the paper provides an effective solution with convincing results, I will raise my score by 1.", "rebuttal": "We sincerely thank you for your rigorous and thoughtful feedback. We will begin by summarizing your concern, followed by a detailed response. Comba has already been open-sourced and is currently under evaluation by the broader LLM research community.\n\n# Q1: Training Cost.\nThank you for raising this point. We believe deep learning architectures are undergoing a fundamental shift, with increasing attention to Transformer alternatives. To be viable, new architectures must match Transformers in pretraining efficiency.\n\nComba achieves this through carefully designed kernels while remaining fully aligned with the standard Transformer parameterization—relying primarily on `q`, `k`, `v`, and output projections (**lines 173–176**). The additional terms ($\\alpha_t$, $\\beta_t$, $b$, $d$) are head-wise scalars with negligible memory overhead.\n\nAs detailed in our response to **Question 5**, Comba reduces memory usage by avoiding half of the matrix inversions required in Gated-DeltaNet. Besides, Comba's kernel minimizes intermediate storage, and compared to RWKV7, Comba achieves nearly **100% memory reduction** and up to **2× throughput** improvement. Below we show the GPU memory usage for a 340M model on a single A800-80GB GPU (totally 8×A800 GPUs with batch size 32):\n| A800 | FlashAttentionV2 | Comba |Gated-DeltaNet |RWKV7 |\n|-|-|-|-|-|\n|80GB| 62.26G  | 68.58G |  72.34G  | 132.52G (Out of memory)\n\n# Q2: The $\\alpha$ in equation 3.\nIn fact, DeltaNet has two versions. The first version was published at NeurIPS 2024 [1], and the second version, Gated-DeltaNet, was published at ICLR 2025 [2]. In line 103, we added a parenthesis around `Gated` to indicate that Equation 3 is explained using the basic DeltaNet, and therefore does not include the parameter $\\alpha$.\n\nIn fact, the gating formulation in Gated-DeltaNet is incorrect, as it cannot be explicitly written in the standard Stochastic Gradient Descent (SGD) form, while Comba adopts the correct formulation. This issue has also been pointed out in several recent blogs (in accordance with NeurIPS rebuttal guidelines, we cannot include external links):\n\n**Gated-DeltaNet**: $S_t = \\alpha_t S_{t-1} - \\beta_t \\nabla_S\\|\\| \\frac{1}{\\alpha_t}v_t-\\alpha_t S_{t-1}k_t \\|\\|^2$\n\n**Comba**: $S_t = \\alpha_tS_{t-1} - \\beta_t \\nabla_S\\|\\| v_t-S_{t-1}k_t \\|\\|^2$\n\n[1] Parallelizing Linear Transformers with the Delta Rule over Sequence Length\n\n[2] Gated delta networks: Improving mamba2 with delta rule\n\n\n# Q3: The Dimension for Mamba2.\nIn our paper, the dimensionality related to Mamba2 ($\\mathbf{u} \\in \\mathbb{R}^{2D/H}$) is correct. Because the State size in the table refers to the total dimension within one layer of the model, and the input value refers to the head dimension. Therefore, the dimension of the input value in Mamba2 carries a denominator of $H$, while the State size eliminates $H$.\n\nWe apologize for the confusion caused by our unclear notation. In the revised version of the paper, we will update `input value` to `input value per head` and `State size` to `All State Size`. A discussion on the dimensionality of Mamba2 can also be found in Appendix A.1 of another paper [3].\n\n[3] Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling\n\n# Q4: The Appendix A.1.\nWe apologize for this oversight. In fact, the paper does not contain an Appendix A.1. This is because we have already provided a comprehensive review of the development of linear RNNs in the Introduction and Section 2. The heading for Appendix A.1 was mistakenly left in the submission version of the paper, and we will remove it in the revised version.\n\n# Q5: Why is Comba faster than Gated-DeltaNet\n\nWe believe the primary contribution of Comba lies in its direct acceleration of DeltaNet-style operators. Specifically, Comba achieves up to a **46% speedup** compared to Gated-DeltaNet in forward propagation, which is of high practical value. Comba has already been open-sourced and supports verification. This performance gain primarily originates from the structural difference in how Comba constructs the **WY representation** (Equation 7), which leads to the **elimination of one matrix inversion** in the UT transformation process (Equation 10). Specifically:\n\n**Gated-DeltaNet**: $M_1 = I-\\text{lower}(\\text{Diag}(\\beta)KK^T)^{-1}, ~~~~M_2 = I-\\text{lower}(\\text{Diag}(\\beta)(\\Gamma\\odot KK^T))^{-1},~$\nwhere $\\Gamma_{ij}=\\prod_{t=1}^j \\alpha_t/\\prod_{t=1}^i \\alpha_t$.\n\n**Comba**: $M = I-\\text{lower}(\\text{Diag}(\\beta)(\\Gamma\\odot KK^T))^{-1},~$\nwhere $\\Gamma_{ij}=\\prod_{t=1}^{j-1} \\alpha_t/\\prod_{t=1}^i \\alpha_t$ **(The numerator contains a shift)**.\n\n\nMatrix inversion is a major performance bottleneck for kernel operators. In models like Comba and Gated-DeltaNet, the matrices to be inverted are masked with lower-triangular patterns, breaking symmetry and preventing the use of the Woodbury identity to reduce complexity to $\\mathcal{O}(n^2)$. As a result, inversion typically relies on Gaussian elimination with $\\mathcal{O}(n^3)$ complexity. By eliminating one matrix inversion, Comba achieves substantial speed gains.\n\nImportantly, this benefit can be extended to other DeltaNet-style models. Last month, inspired by Comba, the implementation of the open-sourced Gated-DeltaNet was updated: it now first derives a Comba-like formulation and then computes using the Comba operator. **Comba is, to our knowledge, the first model to introduce this operator structure**. This contribution was not emphasized in our original submission but will be further clarified in the revised version.\n\n# W1-W2: The Closed-loop Control in Comba.\n\nThank you for your valuable feedback. Comba adopts the fundamental definition of closed-loop systems as a **design intuition**. As stated on Wikipedia [8]:\n\n> *A closed-loop controller is a control loop which incorporates feedback. A closed-loop controller uses feedback to control states or outputs of a dynamical system.*\n\nFollowing this definition, consider the standard RNN update: $S_t = S_{t-1} + v_t k_t^T$. Here, $S_{t-1}$ passively receives the update and does not influence the update behavior through feedback. Therefore, models like **Mamba2** can be viewed as **open-loop systems**.\n\nIn contrast, **Comba** introduces two explicit forms of feedback:\n\n- **State correction:** $S_t = S_{t-1} (\\alpha_t - k_t k_t^T) + v_t k_t^T$\n\n- **Output correction:** $o_t = S_t q_t + d(v_t - S_t k_t) = S_t(q_t - dk_t)$ (Residual connection $d * v_t$ is omitted, as is also done in Mamba2.)\n\nThese mechanisms correspond directly to the two parts of the closed-loop definition: feedback for both **states** and **outputs**. In comparison, models like **RWKV-7** and **DeltaNet** only implement feedback at the **state** level and thus cannot be considered end-to-end closed-loop systems. Empirically, we observe that introducing output correction significantly boosts model performance. Notably, this mechanism can be implemented with a **single line of code** `q_t - d*k_t`, and can be easily integrated into many RNN variants. \n\nSince the design of Comba is primarily intuition-driven, it is difficult to establish a one-to-one correspondence between its components and those of classical control systems. Moreover, if the output correction were to directly influence the state $\\mathbf{S}_t$, it would introduce additional nonlinearity that compromises Comba's ability to support hardware-efficient **blockwise parallel computation**—a critical requirement for scaling to large language models.\n\nDue to space limitations, we kindly refer you to our response to reviewer **ScfA** (second to last), specifically in the section titled **\"Q2: How does the feedback improve the performance?\"**, where we provide a comprehensive explanation of how the feedback mechanisms in Comba contribute to its success from multiple perspectives.\n\n[8] Wikipedia: Closed-loop controller\n\n# W3: Initial value for d.\n\nIn our initial experiments, we consistently observed that setting $d = 1$ resulted in the lowest loss. However, during downstream task evaluation with the 340M-scale model, we occasionally found that $d = 0.02$ yielded better results. Out of scientific rigor, we reported this observation faithfully in the original manuscript.\n\nDuring the review period, we conducted a more detailed investigation into the parameterization of $d$, and we would like to share our latest findings here. The new results confirm that $d = 1$ provides the best performance overall. We believe the earlier deviation was due to the relatively small size of the 340M model, where certain patterns are more prone to noise or instability. \n| Acc. in Commonsense Reasoning|340M/15B (seed=42)| 340M/15B (seed=2025)| 760M/50B | 1.3B/100B |\n|-|-|-|-|-|\n| `w/o. q - dk`|42.45|42.98|45.82|49.72|\n| `d=0.02`|43.36|43.40|45.66|51.78|\n| `d=0.5`|**43.71**|43.37|46.39|51.62|\n| `d=1`|43.26|**43.81**|**46.70**|**51.91**|\n\nBesides, we find that integrating this mechanism into other modern RNNs, such as Gated-DeltaNet and GLA, led to notable loss reductions. (1.3B/100B Gated-DeltaNet dropped from 2.213 to 2.175, and 1.3B/100B GLA dropped from 2.232 to 2.204)\n> We will incorporate all the details into the revised version of the manuscript. If your concerns have been addressed, we would be sincerely grateful if you would consider raising the score. Should you have any further questions, we would be more than happy to respond."}, {"review_id": "ZQaqhJinw6", "reviewer": "Reviewer_ScfA", "summary": "This paper introduces a new nonlinear recurrent neural network (RNN) model called Comba, which is inspired by control theory. The authors review recent nonlinear RNNs that use Delta learning rules to improve memory control. Unlike earlier models, Comba uses a scalar-plus-low-rank (SPLR) state update and adds feedback both to the state and output, forming a closed-loop system. It is also designed to be hardware-efficient by supporting chunk-wise parallel computation. The authors test Comba on various language and vision tasks, showing it has better performance and faster training compared to other models. They also explore hybrid designs and provide detailed experiments and analysis.", "questions": "1. I find the discussion on whether models like Mamba2 are considered open-loop or closed-loop somewhat confusing. From a general perspective, most RNNs, including Mamba2, are inherently “closed-loop” in the sense that the current state S_t depends on the previous state S_{t-1}, which serves as a form of feedback. This contrasts with autoregressive models like Transformers (with KV cache), which are more “open-loop” because they do not incorporate such recurrent state feedback in the same way. However, in Section 3, the paper seems to classify Mamba2 as an open-loop control system. Does this mean that the term “closed-loop” in this work specifically refers to the inclusion of negative feedback mechanisms, such as corrective terms that subtract projections from previous states (e.g., v_t - P_t(S_{t-1}))? If so, it would be helpful to explicitly distinguish this interpretation from the broader control-theoretic or RNN-centric notions of feedback.\n\n2.The experimental results convincingly demonstrate that the proposed model achieves strong performance. However, I still find it unclear whether the negative feedback mechanism is truly the key contributing factor. Could the authors provide more explanation, either theoretical or empirical, on why negative feedback helps in this setting? For instance, does it improve stability, memory precision, or long-range dependency handling in a measurable way? A deeper discussion or ablation focusing specifically on the feedback terms would be very helpful.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces Comba, a new nonlinear RNN architecture that incorporates closed-loop control by combining state feedback and output correction.\n2. The authors provide an optimized chunk-wise parallel implementation in Triton, achieving a good speed improvement in forward pass. This makes the method more practical for large-scale training.\n3. Comba is rigorously evaluated across multiple benchmarks, including language modeling, commonsense reasoning, long-context understanding (LongBench), and etc.. Results consistently show competitive or superior performance.\n4. The work nicely bridges the gap between SSMs, gated linear attention, and Delta-rule-based updates. It also provides a unified perspective on “Nonlinear RNNs” and positions Comba as a foundation model candidate.\n5. The authors conduct comprehensive ablations on architecture variants (e.g., SPLR vs. IPLR), feedback mechanisms, and output correction terms, giving credibility to their design choices.\n\nWeaknesses:\n1. The paper is not easy to follow for readers who are not already familiar with recent state-space models like Mamba, or with LLM-centric efficient RNNs. \n2. The term “Nonlinear RNN” in the title may be confusing for researchers coming from classical RNN/statistical learning backgrounds, where RNNs are already nonlinear in most cases by design. It might benefit from a subtitle or clarification.\n3. The notation is dense and not well-initialized early in the paper. Symbols  are used heavily but not clearly introduced or contextualized for new readers. This affects the clarity of theoretical sections.", "was_revised": true, "final_justification": "I appreciate the authors’ response and will keep my original score.", "rebuttal": "We sincerely thank you for your appreciation of our work. Below, we provide further clarification regarding the concerns you raised.\n\n# W1: Hard to understand for RNN beginners.\nThank you for bringing this to our attention. In fact, during the writing of this paper, we made a concerted effort to outline the development of modern RNN architectures as thoroughly as possible in the Preliminaries section. We also sought feedback from several non-expert readers to ensure accessibility. However, linear-time sequence modeling architectures often rely on substantial prior knowledge, such as orthogonal polynomial projections, neurobiology, and algebra, which poses a significant challenge for beginners to grasp quickly. We sincerely appreciate your suggestion, and in the revised version, we will include an extended Related Work section in the appendix to help readers better understand the evolution of modern RNNs.\n\n# W2: The concept of Nonlinear RNNs.\n\nThank you for your valuable feedback. We also became aware of this issue during the review process, and have accordingly updated the term `Nonlinear RNN` to `Bilinear RNN` in the revised version of the paper. This change only involves replacing the terminology, and does not affect any other content in the manuscript. (Due to NeurIPS rebuttal policies, we are unable to update the submitted version.)\n\nWe believe that `Bilinear RNN` is a more appropriate designation. Comba is neither strictly nonlinear in the way that LSTMs are, where nonlinearity is applied directly to the hidden state, nor is it merely a linear key-value memory mechanism as in linear attention or state space models. Instead, Comba introduces a product between the key $\\mathbf{k}$ and the hidden state $\\mathbf{S}$, which corresponds to a `Bilinear System` in control theory [1, 2]. Such systems are widely used for modeling biological populations and physical dynamics, and are also regarded as one of the most expressive controllable system classes.\n\n[1] Bilinear systems: An appealing class of ”nearly linear” systems in theory and applications. \n\n[2] Expectation-maximization algorithm for bilinear state-space models with time-varying delays under non-gaussian noise.\n\n# W3: The notation is unclear.\nThank you for highlighting this important point. The notation used in Comba is aligned with many prior works on linear time-series modeling [4,5,6,7]. Additionally, we provided a clarification of the symbols in the footnote at the bottom of Table 1. \n\nWe apologize for any confusion caused by the lack of explicit notation definitions. In the revised version of the paper, we will include a dedicated **Notation** section before the **Preliminaries** to explain the meaning of symbols used throughout the paper, making the content more accessible for beginners.\n\n[4] Gated slot attention for efficient linear-time sequence modeling\n\n[5] Gated delta networks: Improving mamba2 with delta rule\n\n[6] Gated linear attention transformers with hardware-efficient training\n\n[7] Understanding Transformer from the Perspective of Associative Memory\n\n# Q1: The Concept of Closed-loop System in Comba.\n\nThank you for your suggestion. In the revised version of the paper, we will clarify the concept of **closed-loop systems**, and confirm that the term \"feedback\" in Comba refers specifically to **negative feedback**.\n\nIn this work, we adopt the fundamental definition of closed-loop systems as a design intuition. As stated on Wikipedia [8]:\n\n> *A closed-loop controller is a control loop which incorporates feedback. A closed-loop controller uses feedback to control states or outputs of a dynamical system.*\n\nFollowing this definition, consider the standard RNN update: $S_t = S_{t-1} + v_t k_t^T$. Here, $S_{t-1}$ passively receives the update and does not influence the update behavior through feedback. Therefore, models like **Mamba2** can be viewed as **open-loop systems**.\n\nIn contrast, **Comba** introduces two explicit forms of feedback:\n\n- **State correction:** $S_t = S_{t-1} (\\alpha_t - k_t k_t^T) + v_t k_t^T$\n\n- **Output correction:** $o_t = S_t q_t + d(v_t - S_t k_t) = S_t(q_t - dk_t)$ (Residual connection $d * v_t$ is omitted, as is also done in Mamba2.)\n\nThese mechanisms correspond directly to the two parts of the closed-loop definition: feedback for both **states** and **outputs**. In comparison, models like **RWKV-7** and **DeltaNet** only implement feedback at the **state** level and thus cannot be considered end-to-end closed-loop systems. Empirically, we observe that introducing output correction significantly boosts model performance. Notably, this mechanism can be implemented with a **single line of code** `q_t - d*k_t`, and can be easily integrated into many RNN variants. For example, adding this to Gated-DeltaNet and GLA yielded consistent improvements:\n\n- Gated-DeltaNet (1.3B/100B): loss reduced from 2.213 → 2.175\n\n- GLA (1.3B/100B): loss reduced from 2.232 → 2.204\n\nTo avoid potential ambiguity, we will explicitly address these clarifications in the revised manuscript.\n\n\n[8] Wikipedia: Closed-loop controller\n\n\n\n\n# Q2: How does feedback improve performance?\n\nComba incorporates two forms of feedback (as shown in Equation 5): **state correction**, represented by $(\\alpha_t - \\mathbf{k}_t\\mathbf{k}_t^{T})$, and **output correction**, represented by $(\\mathbf{q}_t - d\\mathbf{k}_t)$. We provide a detailed explanation of both mechanisms below. The effectiveness of **negative feedback** has been well established in the control systems literature. In this work, our main contribution is to reinterpret and implement such feedback mechanisms within the context of **deep learning RNN architectures**:\n\n## State Correction.\n\n(a). Comba can be reformulated in the following equivalent form: $S_t = \\alpha_t S_{t-1} + \\beta_t(v_t -  S_{t-1} k_t)k_t^{T}$\n\nHere, the term $(v_t - S_{t-1} k_t)$ can be interpreted as a **new value**, which replaces the original $v_t$ in the outer product update. The intuition behind this formulation is as follows:\n\nIn modern RNN architectures, the hidden state $S$ is typically updated via an outer product between the key and value vectors. When querying the memory with $k_t$, the model retrieves the most relevant historical value using $S_{t-1}k_t$. Since the state matrix $S$ has a fixed size, its memory capacity is inherently limited. By using the **difference** between the new value and the retrieved one, i.e., $(v_t - S_{t-1}k_t)$, the model effectively learns **only the information that was not already stored**. This can be seen as a form of memory-efficient update: the smaller the difference, the more redundant the current value is with respect to past knowledge. Thus, this correction mechanism equips the model with **stronger memory management**, promoting efficient utilization of limited hidden state capacity.\n\n\n(b). The state correction matrix $(\\alpha_t - b \\cdot k_tk_t^{T})$ can be interpreted as a mirror transformation. As illustrated in Figure 1, its geometric meaning is to reflect each column of the memory matrix $\\mathbf{S}_{t-1}$ across a hyperplane whose normal is defined by the vector $\\mathbf{k}_t$. Intuitively, whenever the model encounters new information, this transformation—scaled by $\\beta_t$—suppresses the component of the memory update that lies in the direction of $\\mathbf{k}_t$, preserving only the orthogonal component, i.e., the part that represents *novel* or *previously unseen* knowledge.\n\n\n(c). Comba can be rewritten in an explicit Stochastic Gradient Descent (SGD) form:\n\n$$\nS_t = \\alpha_t S_{t-1} - \\beta_t \\nabla_S\\left \\|\\| v_t - S_{t-1} k_t \\right \\|\\|^2\n$$\n\nThis formulation implies that the model not only updates its parameters during training, but also continuously adapts its memory state (as a special parameter) during inference. This behavior reflects the core idea of **negative feedback**: the system adjusts itself in real-time to better align with external inputs, enabling continual adaptation to changing environments.\n\n\n\n## Output Correction\n\nWhile prior RNN research has largely focused on improving gating mechanisms or optimizing state updates via key-value operations, few models have explicitly modified the output pathway (i.e., the query @ State). We think there appears to be a potential connection between Comba and MesaNet [9] (a recent SOTA model from Google), as MesaNet also performs query correction in the output stage by solving a closed-form recursive least squares problem, resulting in $q_t=(H_t + \\Lambda)^{-1}q_t$. Notably, the correction in Comba is much simpler **with a single line of code** `q_t - d*k_t`, and is applicable to nearly all RNN variants.\n\n[9] MesaNet: Sequence Modeling by Locally Optimal Test-Time Training\n\n> We hope our responses address your concerns, and we warmly welcome continued discussion. Thank you again, and we wish you all the best."}, {"review_id": "WEMrhnKYV9", "reviewer": "Reviewer_npnS", "summary": "## Summary\n\nIn this article, the authors provide a detailed literature review on recursive models for efficient processing of sequential data. This synthesis highlights the strengths and weaknesses of each model and thus allows them to propose a novel RNN architecture that aims to incorporate dependencies on the previous state and the output while remaining efficient during inference. In addition to the detailed literature summary, the authors support their new architecture with experiments that show that the new model is competitive with the state of the art, and in a number of tasks outperforms the state of the art.", "questions": "## Questions\nAnglais\n\nDo you have any idea if it is theoretically possible to show the expressiveness of your model and if possible place it in the ecosystem of existing models?", "limitations": "Due to the unavailability of these state-of-the-art models, the authors were unable to compare their models to the latest state-of-the-art models. Due to infrastructure limitations, they were unable to conduct very large-scale experiments.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The article is very well written, well -illustrated and clear.\n- The authors have a deep knowledge of literature and know exactly how their work is positioned.\n- The methodology of numérical experiments is rigorous.\n\n## Weaknesses\n- There is a lack of theory to support the claims. Validations are purely empirical.\n- As the authors themselves said, due to infrastructure limitations, they were unable to conduct very large-scale experiments.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your appreciation of our work. Below, we provide further clarification regarding the two minor concerns you raised. Lastly, we would also like to share our perspective on Comba's positioning within deep learning architectures and its future directions.\n\n# Q1: The lack of theory to support the claims.\n\nThank you for raising this point. We think that the development of modern deep learning architectures has been largely driven by empirical insights and intuition. For example, the attention mechanism in the original Transformer was introduced empirically, with theoretical analyses only emerging in subsequent years. Similarly, the early development of structured state-space models (SSMs) relied on theory-driven parameter initialization, such as using HiPPO-based matrices. However, these were eventually outperformed by models like Mamba, which are entirely data-driven.\n\nWe acknowledge that theory plays a crucial role in understanding **why** a model or mechanism works. Nonetheless, practical deployment often involves complexities that exceed the reach of current theoretical tools. In the revised version of the paper, we will provide a theoretical characterization of Comba's expressiveness. In particular, from the perspective of computational circuit complexity [1,2,3,4], we show that Comba belongs to the **NC¹** class, which strictly exceeds the **TC⁰** class associated with Transformer-based architectures.\n\n\n# Q2: Very large-scale experiments.\n\nSince Comba has open-sourced in Flash-Linear-Attention library, it has attracted growing interest from the community. We are now collaborating with a technology company to conduct large-scale experiments to further evaluate Comba's performance, and we warmly welcome you to try out Comba.\n\n\n# Q3: The Position of Comba in the ecosystem of deep learning architectures.\n\n| Expressive/Efficient | `O(T)` | `O(Tlog T)` | `O(T^2)` |\n|------------|--------|------------|--------|\n| P      | LSTM |       --     |    --    |\n| AC¹     |    --      |      --      |    --    |\n| NC¹     | Comba, DeltaNet, RWKV7 | Loglinear Comba | Deltaformer, 2‑Simplicial Attention |\n| TC⁰     | Mamba, MesaNet, RWKV6   | Loglinear Mamba | Transformer |\n|\n\nWe believe that deep learning architecture is undergoing a fundamental shift. As the limitations of the Transformer architecture become more widely recognized, a new wave of alternative designs has emerged. When evaluating the quality of a model, two key aspects are typically considered: **expressive power** and **computational efficiency**. As shown in the table, existing architectures can be roughly categorized by their time complexity into **linear**, **log-linear**, and **quadratic** classes (represented on the horizontal axis). \n\nRegarding expressive power, **circuit complexity** has become a common theoretical framework, where one studies the ability of a circuit to simulate the behavior of a neural network. The standard hierarchy of circuit complexity classes is as follows:\n\n$$\n\\mathsf{P} > \\mathsf{TC}^n > \\mathsf{AC}^n > \\mathsf{NC}^n > \\mathsf{TC}^{n-1} > \\mathsf{AC}^{n-1} > \\mathsf{NC}^{n-1} > \\cdots > \\mathsf{TC}^0 > \\mathsf{AC}^0 > \\mathsf{NC}^0\n$$\n\nHere, the superscript $n$ denotes the logarithmic depth of the computation paths, while $\\mathsf{NC}$, $\\mathsf{AC}$, and $\\mathsf{TC}$ refer to distinct classes of circuit complexity. $\\mathsf{P}$ denotes the class of problems solvable in polynomial time, which includes the vast majority of real-world tasks.\n\nRecent studies have shown that models with limited depth and finite-precision arithmetic—such as Transformers and Mamba [5]—are provably within the $\\mathsf{TC}^0$ class. In contrast, models like **Comba**, **RWKV-7** [4], and **DeltaNet** [6] can theoretically achieve $\\mathsf{NC}^1$-level complexity, which provides strictly greater expressive power. Among these, **Comba** is the most expressive due to its dual-stage feedback design. Although **LSTM** architectures apply nonlinear activation functions to the hidden state, thereby achieving expressive power equivalent to the complexity class **P**, they inherently lack the ability to perform parallel computation across time steps. This sequential dependency renders them unsuitable for large-scale language model pretraining, where parallelism is crucial for computational efficiency. As a result, despite their theoretical expressiveness, LSTMs have limited practical relevance in modern large-scale sequence modeling tasks.\n\n\n\nMoreover, recent efforts in optimizing quadratic-time models have also elevated their theoretical capacity to the $\\mathsf{NC}^1$ class, such as Deltaformer [1]. We believe that a promising future direction is to maintain the **efficiency of RNN-style architectures** while further advancing their **expressiveness toward the $\\mathsf{AC}^1$ class**.\n\n> We hope our responses address your concerns, and we warmly welcome continued discussion. Thank you again, and we wish you all the best.\n\n[1] Understanding Transformer from the Perspective of Associative Memory\n\n[2] The Illusion of State in State-Space Models\n\n[3] The Parallelism Tradeoff: Limitations of Log-Precision Transformers\n\n[4] Rwkv-7\" goose\" with expressive dynamic state evolution\n\n[5] Mamba: Linear-time sequence modeling with selective state spaces\n\n[6] Parallelizing Linear Transformers with the Delta Rule over Sequence Length"}]

Author rebuttal:

Meta-review: This paper introduces Comba, a new nonlinear RNN architecture inspired by control theory. Reviewers agreed that the paper is very well written, and well motivated by prior work. This paper leverages the recent developments of nonlinear RNNs and demonstrates some of their advantages. The empirical results are extensive, covering language modeling, reasoning, long-context benchmarks, and vision tasks, with Comba showing consistent improvements. The ablation studies are detailed. And the release of code enables reproducibility and transparency.  Some minor weaknesses were noted: the control-theoretic framing could be clarified (e.g., with block diagrams); some terminology may be confusing; some of the baselines should be better tuned for fairness. All of these issues can be addressed in the camera ready version.

Overall, this is a strong paper that provides both methodological novelty and convincing empirical evidence. Investigating nonlinear RNNs provides a fresh perspective and new impulses, which I think is important. This paper is a strong accept.
Decision: Accept (spotlight)