Mamba-3: Improved Sequence Modeling using State Space Principles
Authors: Aakash Lahoti
Kevin Li
Berlin Chen
Caitlin Wang
Aviv Bick
J Zico Kolter
Tri Dao
Albert Gu
Venue: ICLR 2026 Oral (2026)
Keywords: State Space Models
Mamba
LLMs
Subquadratic Models
Primary area: foundation or frontier models, including LLMs
TL;DR: Mamba-3, an inference-first SSM that pushes on core SSM principles: improved discretization for better quality, complex dynamics for new capabilities, and MIMO updates for efficient inference.

Abstract: Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While current Transformer models deliver strong quality, their quadratic compute and linear memory requirements make inference expensive. This has spurred the development of sub-quadratic models with reduced compute and constant memory requirements.
However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice.
Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule enabling richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation that improves model performance without increasing decode latency.
Together with architectural refinements, Mamba-3 achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with the MIMO variant further improving accuracy by an additional 1.2 points, for a total gain of 1.8 points.
Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half the state size. These results demonstrate that Mamba-3 advances the performance–efficiency frontier.

Published as a conference paper at ICLR 2026
MAMBA-3: IMPROVED SEQUENCE MODELING USING STATE SPACE PRINCIPLES
Aakash Lahoti'
Kevin Y. Li
Berlin Chen
Caitlin Wang??
Aviv Bick'
J. Zico Kolter'
Tri 2
Albert Gu
Camegie Mellon University Princeton University Together AI
*Cartesia Al
(alahoti, ky12, abick, zkolter, agu) edu
[bc2188, caitlinwang, tridao/@princetton.c
ABSTRACT
Scaling inference-time compute has emerged as an important driver of L.M performance, making inference efficiency a central focus of model design alongside model quality. While the current Transformer models deliver strong model quality, their quadratic compute and linear memory makes inference expensive. This has spurred the development of sub-quadratic models with reduced linear compute and constant memory requirements. However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains
in practice. Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (I) more expressive recurrence derived from SSM discretization, (2) a complex- valued state update rule that enables richer state tracking, and (3) multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency. Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's 's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's 's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficien frontier.
INTRODUCTION
Test-time compute has emerged as a key driver of progress in LLMs, with techniques like chain-of of-thhuuht reasoning and iterative refinement demonstrating that inference- time scaling can unlock new capabilities (Wu et al., 2025; Snell et al., 2024). The rapid rise of parallel, agentic workflows has only intensified the need for efficient inference and deployment of such models (OpenAI, 2026; Anthropic, 2026). This paradigm shift makes inference efficiency (Kwon et al., 2023; Li et al., 2024) paramount, as the impact of Al systems depends critically on their ability perform large-scale inference during deployment. Model architecture design plays a fundamental role in determining inference efficiency, as architectural choices directly dictate the computational and memory requirements during generation. While Transformer- based models ( (Vaswani et al., 2017) are the current industry standard, they are fundamentally bottlenecked by linearly increasing memory demands through the KV cache and quadratically increasing compute requirements through the self-attention mechanism. These drawbacks have motivated recent lines of work on sub- cub-quadratic models, c.g., state space models (SSMs) and linear attention, which retain constant memory and linear compute while attaining comparable or better performance than their Transforeer counterparts. These models have achieved widespread adoption, with layers such as Mamba-2 (Dao & Gu, 2024) and Gated DeltaNet (GDN) (Schlag al., 2021; Yang et al., 2025d) incorporated into large-scale hybrid models that match the performance of pure Transformer alternatiees with much higher efficiency (NVIDIA et al., 2025; Yang et al., 2025a; Kimi Team et al., 2025; Tencent Hunyuan Team et al., 2025).
Despite the success of linear models, significant progress remains in improving their performance, in particular on advancing the Pareto frontier between model quality and inference efficiency. For example, Mamba-2 was developed to improve training speed over Mamha-1 (Gu & Dao, 2024), by sacrificing some expressivity
"quul contribution.

---

Published as a conference paper at ICLR 2026
and thus performing worse for pnterence-matccee models. In addition, they have been shown to lack certain capabilitiie, such as poor satte-trcking abilities, e.g., simply determining parity of bit sequences (Grazzi et al., 2025; Sarrof et al., 2024). Finally, despite these sub-quadratic models being prized for theoretically efficient inference and thus their widespread adoption, their inference algorithms are not hardware efficient. In particular, hecause these algorithms were developed from training perspective, their decoding phase has low arithmetic intensity (the ratio of FLOPs to memory traffic), resulting in large portions of hardware remaining idle.
To develop more performant models from an inference- paradigm, we introduce three core SSM-centric methodological changes to Mamba-2.
Exponential- Trapeziodaall Discretization. We provide a simple technique for discretizing vame-ngrying selective SSMs. Through our framework, we can derive several new discretization methods. One of our instantiations, referred to as "exponential- formalizes Mamba-1 and Mamba-2's heuristic discretization that previously lacked theoretical justification. Our new aeponeentilltrapezoiddee expressive, generalization of "exponential-Faler"" where the recurrence can be expanded to reveal an implicit convolution applied on the SSM input. Combined with explicit B, C biases, Mamba-3 empirically replace the short causal convolution in language model architectures, which was previously hypothesized be essential for recurrent models.
instantiation is more
Complex-valued State Space Model. By viewing the underlying SSM of Mamba-3 as complex-valued, enable more expressive state update than Mamba-2's. This change in update rule, designed to be lightweight for training and inference, overcomes the lack of state-tracking ability common in many current data-dependent rotary embedding and
linear models. Our complex-valuee update rule is equivalent can be efficiently computed (Su et al., 2023), and we empirically demonstrate its ability to solve synthetic tasks outside the capabilities of prior linear models.
Multi-Input, Multi-Output (MIMO) SSM. To improve FLOP efficiency during decoding, we switch from an outer-product prase state update to matrix-multilliatiinn cnultipliiation--aaseee based state update. From the view of the signal processing foundations of SSMs, such a transition exactly coincides with the generalization from a single-input single-output (SISO) sequence dynamic to multiple-input multiple-output (MIMO) one. Here, we find that MIMO is particularly suitable for inference, as the extra expressivity enables more computation during the memory- bound state update during decoding, without increasing the state size and compromising speed.
Put together, these improvements form the core of our Mamba-3 layer. Methodologically, we note that these all arise naturally from SSM-centric perspective but are not immediate from other popular viewpoints of modern linear layers such as linear attention or test-time regression; we discuss these connections further in Section A. Empirically, we validate our new model's abilities and capabilities on suite of synthetic state- tracking and manguage-modeling tasks.
Better Quality. At 1.5B scale, Mamba-3 (MIMO) improves average downstream language modeling accuracy by +2.2 over Transformers, +1.9 points over Mamba-2, and +1.8 over GDN. Mamba-3 (SISO) improves over the next best model, GDN, by +0.6 points. Furthermore, across state size experiments, Mamba-3 (MIO) with state size 64 matches the perplexity of Mamba-2 with state size 128, effectively achieving the same language modeling performance with half the latency.
New Capabilities. Mamba-3's complexification of the SSM state enables it to solve synthetic state-tracking tasks that Mamba-2 cannot. We empirically demonstrate that the efficient RoPE-like calculation is able to near perfectly solve arithmetic tasks, while Mamba-3 without RoPE and Mamba-2 perform not better than random guessing.
Inference Efficiency. Mamba-3 (MIMO) improves hardware utilization. It increases decoding FLOPs by up to 4x relative to Mamba-2 at fixed state size, while maintaining similar wall-clock decode latency, and simultaneously improving perplexity and downstream performance. We release fast training and inference kernels for Mamba-3.
Mamba-3 (SISO) improves quality and capability over prior linear models, and Mamba-3 (MIMO) further improves performance over Mamba-3 (SISO) and other strong baselines while matching inference speed with Mamba-2. Both of our Mambu-3 variants advance the performance-latene Pareto frontier through their strong modeling capabilities and hardware-efficient design.
2

---

Published as a conference paper at ICLR 2026
2
PRELIMINARIES
2.1 NOTATION
Scalars are denoted by plain-text letters (e.g. v.y). Tensors, including vectors and matrices, are denoted by bold letters (c.g.. h,C). The shape of the tensor can be inferred from the context. We denote the input sequence length as T, the model dimension as D, and the SSM state size as N. For time indices, we use subscripts (e.g., 32 for the input at time t). The Hadamard product between two tensors is denoted by c. For a vector size v E Rd, we denote Diag( v) e IR Idddd as the diagonal matrix with the vector v as the diagonal, and for products of scalars acTOSS time steps, we use the notation ot....=a. III
2.2
SSM PRELIMINARIES
State Space Models (SSMs) describe continuous-time linear dynamics via
= -((= h(t), where h(() E R. is the hidden state, sx(1) E IR the input, and E ((I) We will occasionally refer to A(t) as the state state-transition and B (tt)x(t) as the state-input; this also extends to their discretized counterparts. For discrete sequences with step size A,, Mamba-1 and Mamba-2 discretized the system to the following recurrence
hu(t)=
h e A
h4 B,B0t
-. h..
Mamba-2's 2's Parameterization. The core of the Mamba-2 layer (Dao Gu, 2024) is deta-dependent and sardware-efficient SSM. Both the state-transition and state-input are made data-dependent through the projection of A, e R200 and B,C' R. from the current token. By parameterizing the sate-transition A, as scalar times identity, the SSM recurrence can be efficiently computed with the matrix multiplication tensor cores of GPUs. Defining 04 A. (0,1) and 7 = De the update becomes y/c C h4
The data- data-dependent state-transition Ct controls the memory horizon of each SSM within the layer. A, in par- ticular modulates both the state-trame tate-trranition and state-input: larger A forgets faster and up-weights the current token, while smaller A, retains the hidden state with minimal contributions from the current token.
Remark Mamba-3 switches to data- data-dependent A to keep all SSM parameters data-dependent. This change does not change the empirical performance of the model.
2.3 STRUCTURED MASKED REPRESENTATION AND STATE SPACE DUALITY
Mamba-2 showed that a large class of SSMs admit a matrix form that vectorizes the time-step recurrence. Through the state space duality (SSD) framework. recurrent SSMs can be represented within parallel form that incorporates an element-wip mask to model the state- tratsttranitionn decay. SSD provides general framework for duality between linear recurrence and parallelizable (matrix- (matrrx-multiplicatee multipliccation- based) computational forms
Y (=LLCB
(2)
where L RTTT is a structured mask, B B.CERTxN, B XERTXD are the inputs to the SSM and Y Y T its output. Different structures on L give rise to various instantiations of SSD. Equation (2) also draws a general connection between recurrence and attention, by setting Q Q==C. C, k= B, V == X and viewing L as data-dependent mask. In fact, the simplest case of SSD is (causal) linear attention (Katharopoulos et al., 2020), where L is the causal triangular mask. Mamba-2 is generalization where Lij- =0 for ون Oi....7j for i 2 j, where the terms at,"t are from equation (1) and on 11. In Section B. we show that Mamba-3 generalization of Mamba-2 with a more expressive L, and hence also an instance SSD.
3 METHODOLOGY
We introduce Mamba-3, with three innovations rooted in classical state space "expoyy-"expoonntaaltrrraar discretization for more expressive dynamics, complex- valued SSM for state tracking, and multi-input multi-output (MIMO) for improved performance and hardware utilization for inference address the quality, capability, and efficiency issues of current linear models.
3.1
DISCRETIZATION
Structured SSMs are naturally defined as continuous pattinuous-tmme dynamical systems that map input functions, rtt) R. to output functions, (t) E R. for time 0. The underlying system is first-order ordinary differential equation (ODE) for the state ht()) and a dot- product readout for the output y(t). In sequence
In this paper, B, represents the continuous parameter, whereas Mamba-2, B, represents the discretized parameter which equivalent 7/1 B., folding in the term.

---

Published as conference paper at ICLR 2026
(dilte-)) ((r)x(r)dt
P. Y1
M
85a
1/2
f
Figure 1: Left: The structured mask induced by the exponentialtrapeezodaal rule (Section 3.1) is product of the decay and two-band convolutional mask. Right: Euler (hold endpoint) versus Trapezoidal (average endpoints) integral approximation.
modeling, since the data is observed discrete time steps, it requires applying discretization step to the SSM to transform its continuous-time dynamics into a discrete recurrence.
Discretization methods are well-studied in classical control theory with several canonical formulas used in ear- lier SSM works deep learning (Gu et al., 2022a;b; Smith et al., 2023). These mechanisms were traditionally stated and applied to linear-time invariant (LTT) systems such S4, and their derivations do not directly apply to linear-time varying (LTV) systems such as Mamba- 1/2. Additionally, while Mamba-l adapted the zero-ord zero-order hold (ZOH) method to LTV systems, the complexity associated with selective SSMs prompted the use of heuristic approxinations without theoretical justification and did not correspond to any established discretiza- tion technique. In the following subsection, formalize the previous heuristics used in current LTV SSMs through our discretization framework and utilize it to propose a more expressive discretization scheme.
3.1.1 OVERVIEW OF EXPONENTIAL-ADIUSTIE
DISCRETIZATION
We introduce a simple derivation that leads to a class of new discretization methods for LTV state space models. The method can be instantiated in various ways; show that instantiation results in the heuristic used in Mamba- 1/2, thereby theoretically justifying it (exponential-Eulee We also introduce a more powerful discretization (exponential-rraezzzddaa) used in Mamba-3.
We begin with the observation that the closed form solution of the simple linear ODE, h'(1== Ah((), e())) where the dynamics are dominated by exponential scaling. We extend this analysis to the "adjusted system" that features a time-dependent transition scalar A(t) as well as a system input (t))x(t). Solving this.
h(T)) (J" A(s)ds h(Tr-))+ I" exp (f- A(s)ds B(r)a(r)dr Te-s
M
solved analyticaly
approximated
We approximate the first integral by using A(s)= = A(Ti). for sE 1T
h ())
exp((77-7AA)
T--1
which serves as the foundation for further discretization techniques in a TV setting. The full derivation is de- tailed in Proposition 5. We recover the prior Mamba discretizations from framework Section B.I.
Exponential- Trapezoodaa (Mamba-3). However, Euler's rule from previous Mambas provides only first-order approximation of the state-input integral: the local truncation error is O(A)). which accumulates across steps to yield a global error of O(d,) over the sequence. In contrast, we introduce a generalized trapezoidal rule, which provides a second-order accurate approximation of the integral, offering improved accuracy over Fuler's rule. Specifically, approximates the integral with a data-dependent, convex combination of both interval endpoints. This generalization extends the classical trapezoidal rule (Süli & Mayers, 2003), which simply averages the interval endpoints (Figure I).
Proposition (Generalized Trapezoidal Recurrence), Approximating the stateiinut integral Equation (10) by the general trapezoidal rule yields the recurrence,
h = A h
A...A.,BB...... B (
AA
(3)
= -0 3, B, B
(4)
where X E {00,1] is data-dependent scalar, == (1- =
'The actual Mamba implementation follows http:///ghhhh..mmmmmma.ccccsccaa

---

Published as a conference paper at ICLR 2026
Remark (Expressivity). The exponential apponentia--rapezzoiaal rule is generalization of (a) the classical trapezoid rule, recovered when X - 2. and (b) Mamba-2's is Euler's rule, recovered when d -y=1
Remark (Error Rate). This is second-order discretization with local truncation error O(4)) and global error (())) over the sequence under standard stability assumptions, provided that the trapezoidal parameter satisfies A 5 + (A+). However, our ablations indicate that not enforcing this constraint is the best for empirical performance. See Appendix B.3,B.4 for details.
Our discretization framework and its two instantiations, exponential- Bulerr and angooentialltrapezoiinni are, to the best of our knowledge, novel for structured SSMs in deep learning. Table 5 compares and summarizes the canonical and common discretization schemes for SSMs.
3.1.2 EXPONENTIAL CEPONENTIAL-TRAAEZODDOO TRAPEZOIDAL RECURRENCE AS AN IMPLICIT CONVOLUTION
Our generalized annooeentialtraaezziiiii discretization is equivalent to applying data-dependent convolution of size two on the state-input. A normal recurrent SSM materializes the state-input vy = Br then computes a linear recurrence h2 a +- Now, instead first apply a width-2 convolution vs (weighted by B,7) before passing it into the linear recurrence (4).
Remark (Convolution Differences). There is a distinct difference between the "convolution" induced by exponential trapezooidal discretization, and the standard short convolutions used by sequence models such as Mamba and Gated DeltaNet. Standard short convolutions are independent operations applied on 24 (and often C2 outside the core recurrence, while our new discretization can be interpreted as a convolution on the state-inpul B, within the core recurrence.
We also show that the exponential-taapezodaal discretization can be viewed as convolution applied to the mask matrix L in the SMA viewpoint (Appendix B.3.1)
3.2 COMPLEX- VALUED SSMS
Modern SSMs are designed with efficiency as the central goal, motivated by the need to scale to larger models and longer sequences. For instance, successive architectures have progressively simplified the state transition matrix: S4 (Gu et al., 2022a) used complex-valued Normal Plus Low Rank (NPLR) matrices, Mamba (Gu & Dao, 2024) reduced this to a diagonal of reals, and Mamba-2 (Dao & Gu, 2024) further simplified it to single scaled identity. Although these simplifications largely maintain language modeling performance, recent works (Merrill et al., 2025; Sarrof et al., 2024; Grazzi et al., 2025) have shown that the restriction to real, non-negative eigenvalue transitions degrades the capabilities of the model on simple state-tracking tasks. This limitation, formalized in Theorem of (Grazzi et al., 2024), arises from restricting the eigenvalues of the transition matrix to real numbers, which cannot represent "rotational" hidden state dynamics. Parity, the function on binary inputs {0.1}. defined as E, mod 2, is one such task. It can be performed using update: h4 = ((TT)), where R(-) is a 2-D rotation matrix. Such rotational dynamics cannot be expressed with real eigenvalues.
3.2.1 COMPLEX SSM WITH EXPONENTIAL- DISCRETIZATION
To recover this capability, we begin with complex SSMs (5), which are capable of representing state-tracking dynamics. We show that, under discretization (Proposition 5), complex SSMs can be formulated as real SSMs with a block-diagonal transition matrix composed of 2 x 2 rotation matrices (Proposition 2). We then show that this is equivalent to applying data- data-dependent rotary embeddings on both the input and output projections B,C respectively, which establishes a theoretical connection between complex SSMs and data-dependent RoPE embeddings (Proposition 3). Finally, the "RoPE trick" used in Su et al. (2023) allows for an efficient implementation complex-valee valued state-transition matrices with minimal computational overhead compared to real-valued SSMs.
Proposition 2 (Complex- to Real SSM Equivalence). Consider a complex-valued SSM
(1( - Diag (A0+))(()((()))())(((((((0()))))
(5)
v(t) Re (CC((((((())) (h()))
where h(t) E N/2, (tt), B(), C(t)),C(()CC))) N/2, and ((t), e IR. Under exponential- Euler discretization, this system is equivalent real-valued SSM
h = - Acd R A Brs
(6)
y/- M C. hi,
5

---

Published as a conference paper at ICLR 2026
with state h ERR, projections
Bi Br
C C
B
ERA,
ER.
C
and a transition matrix
RS Block (1R(,,0,0)))))
ERNXN
R(0)=
cos(O) sin
sin(0) cos(
The proof is given in Section C.1.
Proposition 2 shows that the discretized complex SSM of state dimension N/2 has an equivalent real SSM with doubled state dimension (N), and transition matrix is scalar decayed block-diagonal matrix of 2x2 data-dependent rotation matrices (e Rd).
(Complex SSM, Data- Sata-Dependent RoPE Equivalence). Under the notation established in Proposition 2, consider the real SSM defined in equation (6) unrolled for I time-steps. The output of the above SSM is equivalent to that of a vanilla scalar transition matrix-based SSM (11) with data-dependent rotary embedding applied on the B,C components of the SSM, defined by:
Proposition 3
[IRTT i==
y/: (ILEE
h E h
B B
C
he
where the matrix product represents right matrix multiplication, e.g., IT: R Ro We refer the usage of a transformed real-valued SSM to compute the complex SSM as the "RoPE trick""
The proof is given in Section C.2.
To observe the connection of complex SSMs to RoPE embeddings, note that in the above proposition, the data-dependent rotations R3 are aggregated across time-steps and applied to C.B, which, by the state space duality framework, correspond to the query (Q) and key (K) components of attention (Section 2.3). Analogously, vanilla RoPE (Su al., 2023) applies data-indepenn intte-indeeedent rotation matrices, where the rotation
angles follow fixed frequency schedule O[i] 10000 10000-2//X, 21/N
3.2.2 COMPLEX SSM WITH EXPONENTIAL-TRRAPZOIDDAA TRAPEZOIDAL DISCRETIZATION
After deriving the recurrence for complex SSMs with exponential- Buler discretization, generalization to exponential-rappezodall discretization is similar. Proposition 4 provides the full recurrence with the RoPE trick Mamba-3.
Proposition 4 (Rotary Embedding Equivalence with Exporential-Taapezoiaaal Trapezoidal Discretization). Discretizing a complex SSM with the apponential-rappeooide rule (Proposition 1) yields the recurrence
h; h
III
R
B B
R k
B
[ C
y1
h
(8)
1=0
Here R₁ is the block-diagonie rotation matrix defined in Proposition 3.
The proof is in Section C.3.
We empirically validate that complex SSM, implemented via data-dependent RoPE, is capable of solving state-tracking tasks that real-valued SSMs with and without standard RoPE cannot (Table 3b), supporting theoretical claims.
3.3 MULUI-INPUT, MULTI-OUTPUT
Scaling test-time compute has opened new frontiers in model capability, such as agentic workflows, where inference takes up increasing share of the overall compute budget. This has placed renewed focus on inference efficiency of language models and spurred the adoption of SSMs and sub-quadratic layers which feature fixed-sized hidden states and thus offer lower compute and memory requirements. Although these new layers have a lower wall-clock time compared to Transformers, their decoding is heavily memory- bound, resulting in low hardware utilization. In this section, use the SSM perspective introduce a methodological refinement to the Mamba-3 recurrence that allows for increased model FLOPs without increasing decoding wall-clock time, resulting betler model with the decoding speed.

---

Published as a conference paper at ICLR 2026
Decoding Arithmetic Intensity. To improve hardware efficiency. we need to consider the arithmetic intensity of token generation, defined as FLOPs divided by the number of input-output bytes for a given op. Since SSM decoding saturates the memory bandwidth with idle compute (i.e., being memory-bound), we would like to increase its arithmetic intensity to effectively overlay compute with memory I/O. More concretely, the arithmetic intensity for single generation in Mamba is around 2.5 ops per byte (Table 7a), while the arithmetic intensity for bíloat16 matmul is about 295 ops per byte for NVIDIA H100-SXM5 (NVIDIA, 2022). Consequently, SSM decoding falls far short of compule-buund regime, and moreover it is not clear how one can adjust the existing parameters in Mamba to mitigate the lack of hardware efficiency. We note that this observation applies generally to other sub-quadratic models, such as causal linear attention.
From SISO to MIMO. Consider single head a typical SSM with head dimension P. which involves stacking the SISO recurrence h, + hr B, with P copies sharing the same Or and B, The resulting broadcusted recurrence h₂ a ;, takes inputs xe E R and produces states h2 ER NxP.
Note that the memory traffic (input/output size) is dominated by the state h2, while the computation mainly comprises the outer product B which has FLOPss proportional to NP. We observe that by increasing the dimension of the latter terms, transforming B; ER + B, E IR and 2; ER - RR-xt the memory traffic does not significantly increase (for small R) while the FLOPs consumed increases by a factor of R (Table 7a). Thus, this transforatiin increases the arithmetic intensity of the recurrecce Furthermore, the increased arithmetic intensity is translated into practical gains because the outer product B becomes a hardware-effin eefficrreeffii matrix- matrix product (matmul) which incurs very little overhead. As result, the new recurrence is more expressive than the original while practically preserving the decoding speed.
For similar reasons, the computation of the output from the state, y/: + C h, acquires an extra rank R, by transforming the output projection as C. E - E TR NxR Overall, this transformation is equivalent to expanding the original single-input, sungle-output (SISO) recurrence to multi-input, multi-output (MIMO). We note that MIMO SSMs incur a slow-down during training due to an R-fold increase in FLOPs with more compute-bound kemels, even though the decoding wall-clock performance is not significantly impacted due to its neemory-bound kernlls. Further details on the usage of MIMO SSMs in Mamba-3 are in Section D.
3.4 MAMBA-3 ARCHITECTRRE
The overall architecture follows Llama (Grattafiori et al., 2024), alternating Mamba-3 and SwiGLU blocks with pre-norm. The Mamba-3 block retains the overall layout of its predecessor, while introducing several key modifications.
Updated SSM Recurrence. The SSD layer is replaced with the more expressive complex-valuee SSM defined in Proposition Mamba-3 employs the SISO SSM by default to enable fair comparisons with other SISO-like models, but its MIMO variant can be trained and deployed as a stronger alternative to baseline Mamba-3 (Table 1).
BC QK Normalization. RMS normalizations are added following the B,C projection, mirroring the QKNorm commonly used in modemn Transformers (Henry et al., 2020; Wortsman et al., 2023) and other recent linear models (Yang et al., 2025c; Hu et al., 2025). We call this either BCNorm or QKNomm interchangeably. We find that BCNom is also able to stabilize large-scale runs, resulting in the removal of the post-gate RMSNorm layer (introduced in Mamba-2 for stability) in our pure Mamba-3 models. However, in hybrid models, the removed RMSNorm is crucial for long-context extrapolation (Table 2).
C. Biases. Similarly to Yu & Erichson (2025), which proved adding channel-specific bias to B in blockwise variant of 1amba-1 grants universal approximation capabilities, Mamha-3 incorporates learnable, head-specifie, channel-wise biases into the B and C after BCNorm. We hypothesize that these biases also endow the model with more convolution-like behavior, where the added biases in B,C' create data-independent SSMs that function more similariy to convolutions. Ablations on the bias parameterization are located in Section G). The combination of data-indeppnddet bias parameters, together with exponential-trapeaae trapezoidal discretization (which itself induces a convolution on the state-input), is empirically able to obviate the short causal convolution and its accompanying activation function present in Mamba-2 and most modern recurrent models (Section 4.2).
EMPIRICAL VALIDATION
4.1 LANGUAGE MODELING
All models are pretrained with 100B tokens of the FineWeb-Edd dataset (Penedo et al., 2024) with the Llama-3.1 tokenizer (Grattafiori et al., 2024) at 2K context length with the same standard training protocol. Training and evaluation details can be found in Section E.
7

---

Published as a conference paper at ICLR 2026
Table 1: Downstream language modeling evaluations on models trained with 100B FineWeb-Edu tokens. Best results are bolded, and second best are underlined, excluding Mamba-3 MIMO variants. All models are trained with the same procedure. Mamba-3 SISO outperforms Mamba-2 and others at every model scale, and MIMO with rank R R=4 further improves modeling capabilities.
Model
FW-Edd
LAMB.
LAMB.
HellaS.
PIQA
Arc-E
Arc-C
WinoGr. acc 1
OBQA
Average
ppl
ppl 4
acc 1
acc_n
acc f
acc
aoc.n 1
acc
acc
Tunnformar-180X
16.89
45.0
32.5
39.0
67.1
59.8
27.9
51.2
21.8
42.8
GDN-I80M
16.52
40.8
31.3
40.2
66.3
62.3
28.2
51.7
22.0
43.2
Mamba-2-180M
16.76
41.8
30,9
10.1
66.8
60.1
27.3
52.0
23.2
42.9
Nummb----SSS-1800
16.59
37.7
32.5
40.8
66.1
61.5
27.9
52.0
22.8
43.4
Mamba-3-
Mamba-3-MIMO-BBa MIMO-18M
16.46
32.1
34.0
41.0
66.7
60.6
27.7
52.9
22.0
43.5
Transfoma-4400
13.03
21.2
41.7
50.5
69.9
67.6
34.6
56.7
26.0
49.6
GDN-440M
13.01
18.0
41.9
50.9
70.0
67.0
316
56.1
27.6
49.7
Mamba-2-440M
13.00
19.6
40.8
51.7
70.6
08.8
35.0
54.1
26.0
49.6
Mumba-3-SI50-4400
12.87
19.6
40.2
51.7
71.9
68.9
344
55.8
26.0
49.8
Mamba-3-MIMO--4e
12.72
17.1
43.4
52.8
70.8
69.6
35.6
56.3
28.4
51.0
Transfomer-8800
11.42
15.0
44.7
57.2
71.6
39.2
57.7
52.8
GDN-880M
11.37
12.9
47.6
57.3
73.3
71.4
38.7
58.8
28.6
03.7
Mamba- 2-800
11.35
13.8
45.0
58.1
72.5
72.3
38.7
56.8
30.2
53.4
Mamba-3-S150-890M
11.23
12.9
47.2
58.8
73.6
72.7
40.2
58.4
30.0
54.4
11.11
11.8
49.5
59.2
73.7
74.7
41.2
59.9
28.6
55.3
Transfomer-1
10.51
11.1
50.3
60.6
73.8
74.0
40.4
58.7
29.6
55.4
GDN-1.5B
10.45
10.9
61.3
75.3
41.2
58.0
Mamha-2-1.55 2-1hha-2-11.B-
10.47
12.0
47.8
61.4
73.6
75.3
41.8
57.5
32.6
55.7
Mamba-3-S150-155B
10.35
10.9
49.4
61.9
73.6
75.9
42.7
59.4
32.0
56.4
Mamba-2-MIMM0-1158
10.24
10.2
51.7
62.3
75.3
76.5
44.5
60.6
32.6
57.6
Across all four model scales, Mamba-3 outperforms popular baselines at various downstream tasks (Table 1). We highlight that Mamba-3 does not utilize the external short convolution that has been empirically identified an important component in many performant linear models (Gu & Dao, 2024; Yang al., 2025c).
4.1.1 MIMO
We further verify the gain from MIMO by investigating its language-modeling abilities by training MIMO models with rank R= following the same settings as Section 4.1. To ensure that the total parameter count comparable to SISO-based models, we decrease the inner dimension of the MLPs compensate for the increase due to the MIMO projections. See Section D for details.
On both validation perplexity and our suite language evaluation tasks (Table 1), we see significant gain when moving from SISO to MIMO for our Mamba-3 models. Namely, we achieve a significant perplexity gain of 0.11 on the 1.5B models, and Figure illustrates the downward shift our validation loss. On the language evaluation front, gain of 1.2 percentage points over SISO.
gains most tasks when compared to SISO, resulting in average
4.1.2 RETRIEVAL CAPABILITIES
Beyond standard language modeling, an important measure for linear models is their retrieval ability-hor well they can recall information from earlier the sequence (Arora et al., 2025a;b). Unlike attention models, which can freely revisit past context with the growing KV cache, linear models must compress context into fixed-size state. This trade-off's reflected in the Transformer baseline's substantially stronger retrieval scores. To evaluate Mamba-3 under this lens, Table compares it against baselines both real-world and synthetic needle-in-a-hee ina-hhastacc (NIAH) tasks (Hsieh et al., 2024), using our pretrained 1.5B models from Section 1.1. We restrict the task sequence length to 2K tokens match the training setup and adopt the cloze-style format for our real-world tasks to mirror the next-token- prediction objective, following Arora et al. (2025b; 2024).
next-tokee--rrdiciion
Mamba-3 competitive on real-world associative recall and question-ans struggles on information extraction from semi-structured or unstructured data (SWDE, FDA), On synthetic NIAH, however, Mamba-3 surpasses or matches baselines on most cases and demonstrates markedly better cut-of-ddisributtio retrieval abilities than its Mamba-2 predecessor.
(TQA, SQUAD) but
Improving Retrieval with Hybrid Models. Because of the natural retrieval-based weaknesses of fixed state-size, we predict that linear layers will be predominately used in hybrid models that mitigate with quadratic self-attention layers. We evaluate how Mamba-3 performs within this architectural paradigm by
this downside
8

---

Published as a conference paper at ICLR 2026
Table 2: Retrieval capabilities measured by mixture real-world and synthetic retrieval tasks, Real-world retrieval tasks utilize cloze variants the original datasets and are truncated to 2K length. Mamha-3 demonstrates strong associative recall, weession-nnswriing, and length generalization on seedle-in-a-haystare (NIAH), but suffers with information extraction of remi-structurce and unstructured data. The Transformr baseline uses RoPE which may explain its length generalization issues, and hybrid models utilize NoPE (no positional embeddings). We find pre-gale, grouped RMSNorm can be added to Mamba-3 SISO hybrid models to improve the length generalization of the NIAH tasks slight decrease weal-world retrieval performance.
Mudel (1.58)
SWDE
SQD.
IDA
TQA
NQ
Drup
NIAll-ingle-1
Single2
NIAH--inglees
Context Length
2048
1024
2048
4096
1024
2048
4096
1024
2048
4096
Transforner
48.9
46.6
08.4
67.5
31.7
26.4
100.0
100.0
0.0
92.2
100.0
0.0
98.6
99.4
0.0
Pire
GDN
82.7
40.0
28.3
63.5
25.7
24.5
100.0
100.0
99.8
100.0
93.8
49.8
83.8
08.4
34.2
Mamika22
30.7
39.1
23.7
61.3
26.1
28.5
100.0
99.6
62.0
100.0
53.8
11.8
95.8
87.4
13.1
Mumba-3 SISO
SISO
28.5
40.1
23.4
64.5
26.5
27.4
100.0
100.0
88.2
100.0
95.4
50.6
92.4
81.4
34.2
Mumba-3 MIMO
36.3
41.7
29.3
64.5
26.2
26.3
100.0
100.0
93.0
100.0
86.0
40.4
95.8
64.4
25.6
GDN
54.6
48.4
58.8
64.9
32.7
30.0
100.0
100.0
71.4
98.6
100.0
60.2
70.0
96.2
24.0
Hybod
Mambu-2
08.2
45.6
71.0
66.1
353.4
28.1
100.0
100.0
98.6
98.8
98.0
0.0
Mamia-3 SISO
08.5
17.0
65.9
64.8
33.4
27.0
100.0
100.0
36.2
100.0
100.0
9.1
99.8
100.0
8.8
Mamba-3 SISO Norm"
58.6
47.3
32.4
60.7
33.3
28.5
100.0
100.0
100.0
100.0
100.0
966.0
980.8
97:22
56.8
Table 3: Left: Ablations on core modeling components of Mamba-3 SISO, Right: Formal language evaluation (scaled accuracy, %). Higher is better. SISO models are trained on short sequences and evaluated on longer lengths to test length generalization. For GDN report the variant with eigenvalue range [-1,1].
Arith. w/o
Arith. w/ brackets f
Model Variant
ppl 4
Model
Parity
brackets f
Mamba-3 bias trap
16.68
Mamba-3
100.00
98.51
87.75
Mamba-3 bias
16.49
Mamba-3 (w/ Std. RoPE)
1.56
20.70
2.62
M
15.72
Mamba-3 (w/o RoPF)
2.27
1.49
0.72
Mamba-3 + conv
15.85
Mamba-2
0.90
47.81
0.88
GDN [-1,1]
100.00
99.25
93.50
(a) Component ablation at 440M scale. Com- bining BC bias and exponential -tapeezodarrezzi dis-
(b) Performance comparison on formal language tasks, Unlike Mamba-2, Mamba-3 features state-tracking ability stemming from data-dependent RoPE embeddings.
cretization makes the ubiquitous short convolu- tion optional.
training hybrid models with an interleaving fashion of a 5:1 ratio of linear layer to NoPE self-attention (Yang al., 2025b). As seen in prior work (Waleffe al.. 2024), hybrid models outperform the Transformer bascline. We find that the reintroduction of the pre-output RMSNor (pre-gate, grouped RMSNorm in Table 2) to the Mamba-3 layer improves the length generalization retrieval abilities at the slight cost of in-context, real-world retrieval tasks and is highly competitive as a linear sequence mixing backbone when mixed with self-attention. However, the ideal norm type (grouped vs default) and its placement (pre- vs post-gate) still unclear due to competing tradeoffs (Table 8), as we find that hybrid models and their characteristics and dynamics are complex and oftentimes unintuitive, a point echoed in Cabannes al. (2025).
4.2 SSM METHODOLOGY ABLATIONS
Table 3a ablates the changes that Mamba-3 introduces core SSM components, mainly the introduction of BC bias and exponential ssponential-trapzzziiaa discretization. We report the pretraining test perplexity on models at the 440M scale, trained for Chinchilla optimal tokens. We find that the bias and ayponential-trapzzoii SSM synergize well and make the short convolution utilized by many current linear models redundant.
We empirically demonstrate that data-dependent RoPE in Mamba-3 enables state tracking. Following Grazzi et al. (2025), we evaluate on tasks from the Chomsky hierarchy- -Pritty Modular Arithmetic (without brackets), and Modular Arithmetic (with brackets) and report scaled accuracies in Table 3b. Mamba-3 solves Parity and Modular Arithmetic (without brackets), and nearly closes the accuracy gap on Modular Arithmetic (with brackets). In contrast, Mamba-3 without RoPE, Mamba-3 with standard RoPE (Su et al., 2023), and Mamba-2 fail to leam these tasks. We use the state-tracking-endle that Mamba-3 is competitive-matchiin parity and approaching its performance both modular--rithmetic tasks. Experimental settings are covered in Section E.
GDN variant of and observe
4.3 INFERENCE EFFICIENCY TO PERFORMANCE TRADEOFF
As damt impacts the decode runtime for the subquadratic models considered in this paper (Section 3.3), we opt to use as proxy for inference speed. By plotting the validation perplexity (itself a proxy for model per-

---

Published as a conference paper at ICLR 2026
Relative Total State Size vs Pretraining Perplexity
15.2
Pretraining Perplexity
Mamba
15.0
Mamba-3
Model
FP32
BF16
des 61
des = 128
dua =1
dlase = 128
14.8
Mambaa2
0.296
0:403
GDN
0.341
0:423
0.176
0.257
14.6
Mambu.3 (SISO:3(SISO)
0.310
0.300
0.110
0.156
Mamba-3 (MIMO)
0:431
Relative Total State Size
Table 4: Kernel latency (in milliseconds) comparison
across models, precision, and dawte values. Mamba-3 introduces minimal overbead compared to Mamba-2 and features highly efficient practical implementa- tions. Our Mamba-3 SISO kernels are faster than reference Mamba-2 and GDN kernels at the com- monly used 6f16, daate = 128 setting. Mamba-3 MIMO 4R incurs little additional cost compared to SISO.
Figure 2: Exploration of state size (inference speed proxy) versus pretraining perplexity (performance proxy) across different Mamba variants. Mamba-3 improves the Pareto frontier compared previous recurrent SISO models, while incorporating MIMO further shifts the frontier through better modeling performance without increasing state size.
formance) as a function of damee we aim to formulate a holistic picture about how the subquadratic models can trade off performance with inference speed. Figure shows such Pareto frontier for the Mamba models con- sidered in this paper. We train a 440M parameter model to 2x Chinchilla optimal tokens on the Fineweb-Edu dataset, where the model is configured with a dsaate of [16,32.64,1288 As expected, we observe an inverse correlation between validation loss and daasee- Moreover, there is general downward shift on the Pareto frontier moving from Mamba-2 to Mamba-3, indicating stronger model: this setting, Mamba-3 with 2x smaller state size achieves better pretraining perplexity than its Mamba-2 counterpart, resulting in faster model with the same quality or better model for the same speed. A further downward shift is observed when moving from the SISO variant of Mamba-3 to the MIMO variant of Mamba-3 (where set the MIMO rank R 4 and de- crease the MLP inner dimension to parameter match the SISO variants). We expand the comparison to include the GDN baseline in Section F, Figure 6, which also shows Mamba-3 comparing favorably to GDN.
4.4 FAST MAMBA-3 KERNELS
We complement Mamba-3's methodological advances with optimized kemels that deliver fast inference in practical settings. Specifically, we implement a new series of inference kernels for uainn---using Triton for the forward (prefill) path and CuTe-DSL. for decode-and compare their per-token decode latency against the released Triton kemnels for Mamba-2 and GDN in Table 4. The evaluation uses the setting: a decode step at batch size 128 on a single H100 for -SB-aarmmeter models with model dimension 2048, state dimension E (64,128) in both FP32 and BF16 datatypes. Across all configurations, SISO achieves the lowest latency amongst baselines. MIMO, with its higher arithmetic intensity, increases the decoding FLOPs without significantly increasing decode runtime This indicates that our CuTeeDDLL decode implementation is competitive and that the additional components of Mamba-3 (exponentialtrapezzoiaa update, complex-valued state, and MIMO projections) are lightweight. This supports our overall inference-firs perspective: the Mamba-3 admits simple, low-latency implementation while providing strong empirical performance.
Table 12 benchmarks end-to-end latency across different decoding sequence lengths, and additionally benchmarks prefill time for the same sequence length. The decode time is consistent with Table 4, where Mamba-3 (SISO) is fastest, Mamba-3 (MIMO) is on par with Mamba-2, and all linear methods are faster than optimized attention sequence length grows. We also see that MIMO incurs moderate overhead for prefill, as discussed in Section 3.3. Details of the benchmark are provided in Section H.
5 CONCLUSION AND FUTURE WORK
We introduce Mamba-3, an SSM model with three axes of improvement rooted in SSM principles: (i) improved quality, via exponential-raapezoiaall discretization; (ii) new capabilitiess, through complex SSMs that recover state tracking; and (iii) higher inference efficiency, with a MIMO formulation that raises arithmetic intensity. Mamba-3 delivers strong language modeling results and establishes new Pareto frontier the performance-efficiinney axes with respect to strong baseline models. We see hybrid Mamba-3 architectures that integrate retrieval mechanisms as promising path, alongside broader application of our design principles to linear-time sequence models.
Details each kemel DSL. and the exact kemel fusion structure is provided in Appendix H.
10

---

Published as a conference paper at ICLR 2026
Acknowledgments.
We gratefully acknowledge the support of the Schmidr Sciences A12050 fellowship, the Google ML and Systems Junior Faculty Awards, the Google Research Scholar program, Together AI, and Cartesia AL KL supported by the NSF GRFP under Grant DGE2140739, We also thank Sukjun Hwang and Gaurav Ghosal for helpful feedback and discussions.
REFERENCES
Anthropic. Introducing claude opus 4.6, February 2026. URL htt.:ttts///ww.atttrpp.....o
news / cww.ccl-lau--ssseeee
Aryaman Arora, Neil Rathi, Nikil Roashan Selvam, Róbert Csordás, Dan Jurafsky, and Christo- pher Potts. Mechanistic evaluation of transformers and state space models, 2025a. URL https:
Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher Ré. Just read twice: closing the recall gap for recurrent language models, 2024, URL ht https: /ttp/aixiiv./// www.rivww.i.oo...ooooooo abs/ 2407 05483.
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the thoouhhhou tradeoff, 2025b. URL https: wtwaaiivvvvooo abs / 22022422222 18668.
Damitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2014. URL httttp:::/aaxi..oom
abs /1409. 0473.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. http:::::/ :/ap:iis:////vv.v.///a org/ abs/ /1911. 11 641.
Loic Cabannes, Maximilian Beck, Gergely Szilvasy, Matthijs Douze, Maria Lomeli, Jade Copet, Tierre-Emmauul Mazaré, Gabriel Synnaeve, and Hervé Jégou. Short window attention enables long-term memorization, 2025. URL https:tt///.. //ttx/aaavvv///e///e//eeeeaaaeve
org/ abs /2..///5..2552
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Be- langer, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2022. URL
http:///sss200..111. https: http///axvvv.o http:///aawww...//aa/oseoo/ese/e
Peter Clark, Isaac Cowbey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL
https: arxiv. .rgg alss/ 1803. 05457.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL https: http:///xxvv..rr..// wwt.:bs...//.../..o aww.//444000 /2w..2405..0 21060.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL. https: http::///xxxvv..g//aae
/1903. 00161.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL htttps://eenod: org/ records 208 602.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang. Bobbie Cher,, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruy Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin,
11

---

Published as a conference paper at ICLR 2026
Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iiyan Zarov, Imanol Arrieta Iharra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun. Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak. Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Santala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria TT simpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji. Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh. Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende. Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang. Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpiere Coudert, Zheng Yan, Zhengxing Chen, 7oe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, thing-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee. Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj. Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Bapttite Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings. Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya I Lathi, Meghan Keneally, Miao Liu, Michael I.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyyn, Mike Clark, Mike Maccy, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar,
12

---

Published as a conference paper at ICLR 2026
Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy. Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta. Summer Deng. Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Mathhws, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Veto,o Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Ilama herd of models, 2024. URL https: / /arxivv.vvvv/////////////////vv/ org/ abs/2407 21182177 783.
Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, and Frank Hutter. Is mamba capable of in-context learning?, 2024. URL httpt:////aaaaa
.rgg aaaaab//22 2abs/2402.00 03170.
Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K. Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear mns through negative eigenvalues, 2025. URL alss/ 2411. 12537.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL pitts:///rxivy www.vvorg//aaaaa albs//ab/////// /tloo///2113 231..0075.....
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, /2111. 00396.
2022a. URL betttss//arxiv..ott
Alhert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and ini- tialization of diagonal state space models. arXiv preprint arXiv:2206.11893, 2022b. URL
https: http///axvvvv.// org/ al///////22// 11893.
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces, 2022. URL https: //tpxvva.vvvvv/////eae org/ attaab///0.... 2203.2222...2 1 14343.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers, 2020. htRp::/:: http://awww..../a///./.. www.or/////a/aeeeeeeeeeeeeeeeeee 221 1.00000000 04245.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg, Ruler: What's the real context size of your long-context language models?, 2024.
URL https: /arxiviii..////////aaaa///aiaavv
org/ albs/ 24044 06654.
Jiaxi Hu, Yongqi Pan, Jusen Du, Disen Lan, Xiaqiang Tang. Qingsong Wen, Yuxuan Liang, and Weigao Sun. Comba: Improving bilinear mns with closed-loop control, 2025. URL
https:/ / //aaaaxxvv.o...//.. halgg//bs//// /25/ 02475.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL
03551.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. formers are mns: Fast autoregressive transformers with linear attention, 2020.
Trans-
URL
https:/ http:///axvvvv.. /ls///b///// /tt/a//ss..0...// 2006 16236.
Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang. Yu Fan, Longguang Zhongg Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie
13

---

Published as a conference paper at ICLR 2026
Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang. Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, and Yulun Du. Kimi linear: An expressive, efficient attention architecture, 2025. URL https: /tappp:://rrae /ttp///aavvvv/ee//ee/eeeeeeeevva org/ abs///////// 2et.//211........ 26692.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the A Association for Computaatooal Linguistics, 7:452-466, 2019. doi: 10. 11111622aclaa.002700 URL https: / /aaaaaaaanttoooyoooooooooo org/ QQ 9-1666/
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https:
abs /aabsb2/ 2309 06180.
Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. Llm inference serving: Survey of recent advances and opportumities, 2024. URL https: / aaaaiiivvvvv. org/ abs/ abs 240. 12391.
William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models, 2025.
URL https: / /aw.iiiiiii...aa/a.aaaaaaaaa..aa orrg//v.rrr./...o org/ abss/4044 08819.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of ar- mor conduct electricity? a new dataset for open book question answering, 2018. URL 02789.
NVIDIA. Nvidia h100 tensor core gpu white paper. URL
Technical report, NVIDIA, 2022. waapppee--rrcctttctureee
https://resourcee
NVIDIA, :, Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Alek- sander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, Andrew Tao, Anna Shors, Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Buddharaju, Bobby Chen, Boris Ginsburg, Boxin Wang. Brandon Norick, Brian Butterfield, Bryan Catanzaro, Carlo del Mundo, Chengyu Dong, Christine Harvey, Christopher Parisien, Dan Su, Daniel Korzekwa, Danny Yin, Daria Gitman, David Mosallanezhad, Deepak Narayanan, Denys Fridman, Dima Rekesh, Ding Ma, Dmytro Pykhtar, Dong Ahn, Duncan Riach, Dusan Stosic, Eileen Long, Flad Segal, Ellie Evans, Eric Chung, Frick Galinkin, Evelina Bakhturina, Ewa Dobrowolska, Fei Jia, Fuxiao Liu, Gargi Prasad, Gerald Shen, Guilin Liu, Guo Chen, Haifeng Qian. Helen Ngo, Hongbin Liu, Hui Li, Igor Gitman, Ilia Karmanov, Ivan Moshkov, Izik Golan, Jan Kautz, Jane Polak Scowcroft, Jared Casper, Jarno Seppanen, Jason Lu, Jason Sewall, Jiaqi Zeng, Jiaxuan You, Jimmy Zhang, Jing Zhang, Jining Huang, Jinze Xue, Jocelyn Huang, Joey Conway, John Kamalu, Jon Barker, Jonathan Cohen, Joseph Jennings, Jupinder Parmar, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Luna, Keshav Santhanam, Kezhi Kong, Kirthi Sivamani, Krzysztof Pawelec, Kumar Anik, Kunlun Li, Lawrence McAfee, Leon Derczynski, Lindsey Pavao, Luis Vega, Lukas Voegtle, Maciej Bala, Maer Rodrigues de Melo, Makesh Narsimhan Sreedhar, Marcin Chochowski, Markus Kliegl, Marta Stepniewska- Dziubinska, Matthicu Le, Matvei Novikov, Mehrzad Samadi, Michael Andersch, Michael Evans, Miguel Martinez. Mike Chrzanowski, Mike Ranzinger, Mikolaj Blaz, Misha Smelyanskiy, Mohamed Fawzy, Mohammad Shoeybi, Mostofa Patwary, Nayeon Lee, Nima Tajbakhsh, Ning Xu, Oleg Rybakov, Oleksii Kuchaiev, Olivier Delalleau, Osvald Nitski, Parth Chadha, Pasha Shamis, Paulius Micike- vicius, Pavlo Molchanov, Peter Dykas, Philipp Fischer, Pierre Yves Aquilanti, Piotr Bialecki, Prasoon Varsh- ney, Pritam Gundecha, Przemek Tredak, Rabeeh Karimi, Rahul Kandu, Ran El- Raviraj Joshi, Roger Waleffe, Ruoxi Zhang, Sabrina Kavanaugh, Sahil Jain, Samuel Kriman, Sangkug Lym, Sanjeev Satheesh, Saurav Muralidharan, Sean Narenthiran. Selvaraj Anandaraj. Scommyeong Bak, Sergey Kashirsky, Seungju Han, Shantanu Acharya, Shaona Ghosh, Sharath Turuvekere Sreenivas, Sharon Clay, Shelby Thomas, Shri- mai Prabhumoye, Shubham Pachori, Shubham Toshniwal, Shyamala Prayaga, Siddhartha Jain, Sirshak Das. Slawek Kierat, Somshubra Majumdar, Song Han, Soumye Singhal, Sriharsha Niverty, Stefania Alborghetti, Suseella Panguluri, Swetha Bhendigeri, Syeda Nahida Akter, Szymon Migacz, Tal Shiri, Terry Kong, Timo Roman, Tomer Ronen, Trisha Saar, Tugrul Konuk, Tuomas Rintamaki, Tyler Poon, Ushnish De, Vahid Noroozi, Vanun Singh, Vijay Korthikanti, Vitaly Kurin, Wasi Uddin Ahmad, Wei Du, Wei Ping, Wenliang Dai, Wonmin Byeon, Xiaowei Ren, Yao Xu, Yejin Choi, Yian Zhang, Ying Lin, Yoshi Suhara, Zhiding Yu, Zhiqi Li. Zhiyu Li, Zhongbo Zhu, Zhuolin Yang. and Zijia Chen. Nemotron-h: A family of accurate and ef- ficient hybrid mamba-transforme models, 2025. URL https: a/arxiv org/ alss / 25044 03624.
sippniewskaa-ziiuinnkka
14

---

Published as a conference paper at ICLR 2026
OpenAL. Introducing cod-5, February 2026. URL https://openai. /nnee//
introducing-gpt--a -esee-53--o000 3-oodex/o
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bemardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Femández. The lambada dataset: Word prediction requiring broad discourse context, 2016. URL https: / /arxiv. org/ abs//////// / 1606. 06031.
Guilherme Penedo, Hynek Kydlřček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https: //aaaiivvvoo org/ abs 2406. 17557.
Pranav Rajpurkar, Jian Zhang, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In ACL 2018, 2018.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi, Winogrande; An adversarial winograd schema challenge at scale, 2019. URL httpt: /ttpiiv////ivvvae/aeeeee org/ /190//199... 10641.
Yash Sarrof, Yana Veitsman, and Michael Hahn. The expressive capacity of state space models: formal language perspective, 2024. URL
http//awwwv...../aaa///// aawr///////////////////////esaee aww.ab///55.. /2405.10111 1739..
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers secretly fast weight programmers, 2021. URL https: /ttp//awwwww.aaaaaaaaa/aa wttpaa/////2222..... 2102.11172111
Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling, 2023. URL https::///asstiiae. org 22088. 04933.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters, 2024. URL
abs 0/0///4000000 003114
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding, 2023. URL htw.thtttt////tttt http:///wwwwwwwwwwaa/osooo/e/to www./r////.//eeeeeeeeeeo.eeeeee/ 04 09864.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram. Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (leam at test time): Rnns with expressive hidden states, 2025. URL https: aaaaaaivvvvo../ org/ abs /2407. 02600.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023. URL https: http://axxvvvvv /bs/2207.088888
Endre Süli and David F. Mayers. An Introduction to Numerical Analysis. Cambridge University Press, 2003.
Amuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, Jed McCaleb, Yejin Choi, and Yu Sun. End- to-end test-time training for long context, 2025. URL https: www.v. alss/ 251225122222 2512.23675. 23675.
Tencent Hunyuan Team, Ao Liu, Botong Zhou, Can Xu, Chayse Zhou, ChenChen Zhang, Chengcheng Xu, Chenhao Wang, Decheng Wu, Dengpeng Wu, Dian Jiao, Dong Du, Dong Wang, Feng Zhang, Fengzong Lian, Guanghui Xu, Guanwei Zhang, Hai Wang, Haipeng Luo, Han Hu, Huilin Xu, Jiajia Wu, Jianchen Zhu, Jianfeng Yan, Jiaqi Zhu, Jihong Zhang, Jinbao Xue, Jun Xia, Junqiang Zheng, Kai Liu, Kai Zhang, Kai Zheng, Kejiao Li, Keyao Wang, Lan Jiang, Lixin Liu, Lulu Wu, Mengyuan Huang, Peijie Yu, Peiqi Wang. Qian Wang. Qianbiao Xiang, Qibin Liu, Qingfeng Sun, Richard Guo, Ruobing Xie, Saiyong Yang, Shaohua Chen, Shihui Hu, Shuai Li, Shuaipeng Li, Shuang Chen, Suncong Zheng, Tao Yang. Tian Zhang, Tinghao Yu, Weidong Han, Weijie Liu, Weijin Zhou, Weikang Wang. Wesleye Chen, Xiao Feng, Xiaoqin Ren, Xingwu Sun, Xiong Kuang, Xuemeng Huang, Xun Cao, Yanfeng Chen, Yang Du. Zhen Yang. Yangyu Tao, Yaping Deng, Yi Shen, Yigeng Hong. Yiqi Chen, Yiqing Huang, Yuchi Deng. Yue Mao, Yulong Wang. Yuyuan Zeng, Zenan Xu, Zhanhui Kang, Zhe Zhao, ZhenXiang Yan, Zheng Fang, Zhichao Hu, Zhongzhi Chen, Zhuoyu Li, Zongwei Li, Alex Yan, Ande Liang, Baitong Liu, Beiping Pan, Bin Xing, Binghong Wu, Bingxin Qu, Bolin Ni, Boyu Wu, Chen Li, Cheng Jiang, Cheng Zhang, Chengiun Liu, Chengxu Yang, Chengzhong Xu, Chiyu Wang, Chong Zha, Daisy Yi, Di Wang. Fanyang Lu, Fei Chen, Feifei Liu, Feng Zheng, Guanghua Yu, Guiyang Li, Guohua Wang, Haisheng Lin. Han Liu, Han Wang, Hao Fei, Hao Lu, Haoqing Jiang. Haoran Sun, Haotian Zhu, Huangjin Dai, Huankui Chen, Huawen Feng. Huihui Cai, Huxin Peng, Jackson Lv, Jiacheng Shi, Jiahao Bu, Jianbo Li, Jianglu Hu, Jiangtao Guan, Jianing Xu, Jianwei Cai,
15

---

Published as a conference paper at ICLR 2026
Jiarong Zhang, Jiawei Song, Jie Jiang, Jie Liu, Jieneng Yang, Jihong Zhang, Jin Iv, Jing Zhao, Jinjian Li, Jinxing Liu, Jun Zhao, Juntao Guo, Kai Wang. Kan Wu, Lei Fu, Lei He, Lei Wang, Li Liang Dong, Liya Zhan, Long Cheng, Long Xu, Mao Zheng, Meng Liu, Mengkang Hu, Nanli Chen, Peirui Chen, Peng He, Pengju Pan, Pengzhi Wei, Qi Yang. Qi Yi, Roherts Wang. Rongpeng Chen, Rui Sun, Rui Yang. Ruibin Chen, Ruixu Zhou, Shaofeng Zhang. Sheng Zhang. Shihao Xu, Shuaishuai Chang. Shulin Liu, SiQi Wang. Songjia Feng, Songling Yuan, Tao Zhang, Tianjiao Lang, Tongkai Li, Wei Deng, Wei Li, Weichao Wang, Weigang Zhang, Weixuan Sun, Wen Ouyang, Wenxiang Jiao, Wenzhi Sun, Wenzhuo Jia, Xiang Zhang, Xiangyu He, Xianshun Ren, Xiao YiaoYing Zhu, Xiaolong Guo, Xiaoxue Li, Xiaoyu Ma, Xican Lu, Xinhua Feng. Xinting Huang, Xinyu Guan, Xirui Li, Xu Zhang. Xudong Gao, Xun Luo, Xuxiang Qi. Yangkun Chen, Yangyu Tao, Yanling Xiao, Yantao Mai, Yanze Chen, Yao Ding, Yeting Yang. Yifan Song. Yifan Yang. Yijiao Zhu, Yinhe Wu, Yixian Liu, Yong Yang. Yuanjun Cai, Yuanlin Tu, Yue Zhang, Yufei Huang. Yuhang Zhou, Yuhao Jiang. Yuhong Liu, Yuhui Hu, Yujin Lin, Yun Yang. Yunhao Wang. Yusong Zhang. 7ekun Wi, Zelong Zhang. Zhan Yu, Zhaoliang Yang. Zhe Zhao, Zheng Li. Zhenyu Huang. Zhiguang Liu, Zhijiang Xu, Zhiqing Kui, Zhiyin Zeng, Zhiyuan Xiong, Zhuo Han, Zifan Wu, Zigang Geng, Zilong Zhao, Ziyan Tang. Ziyuan Zhu, Zonglei Zhu, and Zhijiang Xu. Slunyuan-turbos Advancing large language models through trama-transformer synergy and adaptive thain-of-thought, 2025. URL https: / arxiv. org/ /abs//////// / 22055 15431.
M. Tenenbaum and H. Pollard. Ordinary Differential Equations: An Elementary Textbook for Students of Mathematics, Engineering, and the Sciences. Dover Books on Mathematics. Dover Publications, 1985. ISBN 9780486649405. URL https: / /books. google. com wwwooooodddiudoaaaa..ea
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017. URL http: / arxiv. .arv.rrg orgg abs////s/ 03762.
Roger Waleffe, Wonmin Byeon, Dunean Riach, Brandon Norick, Vijay Korthikanti, Tri Dao. Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An empirical study of mamba-based language models, 2024. URL https: arxiv. org/ abs / 2400 07 787.
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar. Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein, Kelvin Xu, Jachoon Lee, Justin Gilmer, and Simon Komblith. Small-scale proxies for large-scale transformer training instabilities, 2023. URL https: / arxivv org/ abs 22009 14322.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2025. URL https:/ / arxiv v. org/ alos/ 2408 00724.
An Yang, Anfeng Li. Baosong Yang, Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu. Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. J'anhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan. Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekum Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL https:://// http://xxvvvv.o alos 2505. .0.55.9.0 09388.
Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, and Acyr Locatelli. Rope to nope and back again: A new hybrid attention strategy, 2025b. URL https://aat http:///axvvvvvv worg/ alos 22501118777
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention trans- formers with hardware-efficient training, 2024. URL https: http://axxvvvor/// org//bs/ /2312 06635.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule,
2025c. URL https: / rxiv, wor. / 22412. 0.660066666 64.
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length, 2025d. URL htt.ttttss/eeee //attpss/////ii...// wttg//a/////aaee/oo /200 06484.
16

---

Published as a conference paper at ICLR 2026
Annan Yu and N. Benjamin Erichson. Block-biased mamba for long- long-range sequence processing, 2025. URL https::/aat //ariivaaaivv///aa/ org/ alos/ 2505. 09022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really
finish your sentence?, 2019. URL https: / /twxiaivvvoooo/////a//aaaa org/ aaas /1905 07830.
Tianyuan Zhang. Sai Bi, Yicong Hong. Kai Zhang, Fujun Luan, Songlin Yang. Sunkavalli, William T. Freeman, and Hao Tan. Test-time training done right, 2025. URL
Kalyan
wttpg///////......///// 23884.
17

---

Published as a conference paper at ICLR 2026
LLM Usage. We utilized Large Language Models to polish the writing in our submission well generate latex code for formatting tables and figures.
A RELATED WORK
A.1 LINEAR-TIME SEQUENCE MIXERS
A growing body of work seeks to replace the quadratic softmax-based attention mechanism (Vaswani et al., 2017: Bahdanau et al., 2014) with linear runtime alternatives. Prominent approaches can be categorized under three broad frameworks: linear attention, test-time training. and state space models. Many nascent linear attention models aimed to approximate softmax attention through kernel feature maps (Katharopoulos et al., 2020; Choromanski et al., 2022). while recent models have discarded the feature maps for raw dot-products between queries and keys, modulated by decays or masks (Sun et al., 2023; Yang et al., 2024). More recently, fast-weight programmers Schlag al. (2021) that modulate the state memory with key-value pairs have also fallen under the umbrella term "linear attention." " Yang et al. (2025c;d) originated from this line of work and enhanced traditional linear attention by replacing the additive memory update with delta-rule recurrence. This has further spurred on a host of work improving the efficiency and capabilities of linear models built on the delta rule (Kimi Team al., 2025; Hu et al., 2025).
distinct line of test-time training (TTT) work views sequence modeling as online learning task during infer- ence. Here, the recurrent "state" is implemented a set of weights (c.g.. linear layer or MLP) that are updated across time-steps to minimize a sulf-super inner-loop loss on the observed sequence (Sun et al., 2025). The entire TTT layer then be viewed as meta-leaming, bi-level optimization, scheme where the inner loop updates these fast weights while the outer loop trains the larger, encompassing network. Recent work have im- proved such layer's hardware efficiency (Zhang al., 2025) and training regime (Tandon et al., 2025).
State space models (SSMs) provide linear-time sequence mixing through explicit dynamical states and efficient scan or convolution implementations, offering significant computational advantages over quadratic-ting titeraatictimeeeee attention mechanisms. Classical linear-time time invariant (LTI) SSM layers utilized structured state transition matrices, c.g., diagonal or low-rank plus diagonal, to facilitate efficient computation and stable learning of long-contex: tasks (Gu et al., 2022a; Smith et al., 2023; Gupta et al., 2022). The introduction of time-varying, input-depnndent selectivity to SSMs in Mamba-1 1 (Gu & Dao, 2024) reduced the disparity modalities, notably language modeling. The
between self-attention and linear models on information model's success, coupled with its simplicity in using real values, led the decline of complex-valued SSMs, which were common in LTI settings. Subsequently, Mamba-2 (Dao & Gu, 2024) formalized the connection between SSMs and (linear) attention through the structured state space duality (SSD).
A.2 STATE TRACKING IN COMPLEX STATE SPACE MODELS
Expressivity and State Tracking. Recent work characterizes the types of state that recurrent, constant- memory mixers can maintain, revealing algorithmic deficiencies in previous SSM-based models. Merrill et al. (2025) show that under finite precision, practical SSMs collapse to TC leading to failures on tasks like permutation composition over S3 unless the primitive is extended. Similarly, Yu & Erichson (2025) prove that a single-layer Mamba is not a universal approximator. Several modifications have been proposed to improve expressivity. For instance, the same work shows that a block-biased variant regains the universal approximation property with only minor changes, either through block decomposition or channel-specific bias. Allowing negative eigenvalues non- non-triangular transitions enables linear CNNs-inncudee diagonal and Houscholder/DeltaNeee forms to capture parity and, under mild assumptions, regular languages (Grazzi et al., 2025). Complex- complex-valued parameterizations provide another avenue for enhanced expressivity.
A.3 MULTI-INPUT, MULTI-OUTPUS
S4 (Gu al., 2022a) is a single-input, single-output LTI system where each dimension of the input was assigned its own independent SSM. $5 (Smith et al., 2023) replaced the set of SISO SSMs with a multi-input, multi-output SSM which applied directly on the entire vectorized input. This change reduced the effective state capacity but enabled the use of efficient parallel scans, forgoing the convolutional and frequency-bast based approaches in S4, improving pretraining speeds. While this trade-off between state capacity and modeling performance was less pronounced in LTT models, Mamba-1 1 (S6) (Gu & Dao, 2024) and Mamba-2 (Dao & Gu, 2024) returned to the SISO system due to the importance of a large state size expansion in the time-varying setting. The computational bottleneck associated with the increased state size was addressed with a parallel-scan, hardware-awe aware algorithm for Mamba-1 1 and a matrix multiplication basse algorithm for Mamba-2. Unlike S5, Mamba-3's 's MIMO structure is motivated by inference-first perspective: to improve arithmetic intensity during the semory-bbund decoding process. Accordingly, its state expansion was kept the Mamba- 1/-2 levels to maintain modeling capabilities the cost of additional training compute.
18

---

Published as a conference paper at ICLR 2026
Table Table of canonical linear-time invariant discretizations (top) and custom linear-time varying discretizations derived from our exponential-adjuste framework (bottom), along with their appearance in structured SSMs used in deep leamnng. Our framework formalizes the prior Mamba discretization method. Discretization
exponential-Fubl and extends it with the more expressive exponential-trapezoiiee
methods convert the continuous SSM h(t) (t)) + ((t)x(() into the discrete recurrence h; 1 B,Tr.
Discretization Method
9c
Tt
Appearance
Forward Fuler
1+AA
Backward Euler
-I-AA)-
(I- (I-AA)'A
Trapezoidal
(1- (1++))
(1- (1-2A)-1 3A
S4 S4D,
Zero-Order Hold
exp(AA)
A (exp(AA)-
Zero-Order Hold
exp(ArAA)
A (exp(A.A.)
Exponential-Eube
exp(A.A.)
A.
Mamba-1, -2
Exponential-Trre TTapezziial
exp(A,AA)
(1 (xx))A))))(AAA)))
A,
Mamha-3
Moreover, the generalization from ouler-produuctbasse state update to malrix-product-based recovers a key expressive feature of SSMs classical literature, and while there has been explorations of replacing SISO with MIMO SSMs (Smith et al., 2023), they have been limited to the LTI setting and were primarily inspired by training. not inference, efficiency. Section discusses the history and motivations of SISO and MIMO SSM systems in greater detail.
B AAPONENTIAL-TRAPPZZOO
DISCRETIZATION
Proposition 5 (Variation of Constants (Tenenbaum Pollard, 1985)), Consider the linear SSM
Aa(t)=((th((t) B(t)x(t),
where (t)) R.N, A(L) Th is scalar decay, and B(()x(()) R.N For A, discretized time grid Tt-1 + Do the hidden state satisfies Equation (9), which can then be approximated to Equation (10) with (A)) error. The approximation of the remaining integral the system input can have varying error bounds depending on the method used: an example can be found in Section B.3.
(TT.)=xx) ( A(s)ds h(T exp (fp [[A())ds) B(r)x(f)dr,
(
(9)
14 AeA m- I do d B(T))((())rrr
(10)
Temi
Proof. Starting from the initial linear SSM, an integrating factor =(t)= = A(e)ds is applied to facilitate integration.
Keeping in mind =((f)=A((t)<ff through rearranging the terms and integrating between the time grid [7-11.]]
(=(T)((T))d B(T))(r))())ddr
results in
(() Ti-1
(()))(()((T)drr,
which can be arranged in more familiar form
h(TT)==(()))) (((-)))))(( f (()))
Substituting the integrating factor 2(7) corresponds to
A(s)ds h(T:-))+ / exp (spp[["" A(s)ds) B(r)x((Tdr..
h(T) exp
19

---

Published as a conference paper at ICLR 2026
We sample the continuous underlying signal with right-hold assumption where V's ( A(()) which we refer as Ac.
{"
h = exp(
exp((n-7)A)B((()((>)))))
solved analytically
be approximated
incurring a local truncation enor of order (A))). Thus, we have solved the exponential dynamics the underlying ODE analytically and leave the system input integral be approximated with any host of methods.
The same result can be shown through the Variation of Constants method.
Since A(t) is scalar, the homogeneous system h(t) = A(t)h(() has solution
h(t)= ((t,s)=exp (%. A(E)dE
The Variation of Constants formula gives us,
h(1) h()))((())(()) J. d((T))(())((TT)))
Setting (tt-==(t-1,f4) yields the exact h, given h We approximate S" A(E)dE by setting A(T) x2 Ak over [(x-1,/4), which gives us,
o(ta ((tz:t=-))=cr) [[AeSha exxp (f.A A
Substituting these approximations the Variation of Constants integral, we get the approximation
h, @ 1 c(n=r)4bB(77((7dd
B(T)x(T)dr.
Tc-1
B. RECOVERING PAST MAMBA LTV DISCRETIZATIONS
ZOH. The classical zero order hold discretization method (Table 5) can be derived by holding (() as constants over the interval 1, where the values are fixed to the right endpoint 7
Exponential-Enler (Mamba-1/-2) 1/-2). We can recover Mamba- 1/2 discretization by approximating the state input integral with Euler's rule (Süli & Mayers, 2003) and holding the (right) endpoint constant throughout the interval (Fig. 1)
h = c AcAi h4 (Fr 7- 1) -T) B
-
= e A.Ai A4 B,
(11)
We call equation (11) the exponential-Euue Euler discretization method, stemming from the exponential integration followed by Euler approximation.
B.2 EXPONENTIAL TRAPEZOIDAL DISCRETIZATION MASK MATRIX
7 (700 a2(Nua +181)
(1102+ 72
(12)
Lor LorT..27000 +31)
1
70
0:
81
7/1
Or201
002
1
0
82
7/2
(13)
Cr....
TT
Proof. When viewing the tensor contraction form, let us call C= C= based on the Mamba-2 paper. With this decomposition our mask, we can view /.=comtrac(77722- (Ta
20

---

Published as a conference paper at ICLR 2026
The original contraction can be seen Oonnc(/ZN.SNTSSS-PPPICLLLLL
We can now view it Cottac/7NNSSY1...5--7771CC--.L This can be broken into the following:
Z = contoc((NNNP
Z' (00t
co= (T.0DO00((L../PPPXXXPXP
Y (
We can view this step: Contract(SS,SAi (2NNG(((SS.NN-ZZ)))22222) as convolution of size two applied on the state-input (B,X outer product) prior to the decay with the traditional SSD -= matrix.
B.3 CIOONENTIAL-RRPEEZOONN
DISCRETIZATION ERROR RATE
Standard assumptions. We assume that: ((),,B(0,a(() are bounded and C on each timestep, that g(T) has two bounded derivatives; the map h- A(t)h+ (())a(() is Lipschitz which true for linear systems; De lies in a bounded interval so that the update is zero-stable.
Proof. Let g(T) eltk eltte-)/As T) ((T)x(T) denote the integrand in the second term of Proposition 5. Since A()), are C2 on [4)- the function g has two bounded derivatives. A second-order Taylor expansion of around 4 gives us,
A o ((++1)) (((-)))0((1))
g(r) g(r)dr=
Recall that the trapezoidal approximation to this integral is given by,
Qx A (1->e))(t1=)) )e)g((z=1)) Neg(tts))
Expanding g(t²) using Taylor expansion: g(tx) g(tk-1) + Dig Ai d'(=t)))) + 0(43). Substituting this into Qx.
Q Q - ((1-)))g(tt=1)++t
- (14- + A3
(1a-
Hence, the error is given by:
t
d( (+-> A (tx-1) - ) (1x (t=))()))))
Under the assumption that A = +c, where c = 0(1), then 1 - A -aA =0(A,) and thus the A? term is O(A)). Therefore,
-(7)dr-
which yields an O(A)) local truncation error. Since the update h4 e Aedi h4-1 + Qx is linear and zero-stable for bounded A,, standard numerical ODE results imply an ( global error.
B.3.1 PARALLEI REPRESENTATION OF CRENNENNTAAITTRAPEOOOOO
RECURRENCE
Our recurrence be instantiated case of SSD and has corresponding parallel form to Equation (2). Expanding the state recurrence from hg = "o Bo results in hT= aT... (700) + +... Br where the SSM output yr = oT.... 27000 + 3 C, B5 + + 7T C BTAT. Unrolling these rows shows that the mask induced by the trapezoidal update is no longer a fixed averaging of endpoints (as in the classical trapezoid rule), but date-deenndnnt convex combination of the two interval endpoints.
Under the SSD framework (2) with parallel form Y = =LLCBB XX, Mamba-3 corresponds to mask L whose structure I-semiseparable matrix composed with a 2-band matrix (13).* This formulation leads to
"ncidentally, this is special of 2-semiseparable matrix.
21

---

Published as a conference paper at ICLR 2026
hardwwre-efffcient matmull-asect calculation of the SSM output. Finally, note that the convolutional connec- tion of Mamba-3 can be seen through this parallel dual form, where multiplication by the 2-band matrix in equa- tion (13) represents convolution with weights B,y. In Section B.2 we the SSD tensor contraction machinery to prove that the parallel form is equivalent to a vanilla SSM with a convolution on the state-input.
Remark 5, The structured mask of Mamba-3 can be viewed as generalizing Mamba-2, which instcad of the 2-band matrix has diagonal matrix with " only (13).
B.4 EXPONENTIAL
PARAMETHRIZATION
Table 6: Ablations on A parameterization the exponential-trappeoidda update.
Parameterization
Form of ppl +
Default
((u)
15.72 $
Fixed 1/2
15.76
No trapezoidal (Euler)
15.81
Setting: All the Mamba-3 (SISO) 440M model trained at Chinchilla scale, with the other architectural and optimization hyperparameters being the same as in Table 1.
The default model uses data-dependent gate A = (we). where is learned projection of the current imput token. In Table 6, we try different parameterizations for A and find that the default parameterizationn empirically performs the best. Hence choose the simpler default parameterization that does not enforce
the O + +A).
C
COMPLEX SSM PROOFS
C.I PROOF OF PROPOSITION 2
Proposition 2 (Complex-to-Pe) Real SSM Equivalence). Consider complex-vaued SSM
hi(t) Diag
(5)
v(t) Re (C(()+iC(() h(t)).
where h(t) E N/22, ((t), B(t, C(t), C(t) N/2 and A(t) E IR. Under exponential- discretization, this system is equivalent real-valued SSM
h Ards R, B,
(6)
y1- C h
with state h, E projections
B₁ B.
C Ct
B.
ER-V
C
ER-
and a transition matrix
R == Block (R(A,0(())
NRN
R(0)= = cos(O) (0
sin cos(0)
Proof. We first present the derivation for N = 2: the block-diagonal structure for general even N follows by grouping pairs of coordinates.
Let h2 ih, denote the complexified hidden state, with parameters A(t)+ + +0((t) and ((t)+iB(t) for the transition and input, respectively. By the variation of constants formula (Proposition 5), applying zere-order hold and Euler's rule over step [f gives
hu 1 thg A. 10.)
Expanding the exponential,
(((A++)0))) = A
so in real coordinates h4
ER2 the recurrence becomes
fic]
A
h = A.A.
Tcos/ 1,0,)
-in(A sin( (,,0)) cos( (,))))
h + A
B B
It-
n(A,)))
22

---

Published as a conference paper at ICLR 2026
Stacking across 2/2 such pairs yields the block-diagonal transition
B B.
h:= Block BlokkAAA)
A
Tt.
For the output,
1/1 Re Re (e(((((+iC))) The (hu+shhe)
C₁ -
h+
which defines the real projection C, E. in the proposition. This proves the equivalence between complex SSM and the real block-diagonal system with rotations.
C.2 PROOF OF PROPOSITION 3
Proposition 3 (Complex SSM, Data- Sett--Deeedent RoPE Equivalence). Under the notation established in Proposition 2, consider the real SSM defined in equation (6) unrolled for I time-steps. The output of the above SSM is equivalent to that of a vanilla scalar transition matrix-based SSM (11) with data- dependepn rotary embedding applied on the B,C components of the SSM, defined by:
[
h h h
II
B
y1
R
C
he
(7)
where the matrix product represents right matrix multiplication, e.g., IT. usage of a transformed real-valued SSM to compute the complex SSM as the "RoPE trick."
Rg We refer the
Proof. Consider the SSM
h4 = AeA: A.AAAA R,h,-1 + Brst Vr C h+:
(14)
where (as in Proposition 3) A E IR is scalar (so that AAA h is a scalar and commutes with rotations), and R2 is block-diagonal orthogonal/unitary, hence R," R,
Unrolling the recurrence with the convention that empty product is the identity,
h4
II A Ru
B
(15)
Thus
C
h4 ΣCT
II s=1t1
caids Rs
B
(16)
1=0
Using its unitary property,
II R s=i+1
I
R.
R
R
Rs
I
R
s=0
e==0
Since AA,4, A, are scalars, they commute with rotations; hence
i IIRR
Ut
Σ
c
Il
R
A.A.
B
(17)
1=0
[
R
C.
[
i
Bar
(18)
Define the rotated parameters C = RIT R,
C, and B;
(Ii'o R.
B. Then
y/ = Σ
A,A,
B
(19)
Equivalently, introducing the rotated state he = RT
h
h, A F2-1 + Bitti
81 = - ho
(20)
n
23

---

Published as a conference paper at ICLR 2026
C.3 PROOF OF PROPOSITION 4
Proposition 4 (Rotary Embedding Equivalence with Exponential Trapezoidal Discretization).. Discretizing a complex SSM with the exponential-trapeen trapezoidal rule (Proposition 1) yields the recurrence
ExponentialTrapeezddda
h; - +
II 300
B BI-1
R k-
B
[ +==0
y/
C₁
h4-
(8)
Here R₂ is the block diakonaa rotation matrix defined in Proposition 3.
Proof. We begin from the complex SSM (as in Prop. 2)
- = Ding( (())) + (B()+:B(1))x0) x())
s(t) = ((C(())++))() hht)).
where A(t) E is scalar and (11),B()BB((,CC,,C)))))
Recall from Prop. 5,
h m c ((Aetii)) h- f c (B())+BB(T) x
TC-1
Applying Prop. the above integral, get
h - 10))).11)1111 h4-1
0,0 (B.1+iBL-) + BB.+iB.)
(21)
where
at a (I= m:
Since (A.(A = anee and shown in Prop. 2, multiplication by eidi is block-diagonal rotation in real coordinates, we get the real N-dimensional recurrence
h. = =o; R,h R, BR 18 B y1 C h
(22)
cos8 -in0 sinf
where R, Blook (R(N,,RRA,,)))) (R(A, N/2 where R(0)= =
and projections
costl
B,
C. C₁
B.=
C
=
Note that R2 is orthogonal, R2 R2
B.
We define the following.
1
h =
I R
ho
B,
R B.,
C-==
R
Cr.
1=0
seft-multiplying equation (22) by II
R and using R, R. I,
h; = h4 B B
5/ = h
This is vanilla ocalar-transition SSM with data-dependent rotary embeddings absorbed into B,C via cumulative products of R.
24

---

Published as a conference paper at ICLR 2026
linear projection
Sequence transformation
SSM
SSM
MIMO projection (aptional)
RoPE
Nonlinearity (activation,
Conv
normalization, multiplication, etc.)
Mamba-2 Block
Mamba-3 Block
Figure 3: Contrasting Mamba-2 and Mamba-3 Architectures: Key updates include exponential-trapezoiiie discretization, data-dependent RoPE embeddings, MIMO projections, QK normalization, and learnable biases.
Table 7: Arithmetic Intensity for (a) SISO, (b) MIMO. The batch and head dimensions cancel out. The arithmetic intensity of MIMO increases linearly with rank R, enabling better hardware utilization during memory-bound phases like decode. Here N is the state size (expansion factor) and P is the head dimension. For Mamba-3, typically R N.P.P.
Input
Output
FLOPs
Arithmetic Intensity
Input
Output
FLOPs
Arithmetic Intensity
(N.PN (P:(P)
5NP- P
SI+2N+74NT
6NP-P P
H. (NNPP
(c:(P.R)
4N NP- PR
ANPR NF PR 11+2N144144
21 (P)
2
w:: (P.R)
@(min L.min(N.P.R) P.R))
:x:(1) (1)
ac: dc:(1) (1)
(
(N)
by: (N,R
4: (N(
0: (N,R)
(a) SISO (2-byte data).
(b) MIMO (2-byte data).
D MIMO FOR MAMBA-3
Training MIMO SSMs. While the MIMO formulation is motivated by inference efficiency, the training algorithms for SSMs (including our developments in Section 3.1, Section 3.2) are generally developed for SISO models. We note that MIMO SSMs can be expressed in terms of R² SISO SSMs, where R SISO SSMs sharing the same recurrence are summed for each of the R MIMO outputs. particular, if we define
( ERN BB() E ERN 2
(i) ER, where i.j E (O.....R-1)) then we have
R-1
h + aath (,," x1)
h
Σ
h
(23)
3=
u ( - (c!:)) T h
(c('t)
(24)
Consequently, improvements to standard SISO-based SSM models can be directly applied to MIMO models the underfying SISO training algorithms be utilized black-box. The general sequence-to-sequee to-sequence formulation requires calling the SISO algorithm Ri2 times as a black box. On the other hand, in the recurrent form the computations for equation (23) and equation (24) can be performed sequentially and incurs an R- R-times overhead (recall the discussion on MIMO decoding FLOPs).
Chunked Algorithm for MIMO SSMs. Many modern SISO recurrent models including Mamba-2 are computed using a parallelized chunk-wise algorithm, where the sequence is divided into chunks of length C. and parallelizable (but asymptotically slower) algorithm is within each chunk, while recurrence
25

---

Published as a conference paper at ICLR 2026
is computed between chunks. In these cases, we can reduce the R increase in compute to factor of R by exploiting the structure of the chunked algorithm in a more line-grained way.
All instantiations of state space duality (SSD) for example have chunked algorithms using the quadratic dual algorithm intra-chunk. Dao & Gu (2024) showed that the case when the state dimension and head dimension are similar, the chunk size can also be set to the same value and the overall algorithm has linear cost. More precisely, the intra-chunk computation incurs 2- C computation incurs 4- I PC FLOPs (ignoring negligible terms). Setting C= P= N yields a total FLOP count of 8T'22
FLOPs and inter-chunk
The chunked algorithm for SSD can be naturally generalized into MIMO SSMs. In such case, the intra-chunk computation incurs 2. 2 (CR) N c PN) FLOPs and inter-chunk computation incurs & (NN((CR) FLOPs. Thus, setting CR = N = P yields a total FLOP count of 8TRN?, an R-fold
4. T. increase in FLOP count. Intuitively, setting MIMO chunk size π. times the SISO chunk size, i.e. CMIMO + R CSISo, maintains the SISO intra-chunk FL.OP count while increasing the number of chunks by a factor of R, resulting in an overall R-times increase in FL.OP count.
The training speed of algorithms in practice depend on details of the kernel implementation strategy. architectural choices such as how the MIMO parameters are instantiated, and problem dimensions, but should be no more than It times slower. Our released Triton Mamba-3 SISO kemelss are roughly par with the Triton Mamba-2 kemels, and MIMO kemels incur a slowdown. Table 4 benchmarks the prefill speed of various kemels which is proxy for the forward pass of the training kemel..
MIMO Instantiation. Among various choices for MIMO parameterizaatons, Mamba-3's 's approach achieves a balance that preserves the state size and number of SSMs of its SISO counterpart, while avoiding excessive growth in parameter count. The naive conversion of SISO SSM to a rank R MIMO SSM would incur a Rx increase in parameters as all projections that model the inputs to the SSM, B,C,a, would increase. Block-level components, such as the gate Z (which so far has been ignored for simplicity) and output y projection would also be impacted. This influx in parameter count would be intractable at larger model scales, To counteract this, we make the following change. Mamba's 's multi-valus attention (MVA) head structure results in shared B B, across heads, so these component's 's projections can be directly converted to incorporate the new MIMO rank I with only a slight increase in parameter count from DN to DNR for the entire layer. However, the SSM input Ft, output yt. and gate 2; are unique per head and therefore dominate the parameter count. Here, directly adjusting the projections would increase the parameter count from DP to DPR for each head. Instead, we keep the original SISO projection and clement- wise scale each dimension of the projected output to size R with a learnable, data-independeen vector, resulting in DP+ PP+RR parameters for each head. This mitigates the multiplicative increase Section D details the parameterization, and all MIMO-variants in our paper are garameter-matcher to their SISO counterparts by reducing the MLP width.
reasonable additive parameter count increase.
MIMO Formal Parameterization. With a given batch, head, and sequence position consider the input U.ERD. P. Also denote P,řEN as the head dimension and MIMO rank, respectively. We first obtain SSM parameters via a set of projections defined in terms of tensor contraction notation as follows:
B. = D0=ottn((NDDDDDDNBBVMV
C
X = conttottttt//11 contract(PD,DD---- PD,D- P).x.
X, contract contract(P -
where WB, W., x Wx are model parameters. Additionally, we obtain the residual gate term Z, in the same manner as X, with weights Wz, and Wz to temper the increase in the parameter count. The state update and the SSM output are then computed via the following MIMO SSM:
H = H.H + B, ERNXP, Intermediate output Yú is obtained by the residual function o, Y' + o(Y,, where d(Y, Z,) = Y, o SiLU(Z.) in our case. Finally, the layer output O E RD is computed via the following down projections:
HHH, PXR
D.=contractPRR.PR PPR.P..RPPPPPP-.P... P)o,,,
O, =
D)
This formulation enhances the existing Mamba-3 architecture by providing a lightweight parameterization that transforms the set of independent SISO SSMs within each head into a set of MIMO SSMs. Here, note that the hardware-efficieer chunking technique employed by Mamba-2 for pretraining can be applied with little change, as the decay factor or is tied across the MIMO dimension
26

---

Published as a conference paper at ICLR 2026
MIMO Parameter Matching. The MIMO variant of Mamba3 incurs additional parameters compared to its SISO counterpart. We therefore reduce the hidden dimension of the MLP layers to parameter match the SISO variants as follows:
Model
180M
440M
880M
1.5B
SISO MLP dim
1,500
2,048
3,072
4,096
MIMO MLP dim (R=4)
1,264
1,792
2,800
3,824
Remark 6. For simplicity, all discussion in this section was for simpler 2-term recurrences such that arising from exponential-Eutte discretization; the generalization to the 3-term similar.
recurrence
E EXPERIMENTAL DETAILS
Language Modeling, Our pretraining procedures follow that Dao Gu (2024)'s section D.2. All models each scale follow the same procedure and were trained with bfloat16. The Mamba family of models were trained using the standard expand factor of and a dstate of 128 and head dimension of 64. The Transformer baselines follows Dao & Gu (2024), and the GDN baselines follow (Yang et al., 2025c) where q. q.Kai 128, Udim = 256. We utilize the Llama-3.1 tokenizer (Grattafiori et al., 2024) for all models.
We utilize LM Evaluation Harness (Gao et al., 2024) to test the zero-shot languag modeling capabilities of our pretrained model on LAMBADA (OpenAl version) (Paperno et al., 2016), HellaSwag (Zellers al., 2019), PIQA (Bisk et al., 2019), ct al., 2019), and OpenBookQA (Mihaylov et al., 2018).
(Clark et al., 2018), WinoGrande (Sakaguchi
Real-World and Synthetic Retrieval. For our real-world retrieval tasks, we evaluate on the common suite consisting of SWDE (Arora et al., 2025b), SQUAD (Rajpurkar al., 2018), FDA (Arora et al., 2025b), TriviaQA (Joshi et al., 2017), NQ (Kwiatkowski et al., 2019), and DROP (Dua et al., 2019). We utilize aforementioned tasks provided by Arora et (2025b; 2024), as original datasets are in a muestion-answering format, making it challenge for solely pretrained models. All tasks were truncated match the training context length. The synthetic NIAH tasks (Hsieh et al., 2024) were also run with LM Evaluation Harness.
cloze-formatted version
State-Tracking Synthetics. Training follows a sequence length curriculum that the minimum length to 3 and progresses the maximum length from 40 to 160. Final models are evaluated at 256 length. Each curriculum runs for 10 steps with batch size 256. We use layer models for Parity and layer models for Modular-arithmett tasks. The state size is chosen be 64, and sweep dmodet E (32,64) and 8 learning rates logarithmically spaced between 10-4 and 10-2, reporting the best validation accuracy.
F ADDITIONAL EXPERIMENTAL. RESULTS
Context Length Extrapolation
length
10.8
Gated DeltaNet
Mamba-2
Mamba-3
Perplexity
10.4
10.2
10.0
1K
16K
Context length
Figure 4: Pretrained 1 1.5B models' performance on the held-out FineWeb-FF test set at varying context lengths. Mamba-3 exhibits strong length extrapolation while Mamba-2 falters at longer contexts.
27

---

Published as conference paper at ICLR 2026
Table 8: Ablations of optional norm type (grouped default) and placement (pre- post-gate) on pretrained hybrid Mamba-3 SISO models at the 1.5B scale. All models have BCNorm. No additional norm demonstrates the strongest in-context retrieval performance average, while pre-gate, grouped RMS results in the best performance on synthetic retrieval, especially on lengths longer than its training context.
Manio-33 Norm Type
Avg.
SWDB
SQD.
FDA
TQA
NQ
Drup
NIAH Singl-1
NIAII-Singk-22
NIAII-Singk-3
Conical Length
20188
1021
2018
1056
1021
2018
4096
1021
2018
4096
No Norm
66.4
58.5
65.9
64.8
100.0
36.2
100.0
100.0
100.0
8.8
40.2
Pre Gate Defacit RMS
55.9
55.4
40.9
67.3
65.4
33.0
28.
100.0
100.0
86.2
100.0
100.0
97.8
99.2
97.8
90.2
Poo--Glt Gate Grouped RMS
58.2
51.4
46.7
56.8
61.
30.4
27.6
100.0
100.0
79.1
100.0
100.0
65.8
93.8
97.0
9.6
Pre-Gate Grouped RMS
56.1
68.6
47.3
02.4
65.7
33.3
28.5
100.0
100.0
100.0
100.0
100.0
96.
99.8
97.2
36.8
Validation Perplexity for 18 Pretraining Runs
13.5
13.0
Perplexity
11.5
11.0
GetedDellaNel
Mamba-2
Mamba-3 SISO
10.5
Mamba-3
20000
3000O
40000
50000
60000
70000
80000
90000
Global
Figure 5: Mamba-3 demonstrates better pretraining performance compared to strong baselines like Mamba-2 and Gated Deltanet. These are validation perplexity on FineWeb-Fide of our fully pretrained 1.5B models.
We also compare the effectiveness state size usage Mamba variants to a Gated DeltaNet baseline in Figure 6. We highlight the difficulty of directly comparing GDN versus Mamba-style models due to the differing head structure (multi-head for GDN compared multi-value for Mamba). Our experiments hold GDN's Veapond to 2 and decrease the head dimension accordingly to vary the relative total state size. Similar to Figure 2, we train 440M models to 2x Chinchilla tokens (40x token- to-n-tt-ote to-parameter ratio) and sweep across dauae {32,64, 6,, 128) for the Mamba models and dhead dim (32,64, 62288 for GDN. We parameter match all models.
Relative Total State Size vs Pretraining Perplexity
Pretraining Perplexity
15.0
Mamba- 2
14.9
Mamba-3
Mamba-3 MIMO
14.8
Gated DeltaNet
14.7
14.6
14.5
10a
Relative Total State Size
Figure 6: Exploration of state size (inference speed proxy) versus pretraining perplexity (performance proxy). Mamba-3 and Mamba-3 MIMO continue set the Pareto frontier.
28

---

Published as a conference paper at ICLR 2026
G ARCHITECTURE ABLATIONS
We explore model architecture's 's ablation in this section. All models trained at the 440M scale to Chinchilla optimal number tokens (20x tokens to parameters) with the same experimental procedures our pretrained models covered in Section E unless otherwise stated.
B,C Bias Parameterization. The Mamba-3 model's separate B and C biases are -peeciicciicc and channel-wise and added to both B and C' after the QK-Norm. While the biases in the final Mamba-3 model are trainable, data-independent parameters and initialized to all ones, we explore various bias parameterizations in Table 9a. We find our models are not very sensitive to the initialization of the biases as long as they are positive. We choose the all all-ones initialization due to it's simplicity.
We also explore the impact removing the B or C' bias on performance in Table 9b (bias is initialized with our default parameterization when utilized). Unlike in Yu & Erichson (2025), which finds that B bias by itself is able to improve performance on Mamba-1, our experiments find that only having B bias hurts performance slightly and that B and biases have synergetic properties.
Bias Init.
Trainable
ppl |
B Bias
C Bias
ppl 4
1.0
15.72
16.52 x
0.0
16.57 X 15.80
x 16.68
1.0
15.98
4(0.1)
15.76
15.69
u(-1,1) V
16.07
(b) Applying bias to both B and C/ leads to the best performance. Only applying B bias (Block-Biased (Yu Erichson, 2025) Mamba-3 variant) does not provide significant gains over the no-bias baseline.
(a) Effect of parameterization the and C' bias on model perfonmance, measured by pretraining perplexity. We find our default initialization of oll-ones (first row) provides the best performance, but performance not sensitive as long as hiases positive.
Table 9: Ablations on B,C bias initialization (left) and presence (right) for Mamba-3.
H INFERENCE KERNEL LATENCY ANALYSIS
H.1 KERNEL IMPLEMENTATIONS AND FUSION STRUCTURE
In Table we detail the DSL (Triton, TileLang, CuTe, PoTorh)) and the fusion level of the kernels used in our latency analysis. For Mamba-2 and Gated DeltaNet (GDN), we directly use the publicly released Triton kernels from the respective authors. For Mamba-3, we implement new inference kemnlss with a comparable fusion structure: the forward SISO uses Triton kernel fused with rotary position embeddings and the forward MIMO uses TileLang kemne with the same fusion level while the decode path uses a CuTe kernel fused with gating and MIMO projection.
In Tables and 11, we abbreviate IP input projection, Conv 1D convolution, Gate gating, OP- output projection. Colors indicate implementation backend (Torch, Triton, TileLang, CuTe).
Table 10: Kemel DSL and fusion structure for forward (prefill) kernels.
Model (Forward)
Kernel DSL
Fusion Level
Mamba-2
Triton
IP, Conv, SSM, Gate+OP
Gated DeltaNet
Triton
IP, Conv, Chunked Delta, Gate+OP
Mamba-3 (SISO)
Triton
IP, SSM+Rotary, Gate+OP
Mamba-3 (MIMO)
TileLang
IP, SSM+Rotary, Gate+OP
H.2 EXTENDED PREFILL AND PREFILL+DECODE LATENCY MEASUREMENTS
Models. We benchmark Mamba-3 1.5B (SISO), Mamba-2 1.5B, Gated DeltaNet (GDN) 1.5B, and a strong Transformer baseline implemented via the vLLM engine (v0.11.0) with Llama-3.2 IB 5 All recurrent models are trained at the 1.5B scale with drwdd 2048 and 24 layers. For Mamba variants we set state size as 128 and head dimension 64; for GDN we use QK head dimension as 128.
'httpa: www.nngginngaacccoooo com ama /11 ama 3.22-1 18
29

---

Published as a conference paper at ICLR 2026
Table 11: Kernel DSL and fusion structure for decode kemels.
Model (Decode)
Kernel DSL
Fusion Level
Mamba-2
Triton
IP, Cony, SSM, Gate+OP
Gated DeltaNet
Triton
IP, Cony, Recurrent Delta, Gate+OP
Mamba-3 (SISO)
CuTe Triton
IP, Rotary, SSM+Gate, OP
Mamba-3 (MIMO)
CuTT Triton
IP, Rotary, MMMGa OP
Setting. Sequence lengths were swept over Le (512.0002222244444666.66885 for prefill, with an equal
number of tokens decoded. For sequence lengths (512, 1024, 2046) we use batch size of 128; for sequence lengths (16384), we use batch size of 16. We use single H100-SXM 80GB GPU and report wall-clock times (in seconds) over 3 repetitions.
Table 12: Prefill and Prefill+Deoode latency across sequence lengths. Mamba-3 adds minimal overhead its forward-pass and retains competitive decode latencies. Details in Section H.
Model
512 tokens
1024 tokens
2048 tokens
40% tokens
16384 tokens
Prefill
Prefill-Dec
Prefill
Prefill-Dec
Prefill
Prefill-Dec
Prefill
Prefill-Dec
Prefill
Prefill+Dec
vLLM (Llama-3.2-1B)
4.45
1,08
2.08
122.06
Gated DeltaNet
0.48
4.52
0.95
2.04
1.90
18.07
3.79
36.14
1.91
71.66
Mamba-2
4.62
0.96
1.91
18.48
3.81
36.94
1.92
57.90
Mamba-3 3 (SISO)
0.48
4.36
0.95
8.70
1.90
17.41
3.80
34,83
1.91
54.79
Mambu-3 (MIMO rm4)
0.58
4.70
1.15
940
2.30
18.82
4.55
37.57
2.36
62.88
We observe that (i) Mamba-3 adds minimal forward-pass. cost, showing that the exponential-traeezoidee update, complex state tracking, and MIMO parameterization remain lightweight; (ii) decode latency is
competitive across recurrent models; and (iii) recurrent mixers scale more gently with context length than vLLM 1..ma-3.22-B- which grows much faster with L due to KV-cache overhead.
30


Reviews:
[{"review_id": "BFJlpId6sY", "reviewer": "Reviewer_kFu5", "summary": "This paper introduces Mamba-3, a followup sequence model architecture that builds upon the Mamba family of SSMs. The work is motivated by an \"inference-first\" perspective, aiming to create a model that excels across three axes: quality, capability, and inference efficiency, addressing limitations in prior sub-quadratic models like Mamba-2.  Basically, this paper propose three core, major improvements derived from classical SSM theory and also validate Mamba-3 through extensive experiments, showing that it outperforms strong baselines in language modeling, possesses new state-tracking capabilities on synthetic tasks, and establishes a new Pareto-frontier for performance versus inference cost.   It presents a clear, significant, and well-executed advancement in the design of efficient sequence models. The paper is strong for three main reasons:\n\n1.  **Principled Improvements over existing SSMs/Mamba family:** The core contributions are not arbitrary architectural tweaks but are grounded in control theory and SSM principles. The connection drawn between complex SSMs and data-dependent RoPE is very elegant IMO and provides a clear theoretical justification for the observed gains in capability.\n\n2.  **Holistic Improvement:** The work addresses the multi-faceted challenge of building a practical LLM. It simultaneously pushes forward model quality, unlocks new fundamental capabilities (state-tracking issue which has been rooted in RNNs), and improves hardware-level efficiency(MIMO formulation). This is a rare and impactful combination. (Though it will be more impactful if all artifacts will be open-sourced. \n\n3.  **Compelling and Rigorous Empirical Evidence:** The experimental validation is thorough and persuasive. The head-to-head comparisons on language modeling, the stark success/failure results on formal language tasks, and the Pareto-frontier analysis collectively provide a powerful and convincing case for the superiority of the Mamba-3 design.", "questions": "- The results on retrieval (Table 2) are improved but still trail the Transformer baseline significantly. Do you view this as a fundamental ceiling for fixed-state models, or do you see pathways for future SSM-based architectures to further close this gap? Maybe authors can add some potential exploration about hybrid models? I think we are at the age that Hybrid models are becoming more important and can help alleviate the shortages of RNNs which we should acknowledge and make good use of.\n\n- The \"RoPE trick\" is a very efficient way to implement the complex SSM. Could you briefly comment on how the cumulative product of rotation matrices `(Π R_i)` is handled efficiently during both the parallel training scan and the sequential inference steps?\n\nAdditional questions:\n\n- The MIMO formulation is a compelling idea for improving hardware efficiency. Could you elaborate on the selection of the MIMO rank `r=4`? Have you explored the sensitivity of model performance and inference latency to different values of `r`? Is there a risk of diminishing returns or even performance degradation if `r` becomes too large?\n\n- The ablation in Table 4a convincingly shows that the trapezoidal discretization and BC bias make the short convolution redundant. Can you provide more intuition on *why* this is the case? Does the learned size-two convolution from the trapezoidal rule effectively capture the same local information as a dedicated convolution layer?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Theoretical Elegance and Insight:** The paper's standout strength is its use of classical SSM theory to motivate architectural changes. The derivation of a data-dependent RoPE from a complex state-space is a beautiful and insightful result that bridges two important lines of work.\n\n- **Targeted Problem Solving:** Each of the three methodological changes directly targets a known weakness in prior models. Trapezoidal discretization for expressivity, complexification for state-tracking, and MIMO for hardware utilization. This focused approach makes the paper's narrative clear and its contributions easy to appreciate.\n\n- **Excellent Empirical Validation:** The experiments are top-notch. The Pareto-frontier plot (Figure 3) is a highly effective way to visualize the trade-off between performance and inference cost, clearly showing Mamba-3's dominance. The ablation on state-tracking capabilities (Table 4b) is a decisive demonstration of the complex SSM's impact.\n\n*   **Practicality and Inference-First Focus:** The introduction of the MIMO variant shows a deep understanding of the practical bottlenecks in LLM deployment. By focusing on arithmetic intensity, the authors address a subtle but critical aspect of real-world performance that goes beyond theoretical FLOPs.\n\n-  **High-Quality Presentation:** The paper is extremely well-written, with complex ideas explained clearly and supported by well-designed figures and tables.", "weaknesses": "My major concern is that retrieval Capabilities Still Lag Transformers: While the paper is honest about this, and Mamba-3 shows improvement over Mamba-2, the results in Table 2 confirm that a fundamental gap in retrieval performance versus Transformer models remains. This is an inherent challenge for fixed-state recurrent models and represents a key limitation. Maybe authors can add some potential exploration about hybrid models? I think we are at the age that Hybrid models are becoming more important and can help alleviate the shortages of RNNs which we should acknowledge and make good use of.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FDn90uYjhx", "reviewer": "Reviewer_5Dn9", "summary": "Current State-Space Models have some draw-backs compared to Transformer based models, such as lack certain capabilities, low generalization quality, not hardware-efficient, etc. This paper proposes the Mamba-3 architecture, from inference-first perspective. Specifically, the trapezoidal discretization is used to improve the expressive dynamics, complex-valued state spaces is introduced for state-tracking, and the multi-input multi-output (MIMO) mechanism is proposed for hardware efficiency. Experiments of show that Mamba-3 outperforms Mamba-2 or even transformer based models across 180M to 1.5B parameters.", "questions": "Can the proposed Mamba-3 be generalized to vision tasks? \n\nDoes Mamba-3 still has advantages when compared to sparse attention based transformers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Generalized Trapezoidal Discretization is more accurate than the Euler's rule used in Mamba2, by using the second-order  approximation of the integral. \n2. The complex SSM is novel, which is equivalent to a real SSM with data-dependent rotary embeddings (RoPE). Detailed theoretical analyses are provided.\n3. The MIMO method efficiently solves the I/O problem, which is a key bottleneck for Mamba-2.", "weaknesses": "More complex tasks such as reasoning can be explored to fully demonstrate the capability of Mamba-3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gUkyzxZCN2", "reviewer": "Reviewer_h5BJ", "summary": "Mamba-3 is a state-space model (SSM) designed from an inference-first perspective that addresses the limitations of existing sub-quadratic sequence models (e.g., Mamba-2, Gated DeltaNet) in quality, state-tracking capability, and hardware efficiency. It achieves SOTA performance across language modeling, state-tracking, and retrieval tasks while maintaining linear compute and constant memory, setting a new Pareto frontier for performance-efficiency tradeoffs.", "questions": "1. Complex-Valued SSM & RoPE Equivalence: The paper argues that complex-valued SSMs are equivalent to data-dependent RoPE embeddings and compute-efficient. Could you provide a direct comparison between complex-valued SSMs and *explicitly applying RoPE* (e.g., on B/C projections of Mamba-2) in terms of: (1) training/inference speed, (2) memory usage, and (3) state-tracking performance? \n\n2. MIMO Rank Selection: The MIMO variant uses rank \\( r=4 \\) for 440M models, but there is no analysis of how \\( r \\) impacts performance-efficiency tradeoffs. What is the marginal gain of increasing \\( r \\) (e.g., \\( r=8, 16 \\))? At what point does \\( r \\) become compute-bound (negating efficiency gains)? Additionally, why was \\( r=4 \\) chosen as the default, was it optimized via grid search or heuristic?\n\n\n\n\n3. Ablations. The BC bias synergizes with trapezoidal discretization, but you do not specify its initialization strategy (e.g., zero, Xavier, data-dependent) or whether it is updated during training. Does the bias’s initialization significantly affect performance?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The three core improvements, trapezoidal discretization, complex-valued SSMs, and MIMO formulation, are not isolated tweaks but creative combinations of classical SSM theory and modern LLM needs. Trapezoidal discretization generalizes Euler’s rule (used in Mamba-2) to a second-order accurate recurrence, while the complex-valued SSM recovers rotational dynamics absent in real-valued counterparts.\n\n2. Rigorous Theory & Comprehensive Empirical Validation. The paper includes rigorous proofs for key propositions (e.g., complex-to-real SSM equivalence, trapezoidal discretization error bounds) and formalizes the mathematical underpinnings of each innovation. This theoretical rigor ensures the model’s behavior is well-understood, not just empirically effective.\n\n3. The paper reports detailed experimental settings (training data, tokenization, hyperparameters) and provides clear metrics (perplexity, accuracy, TPS, arithmetic intensity) that enable follow-up work.\nAnd structured presentation & accessible exposition increses the avaliability. \n\n4. Addressing Critical LLM Deployment Pain Points. By pushing the Pareto frontier of performance vs. inference efficiency, Mamba-3 directly addresses a key bottleneck in LLM deployment—scaling inference without prohibitive memory/compute costs. The MIMO variant, in particular, improves hardware utilization without increasing state size, making it viable for real-world applications.", "weaknesses": "1. Inherent limitations of retrieval capabilities and insufficient comparison. Mamba-3 significantly lags behind Transformer in information extraction tasks from semi-structured/unstructured data, and the root causes and potential solutions are not explored in depth.\n\nTable 2 of the paper shows that Mamba-3 with 1.5B parameters has poor accuracy in real-world retrieval tasks such as SWDE and FDA; even in the synthetic \"needle-in-a-haystack\" task, when the context length exceeds 2048, the accuracy of Mamba-3 is still far below the theoretical upper limit of Transformer, and it is not compared with linear models specifically optimized for retrieval (such as Arora et al., 2025b's \"Just Read Twice\").\n\n2. Limited experimental scale of the key variant (MIMO) and incomplete hyperparameter ablation. The MIMO variant was only trained on a 440M parameter model (paper 4.2.3). Because the \"computational constraint\" was not extended to a 1.5B scale, its performance-efficiency tradeoff under larger models could not be verified. Furthermore, the MIMO rank r was fixed at 4, and the impact of different values ​​such as r=2 and 8 on arithmetic strength and inference speed was not tested (Figure 2 only shows the theoretical trend of r, without experimental data).\n\n3. The paper's baseline only includes Mamba-2, Gated DeltaNet, and the standard Transformer, omitting similar linear models such as Rwkv-7, DeltaProduct, and Block-Biased Mamba. These models all innovated on the expressiveness or efficiency of linear models, and some performed exceptionally well on state tracking tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gcdnsFoQQA", "reviewer": "Reviewer_2tn7", "summary": "This paper proposes Mamba-3, an extension of Mamba-2 with a more inference-focused architecture, which introduces three core improvements to its state-space model (SSM) formulation. First, it replaces Euler discretization with a trapezoidal rule, which can offer offers a lower sequence approximation error.  Second, Mamba-3 empowers a complex-valued SSM via a RoPE-like parameterization. The authors actually show the equivalence of complex-valued SSMs to real SSMs with data-dependent rotations on B and C (analogous to keys and values). Third, Mamba-3 introduces a MIMO (multiple-input multiple-output) SSMs that replace single-vector outer products with matrix-matrix updates in order to increase the arithmetic intensity during decoding. The architecture itself includes other modifications, such as QK-style normalization after B and C projections, and removes the short convolution. The paper evaluates 180M-1.5B models trained on 100B FineWeb-Edu tokens (2K context) across language modeling, retrieval (NIAH), formal languages (parity, modular arithmetic), and inference latency. Mamba-3 generally matches or outperforms Mamba-2 and Gated DeltaNet, with notable gains on state-tracking tasks and inference efficiency, particularly with MIMO.", "questions": "- What regularity assumptions on $\\lambda_t(x_t)$ guarantee quadratic global error? \n\n- Does the rotation-absorption argument (Eq. 18–19) hold when $A_t$ is diagonal rather than scalar? \n\n- If $\\lambda_t = \\frac{1}{2}$, the scheme is 2nd order; if $\\lambda_t$ is data-dependent but Lipschitz and satisfies $\\lambda_t = \\frac{1}{2} + O(\\Delta_t)$, it is still 2nd order; otherwise the order collapses to 1---Euler's rule. Is this collapse ensured in Mamba-3?\n\n- Inference implementation details: For Table 3, specify: (a) kernel type (Triton/CUDA) and fusion level for each baseline, (b) exact configuration (batch size, sequence length, heads, dstate), (c) whether all models use comparable optimization levels.\n\n- Did you try evaluating the 1.5B model (trained at 2K) on 8K–32K retrieval tasks, not just LM perplexity extrapolation (Fig. 5)? The LM curve is nice, but the claim of better sequence approximation would be more convincing on long retrieval where the state is actually stressed.\n\n- Since your goal is inference improvements, could you report prefill (1K-4K) and decode (1 token) wallclock-times for a strong transformer baseline using FlashDecoding (a comparable open kernel on the same hardware is also valuable)? Even if Mamba-3 still wins on decode, showing the transformer prefill margin would make the paper easier for readers.\n\n- Hammering on this again, but why SISO for main LM runs? If MIMO is superior at equal state size (per the inference section), why use SISO for 820M/1.5B language modeling?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The three modifications are grounded in classical SSM theory and integrate cleanly into Mamba-2's framework. The generalized trapezoid recurrence (Eq. 3–4) elegantly recovers Mamba-2 as a special case ($λ_t=1$).\n\n- Propositions 2-4 clearly derive how complex SSMs reduce to real SSMs with RoPE-style rotations. This is a clean way to say that \"we can have complex dynamics at only RoPE's computational cost.\"\n\n- The arithmetic intensity analysis (SISO =  $\\theta(1)$, MIMO = $\\theta(r)$) convincingly motivates MIMO for IO-bound decoding, and the matrix-matrix update is of course implementation-friendly.\n\n- The ablation study in Table 4 is quite valuable. It shows that convolutions become optional is. In addition, tthe formal-language ablation where disabling RoPE makes the model underperform directly validates the complex/RoPE mechanism. This is insightful and may open new future research ideas into incorporating other forms of positional encoding into SSMs.\n\n- Training four model sizes (180M–1.5B) on identical 100B token budgets enables fair cross-model comparisons.\n\n- Near-perfect accuracy on parity (~100%) and modular arithmetic (~98%), plus improvements on bracketed tasks, demonstrate that Mamba-3 successfully addresses state-tracking limitations of Mamba-2.", "weaknesses": "- All pretraining uses 2K context, and long-sequence tests mostly stay at 2K (NIAH) or only reach 4K (2x extrapolation). For a model claiming better sequence approximation with the trapezoidal rule, evaluation should stress-test extrapolation at longer scenarios (e.g., 32K), especially since the paper notes performance drops where Gated DeltaNet sometimes wins.\n\n- Table 3 reports single-point latencies without variance. Critically, the authors use \"custom kernels\" for Mamba-3 but \"standard reference\" implementations for baselines—making results difficult to verify. Optimized kernels for competitors could reverse the rankings. The paper must specify kernel implementations, fusion levels, and hardware details. Without this, is very hard to access the real efficiency contributions of Mamba-3.\n\n- Figure 3's missing Mamba-3 MIMO point makes the work seem a bit sloppy. Of course, I think the model will perform well $d_{state}=32$, but still find this unconfortable.\n\n- While the main LM results use SISO, the inference analysis argues for MIMO. In the formal language tables there's only a mention to \"Mamba-3.\", so it is unclear which method was used here. Since MIMO affects FLOPs, parameters, and stability, every benchmark should explicitly state \"SISO\" or \"MIMO (rank=r).\"\n\n- For an \"inference-first\" paper, comparisons should include production-grade transformer inference (FlashDecoding, paged KV cache, vLLM chunked prefill, etc) at realistic batch sizes and sequence lengths. Moreover, only decoding latency is reported, despite prefill being an extremelly important part of inference in LLMs. This omission hides important costs.\n\n- Proposition 1 claims $O(\\Delta_t^2)$ global error with data-dependent $\\lambda_t$ but doesn't state required regularity conditions (e.g., Lipschitz continuity).\n\n- I found the paper to be very condensed. It seems like the distance between letters was reduced to make more things fit into the paper. This make things hard to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)