Mamba Modulation: On the Length Generalization of Mamba Models
Authors: Peng Lu
Jerry Huang
QIUHAO Zeng
Xinyu Wang
Boxing Chen
Philippe Langlais
Yufei Cui
Venue: NeurIPS 2025 poster ()
Keywords: Length Generalization
Mamba
Efficient Method
State Space Models
Length Extrapolation
Calibration
Primary area: deep_learning
TL;DR: We provide a method for enabling length generalization within state-space models by modulating the $A$ matrices per layer.

Abstract: The quadratic complexity of the attention mechanism in Transformer models has motivated the development of alternative architectures with sub-quadratic scaling, such as state-space models. Among these, Mamba has emerged as a leading architecture, achieving state-of-the-art results across a range of language modeling tasks. However, Mamba’s performance significantly deteriorates when applied to contexts longer than those seen during pre-training, revealing a sharp sensitivity to context length extension. Through detailed analysis, we attribute this limitation to the out-of-distribution behavior of its state-space dynamics, particularly within the parameterization of the state transition matrix $A$. Unlike recent works which attribute this sensitivity to the vanished accumulation of discretization time steps, $\exp(-\sum_{t=1}^N{\Delta}_t)$, we establish a connection between state convergence behavior as the input length approaches infinity and the spectrum of the transition matrix $A$, offering a well-founded explanation of its role in length extension. Next, to overcome this challenge, we propose an approach that applies spectrum scaling to pre-trained Mamba models to enable robust long-context generalization by selectively modulating the spectrum of $A$ matrices in each layer. We show that this can significantly improve performance in settings where simply modulating ${\Delta}_t$ fails, validating our insights and providing avenues for better length generalization of state-space models with structured transition matrices.

Mamba Modulation On the Length Generalization of Mamba
Peng Lu" Université de Montréal peng 1u@umontreal.ccmmmmmmmmm. ca
Jerry Huang" Université de Montréal & Mila Quebec Al Institute
jerry huang@mila. quebec
Qiuhao Zeng Western University qzeng530uwo.ca ca
Xinyu Wang
Boxing Chen
McGill University
Noah's Ark Lab
xinyu. www.waaain. www.mmmmmmmmmmml....mmmmmmm
boxing. wwwn@huwwei.ccom com
Philippe Langlais
Yufei Cui Noah's Ark Lab yufei. cui Chuawei.
Université de Montréal
fel felipe@iro. umontreal.C selipe@iro.umontrealll
Abstract
The quadratic complexity of the attention mechanism in Transformer models has motivated the development of alternative architectures with sub-quadratic scaling, such state-space models. Among these, Mamba has emerged as leading chitecture, achieving state-of-the-art results across a range of language modeling tasks. However, Mambas performance significantly deteriorates when applied contexts longer than those seen during pre-training, revealing a sharp sensitivity context length extension. Through detailed analysis, attribute this limitation to the out-of- distrbbbbuinioo behavior of its state-space dynamics, particularly within the parameterization of the state transition matrix A. Unlike recent works which attribute this sensitivity to the vanished accumulation of discretization time steps, Sup--Nn Ac), we establish connection between state convergence behavior as the input length approaches infinity and the spectrum of the transition matrix A, offering well-founded explanation of its role in length extension. Next, overcome this challenge, we propose an approach that applics spectrum scaling to pre-trained Mamba models to enable robust long-context generalization by selec- tively modulating the spectrum of matrices each layer. We show that this can significantly improve performance in settings where simply modulating A, fails, validating our insights and providing avenues for better length generalization state-space models with structured transition matrices. Our code is available https://githnn com/ wwwgggnepuuaaeee -ace/mamba modulation,
Introduction
In the new age of deep learning. the Transformers [ architecture has spurred a new age of re- search into large language models (LLMs) [TI,
that has largely dominated the space of natural language processing (NLP) research since their introduction. Their surprising capabilities and rapid development have led to their wide application across various domains, including chatbots,
[41,
25
y [48]
Eqqual contribution. Cooresspondig author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

intelligent agents, code assistants, etc. However, the Transformer comes with various deficiencies, which has led to research into alternative paradigms that seek resolve these outstanding concern.. One of these competitors, Mamba [AS, 222 is based off the state-space model (SSM) paradigm from control theory [97,, Po that have enabled the training of recurrent models that have overcome the sequential bottleneck of traditional models [T09, D.%, [S}.
Among the various motivation for Mamba and its successors is the goal of length extrapolation, whereby a model that is initially trained on a limited context length (e.g. 2048 tokens in each training sequence) is capable of generalizing to longer sequences test time (i.e. without fur- ther training) due to a more efficient inference-tines token processing methodology. However, various works [56, 29% 57]] have brought about challenges to this claim. Meanwhile, key com- ponent in the Transformer is the position embedding, for which Rotary Position Embeddings (RoPE) [I2II] has been a popular choice and applied in many L.LMs. Various works have stud- ied RoPE [1399 831 and shown it to be intuitive to manipulate to extend the context window within Transformers [14. I. 655, P6], whereas no equivalent method yet exists for Mamba-style models.
A common explanation for the ability to extend this context length is through avoiding out-of- distribution (OOD) rotation angles [8 52 in RoPE, meaning the extended context length (OOD) can be mapped to the in in-distribution (ID) context length which models have been properly trained. However, Mamba does not utilize knowledge of token positions during training, thus making such methods broadly inapplicable.
Recent works [146, 9, [ have meanwhile made attempts exploring how to conduct length general- ization with Mamba models. A shared feature among these is the focus specific input-dependent parameter, A, which is used to discretize the underlying state-space model and control for both the state decay over time as well as the incoming input contribution to the state. Their observations rely on the implicit notion that since the duration of the time-step influences the state, it can act as a proxy to filter out (or ignore) parts of the input, or be scaled to influence the long-term information decay within the model state. However, despite the compelling intuition behind such a notion, there remains no fundamental theoretical justification for this.
In this work, we attempt to build a better understanding of how better scale Mamba models for improved length generalization. We begin with an analysis of the model and the implicit effects this will have on the convergence behavior of the hidden state as the input length goes to infinity. From this, we identify two ways in which this process can be controlled: either through the state- transition matrix A, or through the discretization time-steps A. We then motivate why controlling for adjusting A is more well-founded, through arguments relating the eigenvalue spectrum with long-term information decay. Through experiments on standard long-context extension settings, such as long-context language modeling and passkey retrieval, we demonstrate empirically how scaling A is more effective compared to scaling A, in the case of both Mamba and Mamba2 models. Broadly, we summarize our contributions as follows:
1) We first provide broader understanding of the length generalization ability of Mamba-based models via spectrum analysis of their transition matrix. We demonstrate and justify that the convergence behavior of the hidden states hinders their length generalization Mamba models.
II) Based on our analysis, we identify how the scaling of A as opposed to the more common practice of scaling A is more effective proposition.
III) Results on a series of long-context generalization tasks show such an intuition holds empirically Mamba models, highlighting the potential benefits of using A for length generalization.
2 Related Works
2.1 Language Models and Long Contexts
Being capable of modeling long sequences is an important desiderate in various LLM applications. However, due to the quadratic complexity (relative to the sequence length) of the self-attention mechanism in Transformers, long sequence modeling requires a large computational overhead [025]. Early work in efficient Transformers [74. 117, & IIII, [16] attempted to reduce the computational complexity of attention by inducing greater sparsity. Additional work has explored the use of
linear attention
[72
86.
IEK
[EE
150,
87] to remove the softmax activation that induces this

---

quadratic complexity, however, such methods may impair performance on tasks that demand pre- cise contextual recall [I [33]. Furthermore, hardware optimizations for more efficient computa- tion [22]. 2211 M [0] as well as inference-time acceleration methods [[138] to reduce the compu-
tational and memory complexity of Transformers. However, a broader class of linear recurrent models [ 95%, 93, Z2 IoMI T002, which resemble traditional recurrent neural networks but provide an additional benefit of parallel training over the sequence elements, have emerged as an alterna- tive for long sequences through sub-quadratic complexity relative to sequence length as well as constant-time inference complexity.
2.2 Length Generalization and Extrapolation
Various restrictions on the data available for training make it difficult directly collect data of ex- treme lengths (e.g., 100K+ tokens), hence there have been a great deal of efforts devoted to enabling models to generalize beyond the training length. However, various works have demonstrated the collapse of the performance |لR X3, (13%, thus leaving this an open area of research. Based on the wide dominance of RoPE as the positional embedding of choice, many recent works have focused on extending the context window by scaling the rotary angles [[]]. 355, 96, I=] with potentially some additional tuning, enabling extension to sometimes up to 10x the original training context length. Alternatiely,, linear recurrent models present promise through their lack of direct positional encod- ing; rather, a fixed-size hidden state is often utilized to maintain information from the past while the sequence is being processed. While some promise has been shown synthetic tasks (II [Л], where these models have been shown to be able to filter out noise from the sequence while maintaining useful information within the state, these observations have not extended to tasks such as real-world long-context language tasks [56]]
Yet because many existing methods relevant to Transformer length extrapolation rely explicitly on positional information, remains an open work to find ways to enable such linear recurrent models to generalize beyond their training lengths. Alternatively, recent works [9 14G, 3] have investigated the post-hoc length extension in Mamba models, with a particular focus on using the discretization time-steps A, for context extension. Ben-Kish et al. [E]] use the value of these time-steps to 'deci- mate' or remove tokens from the processing of the sequence at specific layers, resulting in a short- ened sequence length. Similarly, Ye et al. [1466] use the value of the time-steps to filter out tokens. Azizi et al. [33] meanwhile calibrate scaling factors for these time-steps to adjust the long-term decay within the model. extending the context length. Unlike these works, do not analyze the effect of discretization (A) on extrapolation ability. Instead, we focus on establishing a connection between the spectral characteristics of the state transition matrix and the asymptotic convergence behavior of the hidden state as the input length approaches infinity.
2.3 Spectrum Analysis of Linear Recurrent Models
Previous works have provided specific analysis of the eigenvalue spectrum of linear recurrent models as a way of understanding their state dynamics and the downstream influence this can have on performance. Gu et al. [H6] initially provided an understanding of the specific parameterization of the state transition matrix in SSMs, determining the necessity of a Hurwitz matrix for effective sequence modeling. Orvieto et al. [X]] further demonstrated how the eigenvalues have a specific influence state decay as well as long-term dynamics during training. Beck et al. [i]] further bound the state of the recurrence, implicitly bounding the spectrum as well. Finally, Grazzi et al. [[E]] also recently demonstrate the importance of negative eigenvalues for state-tracking tasks.
3 Background
3.1 State-Space Models (SSMs) and Mamba
The SSM-based models, i.e., structured state space sequence models (S4) [49]] and Mamba [5 are inspired by the continuous system, which maps 1-D function or sequence a+)) I Rdm an output ytt) E Am through a hidden state h(t) E dd The system uses evolution parameters A E Rd xda B E dd and C' E BRd xda creating continuous system whose dynamics are governed by
h ht(t) Ah(t) + Br((), v(t) Ch(t)
(I)
3

---

The Mamba model uses the selective SSM blocks, which leverage the input-dependent discretization into the recurrence computation. This is done by including an imput-dependent timescale parameter A (zz) to transform the continuous parameters A, B to discrete parameters A, and Bt. We follow the official implementation of Mamba [A5]]:
h A, B,s,
= C.Ch:
(2)
This method uses Zero-Order Hold (ZOH) for the matrix A, and a simplified Euler discretization for the matrix B,, omiting the computation of matrix inversion for B as required by the ZOH:
A exp A ④ A) B. = A, @ (A.A) (exp (A,A) - -1) B)
(3)
The key improvement of Mamba is making the parameterization (A,, B, and C₁) input-dependent. Specifically, each part of them can be computed as follows:
A softplus (Lineara (zx)), B. Linearg (zt) C, Linearc (ax)
(4)
where Linear a Linear & and Linearc are regular linear projections, @ the Hadamard product, @ is the outer product, A; E Ri and A = diag (a1.... a.) s.t. a₃ O'i E {1...., d). In Mamba, A, B and C' are inputddep,n,ent, such that at each time-step unique transition matrices can be used to update the system (A is left as a fixed parameter in as the dynamics of the state should be consistent across steps). This is based the observation that some elements in discrete sequence may not be as important as others, therefore there an incentive to possibly update the system differently based on this factor. This results unique update matrices at each time-step (A,, A,, Br. Ct), enabling the ability to solve problems that require selective processing of the sequence. In order to maintain computational efficiency A is restricted to having a diagonal structure such that only the diagonal elements of these matrices need to be stored. Mamba2 [22]] further restricts the diagonal matrix to have the form a scalar-times times-identity matrix, enabling further computational improvements.
3.2 Limitations of Mamba Long Context
The output can be reformulated as a matrix product form as follows:
yi y/2
B
0
CAABB
C₂B₂
0
Y
MX
(5)
:
0
Cr. IT A.B, CI A.B.
Br.
L
In this formulation, each output y: is computed weighted sum of all inputs, with each weight involving a product of transition matrices II A.. This product term plays crucial role de- termining the influence of past states and can be further disentangled enable fine-grained analysis, facilitating a deeper understanding of state evolution and transition behavior.
L
II
A
II exp exp( = exp A (=f+1
A
(6)
1=++1
1=j+1
[ has primarily focused analyzing the discretization step A, for long context, particularly the vanishing effect of the accumulated term exp (xp (- A, when N is large and propose different solutions to overcome this out-of-distribution (OOD) issue. For instance, Azizi al. [EI] propose applying scalar values < 1 across different model layers to mitigate OOD discretization steps, ensuring smaller A, to prevent the vanishing issue distant inputs. They introduce two calibration methods and demonstrate superior length generalization performance in calibrated Mamba models with unconstrained scaling factors. However, their work does not explain why some of resulted scaling factors
Previous work (9, [16,,
could still enhance generalization performance.
4 Spectrum-Based Analysis of Mambas Length Generalization
In this section, we examine the length generalization ability of Mamba-based language models from the perspective of spectrum analysis of their transition matrix. Specifically, we analyze the state convergence behavior of the hidden state Mamba. Based on our findings, we propose a spectrum scaling method to enhance the length generalization capability of pre-trained Mamba models.

---

diag (exp (-)) 4.1 Spectrum of Mamba Transition Matrix
We first visualize the spectrum of the continuous transition matrix of Mamba models. The exp (-A) parameterization guarantees all values À E (0, 1). We stack all 48 layers and rank eigenvalues in descending order for each row. The magnituees (appearing to range from 0 to 1) imply that all eigenvalues À lie well inside the unit cir- cle, which critical for the stability of the dynamics governed by the transition matrix for Mamba training. High eigenvalue zones can be viewed as dominant temporal modes, useful for modeling long-term dependencies, especially in language or time series tasks. We also ob- serve that low-eigenvalue regions in the transition matrix spectrum correspond to rapidly de- caying modes, which specialize in modeling local dependencies and high-frequency dynamics.
Heatmap of Eigenvalues
The heatmap in Figure reveals consistent spectral pat- tern across layers: the coexistence of both large (near 1) and small (near 0) eigenvalues. Next, we establish the connection between the failure of length generalization and the spectrum of the transition matrix by showing divergent tendency in the convergence of the state norm,
Layers 48)
4.2 State Convergence in SSMs for Long Contexts
The previous section presented the numerical spectrum of the transition matrix exp(-A) Next, we theoretically investigate its influence on the convergence behavior of Mamba states. We begin by introducing the following lemma, which establishes an expected bound on the norm of inputs.
Feature Dimension (d=64)
Heatmap eigenvalues of diag(exp(- A)) of mamba2-1.
Figure
Lemma 4.1. Let B E Rdxd be a matrix with | d. and let x E IRdd be a vector such that each entry of x satisfies |x:|| S oz. The upper bound for |Bxll2 is:
Bxll2 SB Ox Vd.
Theorem 4.2. (Convergence of State Norm with Real- Valued Diagonal Transition Matrix). Let the real- real-valued transition matrix A dddd be diagonal with eigenvalues >; ~ Uniform Xmins Amaxh where 0 < Amin < Amax < 1. Consider the system dynamics: h, = Ahh-1 Bx, where æ; is are independently sampled as
the imput vector at time step and B is weight matrix whose b ~ N (0, 1). Then, as t - oc, the expected squared norm of the state h2 converges to:
X
E| c0 11
log
E Bx+|
(8)
2(Amax
Amin)
X2 max
Under the setting of Theorem (proofs in Appendix A)). we consider two cases (Mamba and Mamba2) corresponding to the architectural variants of structured state-space models, each char- acterized by a different form of the structured transition matrix.
Corollary 4.3 (Norm of Mamba State). Suppose the diagonal entries of are independently drawn from a uniform distribution on [0, A), moderate discretized step value A and the system evolves as h; = Aht-1 + Ba, diag (exp(- - -a]) h4-1 + ABE Then the growth rate of the expected
squared norm of the limiting state satisfies O
log
)
2A
Corollary 4.4 (Norm of Mamba2 State). Suppose A = = XI = exp(- is scalar multiple of the identity matrix, where X E (0, 1), moderate discretized step value and the system evolves as h Ah Bx, exp( da)= h ABx; Then the convergence rate p of the expected squared norm of the limiting state can be estimated as O (
These provide insight into the asymptotic convergence behavior of Mamba states the input se- quence length grows with different eigenvalues. If À > 1, or - 0, then
lim p 00, X=+1
lim o
(9)
5

---

These rates shed light on challenges in length generalization for structured state-space models (SSMs) with constrained diagonal transition matrices. In particular, both extremely large eigenvalues (approaching 1) and extremely small eigenvalues (approaching 0) can induce instability in the Mamba state norm as input length increasesleading to state explosion vanishing. respectively. While tun- ing the discretization step A can help modulate the convergence rate (as suggested by Corollary 9.3 and E.) does not address the root cause: the distribution of the transition matrix eigenvalues. To directly tackle this issue, we propose spectrum scaling method that adjusts the spectral distribu- tion of pre-trained Mamba model by compressing large cigenvalues and inflating small ones. This rescaling aims stabilize the state norm across longer sequences, thereby improving the models ability to generalize over input length.
4.3
State Norm Analysis across different input lengths
ProofPile
IK
2K
4K
8K
16K
32K
64K
State norm (max)
441.4489
174.6176
892.7114
1330.5321
1441,5553
1130.6537
1157.5122
State norm (min)
0.0046
0.0030
0.0007
0.0007
0,0004
0.0003
0,0005
PG19
1K
2K
4K
8K
16K
32K
64K
State norm (max) State (min)
158.7656 0.0033
160.8465
166.3788 0.0050
876.0833 0.0004
1241.3979 0.0005
1201.5530
1242.7589
0.0014
0.0003
0.0002
Table 1: State norm statistics for ProofPile and PG19 datasets across different sequence lengths.
We conducted series of experiments to examine how the hidden state of the SSM evolves with increasing imput sequence length, as shown in Table Using randomly sampled data from ProofPile and PG19, measured the maximum difference between the largest and smallest SSM state norms across all layers the mamba2-1.. .b model. The observed divergence state norm magnitude as sequence length grows provides empirical validation for the theoretical predictions outlined in the previous section.
Mamba Modulation for Length Extrapolation
In the following sections, we describe series of experiments that we conduct to validate previ- ous intuitions. Appendix C provides more specific implementation details and design choices.
5.1
A Simple Case Analysis on Constant Scaling
Modeling Perplexity
To confirm our intuition, we first attempt simple comparison between the effects of scaling As and A. Here, across all layers, we use a fixed, constant-valued scaling fac- tor. We evaluate language modeling perplex- ity on the ProofPile dataset [+O], following Peng et al. [96],, across varying number of context lengths. This method uses no tun- ing training; the scaling is applied explic- itly during the forward pass. Figure shows these results after applying scaling on per- plexity on context lengths from 2K to 128K tokens, with scaling factors of 2 or 3 applied.
Ma Mbb-2.888
Ma.ba2-2.78
165
PPL
1e3
1e1
2.
2%
2
2/
24
Length
(Alpha, Delta)
(1.11 n1 178
(122 11
113 1)
We see that scaling A by constant scal- ing factor is significantly better at incurring lower perplexity, however, it remains the case that such a constant scaling factor needs to be properly tuned for, particularly in the case of mamba-2. 8b. Given the simple set- ting/scenario on which we experiment, this unsurprising; as we investigated in Section 222 differ- ent layers have different underlying behavior in terms of their eigenvalues, making likely difficult
Figure 2: Language modeling perplexity on Proof- Pile after applying constant scaling factor to either A or A. Lines are distinguished by their colors. (1, 1) means the baseline where nothing is scaled.

---

to find a constant scaling factor that can work across all layers. In the case of mamba2-2. 7b, we can see that upplying these scaling factors can significantly bound the long-context perplexity from exploding.
5.2 Adapting MambaExtend to Scale A
Given observations and analysis re- garding the relationship between A and As, natural method against which we can compare is MambaExtend is a training-free method that scales the dis- cretizations steps at each layer. For a model with I. layers, the objective is learn set of constant scaling factors for each layer {= which can be used to adjust the discretizations steps A.. In general, 8; can be set either scalar or a vector. These scaling factors serve as learnable param- eters in within the model but are consequentially tuned in a manner that does not require training any other parameters within the model. The idea of the algorithm is to take a pre-trained model along with a small set of samples for calibrating the scaling factors; depending on the setting, the calibration function can vary, with the only restriction being that the original model parameters are not modified during calibration. Appendix C33 describes the implementation in further detail.
Algorithm MambaExtend methodology.
1: Input: Model M, calibration set C and function CF
2: Output: Scaling factors S = [si..... sL E I4 X
for L do + (0, 1) end for
4:
6: S t CF(SS,,,
return S
Although the original MambaExtend learns scaling factors only for A,, their methodology is adapt- able to usage with A instead: given the shared dimensionality for both A and A., the scaling factors can be directly used for calibrating A. Furthermore, this means that tuning scaling factors for A does not require any additional computation, time or memory requirements as compared to tuning them directly A,, leading to a simple yet effective algorithm that can directly be applied to the adaptation of A for long Long-contex. generalization. The following sections evaluates the performance and efficiency of tuning these scaling factors for A on a number of standard settings for evaluating long-context generalization of models. As a baseline, we compare directly with MambaExtend.
6 Experiments and Results
6.1 Language Modeling Perplexity
We first experiment by measuring language modeling perplexity after calibrating scaling factors for either Ae or A. In this task, use the black-box zeroth-order calibration method suggested by Azizi et al. [G]; we train a single scaling factor 8; E I for every layer i in the model. For a T-layer model, this means L. individual scaling factors are used. To calibrate, 20 samples of the corresponding context length are used. For example, for a length of 16K, 20 samples of this length are used for the calibration of the set of sj. Figure shows these perplexity results on number of validation datasets, namely ProofPile [20], PG19 [[TK]]] and GovReport [59]].
In particular, scaling A leads to better perplexity on nearly all validation datasets, for both Mamba and Mamba2 models. In many cases, this gap can be significant, particularly in the case of mamba2-2. 7b, where the perplexity at long sequences when calibrating A explodes for all three datasets whereas calibrating A can lead the model to maintain a consistent perplexity up to 1000 lower.
6.2 Passkey Retrieval
Next, we conduct experiments on the Passkey Retrieval task, also known as the Needle-in-A- Haystack. Similar to before, we conduct this compare the effectiveness of tuning scaling factors for A as opposed to A;; we again conduct this experiment across different Mamba models. Unlike the language modeling perplexity task however, we train the model on a training set. This train- ing set contains samples of length 4096 corresponding to the task, where the objective is standard instruction-tuning [B2]]. However, we freeze all parameters except the scaling parameters for each layer. For Mamba, it is equivalent to the number of inner state dimensions, i.e. each inner state uti- lizes the same scaling factor for each dimension of the SSM state. For Mamba2, this is the number of heads, meaning that each head shares the same scaling factor for each component of its state.
7

---

GoVReport Mamba-1.44 1.4B
GovReport Mamba-2.8B
PG19 Mamba-1.44
PG19 Mamba ba-2.8B
ProofPile Mamba-1.. 1.4B
ProofPile
Mamba-2.8B
104
PPL
1e3
1c2
1e1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
GovReport
GovReport
PG19 Mamba2-1.38
PG19
ProofPile
ProofPile
Mamba2-1.38
Mamba2-2.78
Mamb a2-2.7B
Mambaz-1.3
1.38
Mamba2-2.78
1c5
1e4
PPL
1e3
162
1e1
2
2
2
2
2
2
2
24
2
2
2
2
2
2
2
2
2
2
Length
Calibrated Component
B Baseline
Delta
Figure 3: Model perplexity by calibrating scaling factors for either log (A) (red) or A, (turquoise), across different datasets and sizes. Baseline means the base model with no calibration, i.e. the model used directly without modification.
of fixed lengths and depths to evaluate for both generalization ability as well as potential biases to relative location within the sequence. The exact setup follows from Ben-Kish et al. [H], in particular, the task comprises of 5-digit code embedded at random sequence depth within samples from the WikiText-103 dataset [89]. Models are deemed to have solved length/depth pair if they can correctly solve all evaluation examples, i.e. retrieve the code within the example.
Evaluation is conducted
very consistent results similar to our language modeling perplexity results; for Mamba, smaller models appear to fare slightly better when trained to scale A,, but the models get larger, learning to scale A closes the gap and eventually exceeds the performance of scaling A, Similarly, for Mamba2 models, scaling A appears nearly always be constant improvement in performance on the task. Further comparing with a full-fine-tuning of the model, observe that scaling A is effective despite fewer parameters being trained, whereas scaling A, does observe a drop-off in performance.
Figure + and Appendix 2 visualize these results. In particular,
appropriate choice in comparison scaling A,,
by nearly
6.3 LongBench
LongBench [+]] is a popular benchmark for testing the long-context abilitics of LLMs, serving as more suitable real-world benchmark which
explore how the scaling of A as opposed to A, can influence performance. Here, we again use the zeroth-order optimization method as we used for initial perplexity experiments. More specifically, constant scaling factor is used for each individual layer. We compare against both the initial base model, as well as NambaExtend. Table shows results mamba2-2. 7b. In particular, 6% through the calibrated scaling of A, with relative improvement of nearly 10% compared to the scaling was instead calibrated for AŞ.
show that can increase performance by over
Table Results on LongBench [6]]
Model
Strategy
Qusper
HotporQA
2WMHQA
TREC.
TriviaQA
LCCC
RB-P
Average
Base Model
1.17
1.54
2.18
8.33
10.60
23.46
14.97
8.75
mamba2-2 22222 7b
ManbaExtent end
12.53
1.63
5.99
24.63
10.33
23.00
17.09
13.60
Calibrated Scaling
10.49
23.36

---

Passkey Performance
Full FT Mamba-1.48
Full Mamba-2.88
Full FT
Full FT
Mambaz-1.38
Mamba2-2.78
25%
25%
25%
25%
50%
50%
50%
50N
75%
75%
75%
75%
100%
1K 2K 4K 16K3K2
1K 2K 4K 16132K
1K
2K
4K
8K
16K
1K
2K
4K
8K 16K
Delta Mamba-1.4B
Delta Mamba-2.88
Delta
Delta Mamba2-2.78
Mamba2-1.2
0%6
0%
0%
25:6
25%
25%
25%
Depth
50%
50%
50%
50%
75%
130%
100%
100%
1K 2K 4K. 8K 15KG2K
1K 2K 4K 8K 18KSEK
1K 2K
4K
8K 16K
1K
2K
4K
9K
16K
A
A
A
Mamba-1.48
Mamba-2.88
Mamba2-1.38
Mamba2-2.78
0.6
396
036
0%
25-9
25%
25%
25%
50%
75%
75%
100%
100%
100%
100%
1K 2K 4K 8K115K
1K 2K 4K 6X 16K32K
1K 2K 4K 8K
16k
1K
2k
4K RK
16K
Context Length
Score
Passkey Retrieval performance of Mamba models by calibrating scaling factors for either log (A) or A, Turquoise squares mean that the model was able to solve all examples of the given evaluation length/depth pair after tuning scaling factors, while red squares means otherwise.
Figure 4:
Furthermore, if looking more specifically individual tasks, there no settings where calibrating A, results in a meaningful performance increase compared to A, whereas calibrating A instead appears significantly increase performance on HotpotQA and 2WikiMulihopOA..
6.4 Comparison with Alternative Methods
As a final point of comparison of our proposed methodology, we compare against other propos- als that have aimed towards extending the context of Mamba. Unlike MambaExtend, both of these methods use a filtering mechanism rather than directly scale A,; in LongMamba [146),, channels are prevented from exponential decaying by filtering of specific token within the sequence A, is smaller than a preset threshold. DeciMamba [9] instead defines decimating layers that directly filter tokens that are then not passed the following layer, significantly shortening the sequence that the last layers within the model observe. Both mod- els require additional tuning; LongMamba calibrates multiple eyper-parametens to tune their filtering mechanism, while DeciMamba requires training the decimation layers on longer sequences.
tokens from the training sequence the update
For reasons of public code availability and methodology mamba-1. 1.b........... 8b and LongMamba against mamba2-1 -..b. We also provide results using the ini- well as the previous
compare DeciMamba against
tial base model, MambaExtend, constant scaling as well as the calibrated scaling based on MambaExtend. Table B compares the effectiveness of these different methods on perplexity on the PG19 dataset. We note that in all cases. the calibrated scaling of A performs either the best or second best on all context lengths across the different tested models with marginal gaps when the best performing method, while other
ways tested for scaling A, namely
"LongMamba did not release their tuning code: httpgs///iiiiiia..coo//rrrtaaa/p Mambaa only modified Kamba CUDA kernels: http:///ihhhaboooaaaaaaaaa///l/a

---

Table 3: Comparison of PG19 perplexity varying lengths. Cases where scaling A leads to the lowest perplexity are bolded and underlined when second best. If the best method does not involve scaling A, it is highlighted in violet.
Context Length
Model
2K
4K
8K
16K
32K
64K
mamba- 4b
Base Model
9.67
10.23
11.43
17.46
59.77
444.09
DeciMamba
11.45
12.34
14.65
19.83
24.85
28.48
ManbaExtend
4.69
3.89
3.83
3.55
5.31
1648.0
Constant Scaling
44.68
53.46
59.56
63.51
75.86
114.44
Calibrated Scaling A
5.31
4.31
4.13
6.88
14.94
19.13
mamba 2.
Base Model
8.66
9.42
22.78
91.43
202.20
508.89 88
DeciMamba
11.34
13.45
15.63
18.34
21.53
26.54
MambaExtend
4.25
3.80
3.63
5.25
87.00
60.00
Constant Scaling A
28.80
33.93
39.80
69.37
162.77
355.77
Calibrated Scaling A
4.44
4.31
5.63
8.94
12.75
40.00
mamba2-1
1.3b
Base Model
9.52
10.54
25.49
115.65
634.32
1479.45
LongMamba ManbaExtend
10.12
10.31
11.36
11.61
12.81
13.55
4.34
3.69
3.44
5.00
14.94
27.50
Constant Scaling A
11.12
11.83
12.47
12.71
12.85
13.22
Calibrated Scaling A
4.38
3.78
3.44
3.28
4.03
4.72
methods are fairly inconsistent this front. Meanwhile, constant scaling is generally ineffectivee confirming previous doubts from Section 5.1 regarding the usefulness of single constant factor based the previous eigenvalue analysis. These results further support our analysis regarding the use of scaling factors for for length generalization compared to wide variety of methods.
7 Conclusion
In this work, we conduct an in-depth exploration regarding the state transition matrix of Mamba models. We first provide broader understanding of the SSM parameterization and how it can affect length generalization in Mamba models. particular, we analyze the eigenvalue spectrum of both Namba and Mamba2 models, identifying the specific role this can have on the convergence of SSMs given long inputs. Then we identify how the scaling of as opposed the more common practice of scaling A can be more effective at tuning this spectrum, enabling models to better generalize to long- contexts that far exceed the training context. We experiment on multiple long-context generalization tasks to validate that this newly built intuition holds empirically, on both Mamba and Mamba2 models, highlighting the potential benefits of using for length generalization.
Acknowledgements
Jerry Huang supported by the NSERC Canada Graduate Scholarships Doctoral (CGS-D) pro- gram (funding reference number 589326) well as the Bourse d' d'éxcellence Hydro-Québec pro- gram.
10

---

References
[1] S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. Zo- ology: Measuring and Improving Recall in Efficient Language Models. In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https::/ wwt./o/owr/dddd
[2] K. J. Åström. Introduction to stochastic control theory. Courier Corporation, 2012.
[3] S. Azizi, S. Kundu, M. E. Sadeghi, and M. Pedram. MambaExtend: A Training-Free Ap- proach to Improve Long Context Extension of Mamba. In The Thirteenth International Con- ference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net, 2025, URL
[4] D. Bahdanau, K. Cho, and Y. Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In Y. Bengio and Y. LeCun, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: arrxiv..vv.. org/ abs/ 1409.040000 .0733
[5] J. Bai, S. Bai, Y. Chu, 7. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang. W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang,
J. Yang, Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, Zhang, X. Zhang, Y. Zhang, Z. Zhang,
C. Zhou, J. Zhou, X. Zhou, and T. Zhu. Qwen Technical Report, 2023, URL https://doo
brg/ 10. 48550/ arxiv. 2309 16609,
arXiv: 2309.
[6] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang. Z. Du X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li. LongBench: A Bilingual, Multitask Benchmark for Long Con- text Understanding. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 3119-3137. Associa- tion for Computational Linguistics, 2024. doi: 10.18653/V ACO.A//LNGG..ALL URL ottp:://aa..org//00006666// wtt.:///...1111///////100111 acl-long. 172.
[7] M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brand- stetter, and S. Hochreiter. XLSTM: Extended Long Short-Term Memory. In A. Glober- sons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: http:::ppppps.......//// nips ccppaper Blles/paper/024// htm
[8] I. Beltagy, M. Peters, and A. Cohan. Longformer: The Long-Document Transformer, 2020.
URL https://arxi: http://aaxvv...o org/ abss/ 220// 05150, arXiv: 2004.05150.
[9] A. Ben-Kish, I. Zimerman, S. Abu-Hussein, N. Cohen, A. Globerson, L. Wolf, and R. Giryes. DeciMamba: Exploring the Length Extrapolation Potential of Mamba. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore. April 24-28, 2025. OpenReview.net, 2025. URL
[10] X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Fu, H. Gao. K. Gao, W. Gao, R. Ge, K. Guan, D. Guo, J. Guo. G. Hao, Z. Hao, Y. He, W. Hu, P. Huang, E. Li, G. Li, J. Li, Y. Li, Y. K. Li, W. Liang, F. Lin, A. X. Liu, B. Liu, W. Liu, X. Liu,
X. Liu, Y. Liu, H. Lu, S. Lu, F. Luo, S. Ma, X. Nie, T. Pei, Y. Piao, J. Qiu, H. Qu, T. Ren,
Z. Ren, C. Ruan, Z. Sha, Z. Shao, J. Song. X. Su, J. Sun, Y. Sun, M. Tang. B. Wang. Wang.
S. Wang. Y. Wang, Y. Wang. T. Wu, Y. Wu, X. Xie, Z. Xie, Z. Xie, Y. Xiong, H. Xu, R. X. Xu,
Y. Xu, D. Yang, Y. You, Yu, X. Yu, B. Zhang, H. Zhang, L. Zhang, L. Zhang, M. Zhang, M. Zhang, W. Zhang, Y. Zhang, C. Zhao, Y. Zhao, S. Zhou, S. Zhou, Q. Zhu, and Y. Zou. DeepSeek L.M:: Scaling Open-Source Language Models with Longtermism, 2024. URL arXiv: 2401.02954.
https: //dii org/ 10. 48550/ 48550/arXiv. arr.v..iiii.i.vii 2401. 02954.

---

[11] bloc97. AwKre Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.. June 2023. URL https: / //www. reddit. htt.l/ccmo///.eeeeeee httar//Locaaaalaa wttaaaa//ooooettt/aaaaaaa.a ntkaware scaled rope allows 1lama models to have/.
[12] A. Botev, S. De, S. L. Smith, A. Fermando, G.-C. Muraru, R. Haroun, L. Berrada, R. Pascanu, P. G. Sessa, R. Dadashi, L. Hussenot, J. Ferret, S. Girgin, O. Bachem, A. Andreev, K. Kenealy,
T. Mesnard, C. Hardin, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love,
P. Tafti, A. Joulin, N. Fiedel. E. Senter, Y. Chen, S. Srinivasan, G. Desjardins, D. Budden, A. Doucet, S. Vikram, A. Paszke, Gale, S. Borgeaud, C. Chen, A. Brock, A. Paterson, J. Brennan, M. Risdal, R. Gundluru, N. Devanathan, P. Mooney, N. Chauhan, P. Culliton,
L. G. Martins, F. Bandy, D. Huntsperger, G. Cameron, A. Zucker, T. Warkentin, L. Peran, M. Giang. Z. Ghahramani, C. Farabet, K. Kavukcuoglu, D. Hassabis, R. Hadsell, Y. W. Teh, and N. Frietas. RecurrentGemma: Moving Past Transformers for Efficient Open Language Models. CoRR, abs/2404.07839, 2024. doi: 10. 10.48550/RXXV..404..00 orgoo..oo//1..... 408444855aaiii
URL https
07839 arXiv: 2404.07839.
[13] T. B. Brown, B. Mann. N. Ryder, M. Subbiah, Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler,
Wu, C. Winter, C. Hesse, M. Chen, Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H.-T. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpss:
neur cc/ html.
happer/220//ashh//1 http://ssessaooooooooooo0o///aar
[14] S. Chen, S. Wong, L. Chen, and Y. Tian. Extending Context Window of Large Language Mod- els via Positional Interpolation, 2023. URL
23300 2306.
15595, arXiv: 2306.15595.
[15] K. Cho. B. v. Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In A. Moschitti, B. Pang, and W. Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724-1734. ACL, 2014. doi: 10.3115/V1/DD4-1119. URL https: 1tb.i..rg/10..115//111.
[16] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, B. Belanger, L. Colwell, and A. Weller. Rethinking Attention with Performers. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: //openrevier
[17] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, Schuh, K. Shi, Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, Brad- bury, Austin, M. Isard, G. Gur-Ari, Yin, Duke, A. Levskaya, S. Ghemawat, H. Michalewski. X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito. D. Luan, H. Lim, B. Zoph. A. Spiridonov, R. Sepassi, D. Dohan, Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang. B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K.
Dev,
D. Eck, J. Dean, S. Petrov, and N. Fiedel. PaLM: Scaling Language Modeling with Pathways. J. Mach. Learn. org/ wwtgg/apers/aee
Res., 24:240:1 1:::4:::1240:1111 13, 2023. URL https://
wwwppp/aww//////..//a/pp /////2//2222222 htm4.html
[18] H. W. Chung, L. Hou, Longpre, B. Zoph. Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Root
M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, Y. Zhao, Y. Huang, A. M.
Dai, H. Yu, S. Petrov, F. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q V. Le, and J. Wei. Scaling Dnstruction-Finetunee Language Models. J. Mach. Learn. Res., 25:70:1-70:55 2024. URI. https: jmlr. wttp://ppss/////222.00000 0870 html.
12

---

[19] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to Algorithms, 3rd Edition. MIT Press, 2009. ISBN 778-0-262-03384- URL http::/mitpress..iicc. edt. books/ antrrdduction-llgoriii
[20] D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng. X. Yu, Y. Wu, Z. Xie, Y. K. Li, P. Huang. F. Luo, C. Ruan, Z. Sui, and W. Liang. Moe:SeekMoE: Towards Ultimate Expert Specialization in Mixture-of of-xerts Language Models, 2024. URL https://doi.
481/0//r.555//r.ii..... 2401. 06066 arXiv:
[21] T. Dao. FlashAttention-22: Faster Attention with Better Parallelism and Work Partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreevev wtwp:/rreiiiieeeeoorreuee
id=m2n2kyh9Ec,
[22] T. Dao and A. Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
https://openrewee wtt.://www....efoooorodttttttnon
[23] P. Dasigi, K. Lo, Beltagy, A. Cohan, N. A. Smith, and M. Gardner. A dataset of information- seeking questions and answers anchored in research papers. In K. Toutanova, A. Rumshisky, Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-ILLT 2021, Online, June 6-11, 2021, pages 4599-4610. Association for Computational Linguistics, 2021. URL https: / org/ 10. 18653//1/22000 nwacl pasaaclmaii.... mmin.365. 365
L. Zettlemoyer, D. Hakkani-Tür,
[24] DeepSeek-AL AI, A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Guo. D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Yang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai, J. Liang, J. Guo, J. Ni, J. Li, J. Chen, J. Yuan, J. Qiu. J. Song, K. Dong, K. Gao, K. Guan,
L. Wang, L. Zhang, L. Xu, L. Xia, L. Zhao, L. Zhang. M. Li, M. Wang. M. Zhang. M. Zhang, M. Tang, M. Li, N. Tian, P. Huang, P. Wang. P. Zhang. Q. Zhu, Q. Chen, Q. Du, R. J. Chen, R. L. Jin, R. Ge, R. Pan, R. Xu, R. Chen, S. S. Li, S. Lu, S. Zhou, S. Chen, S. Wu, S. Ye, S. Ma, S. Wang, S. Zhou, S. Yu, S. Zhou, S. Zheng, T. Wang. T. Pei, T. Yuan, T. Sun, W. L. Xiao, W. Zeng, W. An, W. Liu, W. Liang, W. Gao, W. Zhang, X. Q. Li, X. Jin, X. Wang, X. Bi, X. Liu, X. Wang, X. Shen, X. Chen, X. Chen, X. Nie, and X. Sun. DeepSeek-V2 V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, 2024. URL https: //doi. 10. 48550/ arxiv. 2405 04434, arXiv:
[25) DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai, J. Liang, J. Guo, J. Ni, J. Li, J. Wang, J. Chen, J. Chen, J. Yuan, J. Qiu,
J. Li, J. Song, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang. L. Zhang, L. Xu,
L. Xia, L. Zhao, L. Wang. L. Zhang. M. Li, M. Wang. M. Zhang, M. Zhang. M. Tang, M. Li,
N. Tian, P. Huang. P. Wang, P. Zhang, Q. Wang. Q. Zhu. Q. Chen, Q. Du, R. Chen, R.
Jin, R. Ge, R. Zhang, R. Pan, R. Wang. R. Xu, R. Zhang, R. Chen, S. S. Li. S. Lu, S. Zhou,
S. Chen, Wu, S. Ye, S. Ye, S. Ma, S. Wang, S. Zhou, S. Yu, S. Zhou, S. Pan, T. Wang, T. Yun, T. Pei, T. Sun, W. L. Xiao, and W. Zeng. DeepSeek-V3. V3 Technical Report, 2024. URL arXiv: 2412.19437.
40tp://00..555//.1...... www.:............... 2412 19437,
[26] DeepSeek-AL, Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu. Y. Wu, Y. Li, H. Gao, S. Ma, W. Zeng, X. Bi, Z. Gu, H. Xu, D. Dai, K. Dong, L. Zhang, Y. Piao, Z. Gou, Z. Xie, 7. Hao, B. Wang, Song, D. Chen, X. Xie, K. Guan, Y. You, A. Liu, Q. Du, W. Gao, X. Lu, Q. Chen, Y. Wang, C. Deng, J. Li, C. Zhao, C. Ruan, F. Luo, and W. Liang. DeepSeek- Coder V2: Breaking the Barrier of Closed- Source Models in Code Intelligence, 2024. URL arXiv: 2406. 11931.
https://doi. org/10. 48550/ arxiv. 2406. 11931,
[27] DeepSeek-AL, D. Guo, D. Yang. H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Wu, 7. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan,
13

---

D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang,
H. Bao, H. Xu, H. Wang. H. Ding, H. Xin. H. Gao, H. Qu, H. Li, J. Guo, J. Li, Wang. J. Chen, J. Yuan, J. Qiu, J. Li, J. I. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L.. Wang, L. Zhang. I.. Zhao, L.. Wang. L. Zhang, L. Xu, L., Xia,
M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang,
Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang. R. Pan, R. Wang. R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, and S. S. Li. DeepSeek- RI: Incentivizing Reasoning Capability in LLMS via Reinforcement Learning, 2025, URL
https://doi org/ 10. 48550/arXiv 2501. 12948,
arXiv: 2501.12948.
[28] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. Universal Transformers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https::/ HyzdRiR9Y7.
net/ ftr//forrmmi
[29] Z. Dong, T. Tang, J. Li, W. X. Zhao, J.-.. Wen. BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 2086-2099, ELRA and ICCL, 2024. URL https: / acl aalannnoooogyyy org/ org///24..........o 2024 Irec main. 188
[30] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer. G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In 9th International Conference on Learning Representations, ICI. 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. URL httts://openrevviee .wtt:/oormmm/d-/iiaaaaa
[31] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang. A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Rozière, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song. D. Pintz, D. Livshits, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Zhang, G. Synnaeve, G. Lee, G. I.. Anderson, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. M. Kloumann, I. Misra, I. Evtimov, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. v. d. Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang. J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, Jia, K. V. Alwala, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, and e. al. The Llama 3 Herd of Models, 2024. URL https: sttps://dor http://oooooo//////// 10. 48550/arXiv 2407 21783, arXiv: 2407.21783.
[32] Y. Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, edi- tors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL ct 7544/80006550000 wttp///adaaaaaaaaaaaaaa www..aaaattttta.........a Conf
nips wwtpccpppppppppcccccccccccce.ece files/
files/paper pape// Seaper//023//nne 2023/hash/
[33] J. C. Duchi, E. Hazan, and Y. Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. In A. T. Kalai and M. Mohri, editors, COLT 2010 The 23rd Con- ference on Learning Theory, Haifa, Israel, June 27-29, 2010, pages 257 269. Omnipress, 2010. URL http://colt205/ hww.aa............mmmmmm pdf #G##page=265)
haifa. i...m.........m.mmm.mmmmm.mm.m.. com ctta///ppees0001110000rreeeeeeee
[34] J. C. Duchi, E. Hazan, and Y. Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. J. Mach. Learn. Res., 12:2121-2159, 2011. doi: 10.5555/1953048. 2021068. URL https::/seee //al. acm. org/ htgg/do/100 5555/ 1951049130848 2021068,
14

---

[35] emozilla. Dynamically Scaled RoPE further increases performance of long context LLaMA with zero fine-tuning, June 2023. URL https://www http://wwww...d.....//. reddit C scaled rope further increases/.
comments/ Hannggp/dynaaicaa
[36] W. Fedus, B. Zoph, and N. Shazeer. Switch Transformers:
Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. J. Mach. Learn. Res., 23:1 13:120:1-120:39 2022. URL https:// http://ppppppp////e/ee.
0998. html
[37] D. Y. Fu, S. Arora, J. Grogan, I. Johnson, E. S. Eyuboglu, A. W. Thomas, B. Spector, M. Poli, A. Rudra, and C. Ré. Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://papers. nips. cc/ wwwpppppppppppppppppppcp.pee.eee files/pagge httppppppp/2//aaaapeee htm
[38] D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Ré. Hungry Hungry Hip- pos: Towards Language Modeling with State Space Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open- Review.net net, 2023. URL https://openrevienn net/eeruuuur DiD.uun?id--CDZDOOYGG
[39] D. Y. Fu, E. L. Epstein, E. Nguyen, A. W. Thomas, M. Zhang, T. Dao, A. Rudra, and C. Ré. Simple Hardware- Efficient Long Convolutions for Sequence Modeling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt. S. Sabato, and J. Scarlett, editors, International Confer- ence on Machine Learning. ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10373-10391. PMLR, 2023. URL https://proceeddngg mmlr prwss/pessssseeee seteess/vv22//1 2222222222222 html.
[40] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800GB Dataset of Diverse Text for Lan- guage Modeling, 2021. URL https://arxa: //tssv//axxv...m/ www.oogg/.../ooooooooooooooooeee htt///bb///111 00027. arXiv: 2101.00027.
[41] J. Goodfellow, D. Warde-Farley, M. Mirza, A. C. Courville, and Y. Bengio. Maxout Net- works. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Confer- ence Proceedings, pages 1319-1327. JMLR.org. 2013. URL http://proceedingg mlr. press/v28/ go22fe/oodeelloo html.
[42] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative Adversarial Nets. In 7. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Infor- mation Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2672-2680, 2014. URL https://proceeddimg wwwpeeiiinnnnnneeeeeeeeeee html
[43] A. Graves, G. Wayne, and I. Danihelka. Neural Turing Machines. CoRR, abs/1410.5401, 2014. URL http://arxiv.orrt org/ abs/ /etp//111....4.. 5401,
arXiv: 1410.5401.
[44] R. Grazzi, J. Siems, A. Zela, J. K. H. Franke, F. Hutter, and M. Pontil. Unlocking State- Tracking in Linear RNNs Through Negative Eigenvalues. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. Open- Review.net, 2025. https: wttp:e//oooruunnddooootttSSS...
[45] A. Gu and T. Dao. Mamba: I Linear-Timee Sequence Modeling with Selective State Spaces. In The First Conference on Language Modeling, COLM 2024, Philadelphia, Pennsylvania, USA, October 7-9, 2024, Oct 2024. URL https: /pwpreeieeeeeeeeeeeoeeeeeee hett/forummid=tEEettee
[46] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré. HÎPPO: Recurrent Memory with Op- timal Polynomial Projections. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H.-T. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020. URL https::/prooceeiinsseeeee wwzt0bnbeth3at 2twt:////0/00/1111221000eeeeee http://222bbbbttaaacca
15

---

[47] A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré. Combining Re- current, Convolutional, and Continuous time Models with Linear State Space Layers. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 572-585, 2021. URL https://proceedingg neurips. wcc/paper/202///aee Il.
[48] A. Gu, K. Goel, A. Gupta, and C. Ré. On the Parameterization and Initialization of Diagonal State Space Models. In S. Koyejo. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neu- ral Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, Novem- ber 28 December 9, 2022, 2022. URL http::/papers: wwwppperrr..is.............e..em cc/paper f files/paper) frence. htmll
[49] A. Gu, K. Goel, and C. Ré. Efficiently Modeling Long Sequences with Structured State Spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: net/
[50] D. Guo, C. Xu, N. Duan, J. Yin, and J. J. McAuley. Longcoder: A long-range pre-trained language model for code completion. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 12098-12107. PMLR, 2023. URL https:://p/ooo https://proceeddngs mlr. bress/v202/ 2atpa//v22gg/233// guo23j .html,
[51] D. Guo, Q. Zhu, D. Yang. Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang. Crepeek-Coders When the Large Language Model Meets Programming The Rise of Code Intelligence, 2024. URL https://doi. org/10. /000 485507 arXiv 2401 14196. arviv: 2401.14196.
[52] C. Han, Q. Wang, H. Peng, W. Xiong, Y. Chen, H. Ji, and S. Wang. LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models. In K. Duh, H. Gómez-Adorno, and S. Bethard, editors, Pruceedings of the 2024 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 3991-4008. Association for Computational Linguisties, 2024. doi: 10.18/53/V 1/002.NAAALLLLLON 222. URL /tttps://doi.ooo http::::/66....0/000116/////////
18653/v1/2024 http:////2222..naa1111a maa.l-long 222.
[53] G. E. Hinton, J. McClelland, and D. Rumelhart. Distributed Representations. In M. A. Boden, editor, The Philosophy of Artificial Inielligence, Oxford readings in philosophy, pages 280. Oxford University Press, 1990.
[54] X. Ho, A. D. Nguyen, S. Sugawara, and A. Aizawa. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In D. Scott, N. Bel, and C. Zong, edi- tors, Proceedings of the 28th International Conference on Computational Linguisties, COL- ING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6609 6625. International Committee on Computational Linguistics, 2020. URL http:///6...rrg/1..16655
2U2U. main. 580.
[55] S. Hochreiter and J. Schmidhuber. Long Short- Term Memory. Neural Comput., 98): 1780, 1997. doi: 19.1162/EECO.1199......1.
URL https://doi.org/// 11tg////...2//ccc.
[56] C.-P. Hsich, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, and B. Ginsburg. RULER: Whats the Real Context Size of Your Long-Contess Language Models? In The First Con- ference on Language Modeling, COLM 2024, Philadelphia, Pennsylvania, USA, October 7-9, 2024, Oct. 2024. URL http:::/opaarreveew..eettt wtt.://www.....e.ee.oooeeeeeeeee
[57] J. Huang. How Well Can Long Sequence Model Model Long Sequences? Comparing Ar- chitectural Inductive Biases on Long-Contexx Abilities. In O. Rambow, I.. Wanner, M. Apid- ianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert, editors, Proceedings of the 31st
16

---

International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pages 29 39. Association for Computational Linguistics, 2025. URL https: https::/aalanthooooo
org/ 2025. coling-main.
[58] J. Huang, Lu, and Q. Zeng. Calibrated language models and how to find them with label smoothing. In Forry-second International Conference Machine Learning, 2025. URL Sttps://openrest
net/forum soLLo 412EEL
[59] L. Huang. S. Cao, N. N. Parulian, H. Ji, and Wang. Efficient Attentions for Long Doc- ument Summarization. In K. Toutanova, A. Rumshisky, L. Zettlemoyer. D. Hakkani-TTe Tür, 1. Beltagy, S. Bethard, Cotterell, T. Chakraborty, and Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1419-1436 Association for Computational Linguistics, 2021. doi: 10. 10.18653/V1/2021. NAACL-MAIN.112 URL https://doi. http::://doooor/100000 11653661/22222111 nat.:i.naaam-mmamam 112,
[60] B. Hui, J. Yang. Z. Cui, J. Yang. D. Liu, I. Zhang. T. Liu, Zhang, B. Yu, K. Dang. A. Yang. R. Men, F. Huang, X. Ren, X. Ren, J. Zhou, and J. Lin. Qwen2.5-Coder Technical Report, 2409. 12186, arXiv: 2409.12186
2024. URL https://doi. org/ 10. 48550/
[61] D. Hutchins, I. Schlag, Y. Wu, E. Dyer, and B. Neyshabur. Block-Recurren Transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Oh, editors, Advances in Neurul Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, IA, USA, November 28- Decem-
ber 9, 2022, 2022. URL http://papers nips catps.cc/panee cc/paper.ppp... files/ walle/paper//a /otpe//2222/haaae htm.l
abeUbbbYI
[62] K. Irie, I. Schlag, R. Csordás, and J. Schmidhuber. Going Beyond Linear Transformers with Recurrent Fast Weight Programmers. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 202 virtual, pages 7703-7717,
btt.
neurips...... www..pp..ccppppelt.. ccppa/
[63] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive Mixtures of Local Experts. Neural Comput., 3(1):79-87, 1991. doi: 10.1 199.1/ URL https:
//doi . 10. 116.0.e/ee............. 1991.3.1.79.
[64] A. Jaech, A. Kalai, A. Lever, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Camney, A. Iftimie, A. Karpenko, A. T. Passos, A. Neitz, A. Prokofiev, A. Wei, A. Tam, A. Bennett, A. Kumar, A. Saraiva, A. Vallone, A. Duberstein, A. Kon- drich, A. Mishchenko, A. Applebaum, A. Jiang, A. Nair, B. Zoph, B. Ghorbani, B. Rossen, B. S.kooloosky, B. Barak, B. McGrew, B. Minaiev, B. Hao, B. Baker, B. Houghton, B. McK- inzie, B. Eastman, C. Lugaresi, C. Bassin, C. Hudson, C. M. Li, C. d. Bourcy, C. Voss, Shen, C. Zhang, C. Koch. C. Orsinger,
Hesse, C. Fischer, C. Chan, D. Roberts, D. Kap- pler, D. Levy, D. Selsam, D. Dohan, D. Farhi, D. Mely, D. Robinson, D. Tsipras, D. Li, D. Oprica. E. Freeman, E. Zhang, E. Wong. E. Prochl,
Cheung. E. Mitchell, E. Wal- lace, E. Ritter, F. Mays, F. Wang, F. P. Such, F. Raso, F. Leoni, F. Tsimpourlas, F. Song, Sulit, G. Salmon, G. Parascandolo, G. Chabot, G. Zhao, G. Brockman, G. Leclerc, H. Salman, H. Bao, H. Sheng. H. Andrin, H. Bagherinezhad, H. Ren, H. Light- man, H. W. Chung, I. Kivlichan, I. O'Connell, I. Osband, I. C. Gilaberte, and I. Akkaya. OpenAI ol System Card. CoRR, abs/2412.16720, 2024. doi: 00-4850/ARXII.2412..7777 URL httts://doi. http///.............6//////.//./ 2412. 16720, arXiv: 2412. 16720.
F. v. Lohmann,
48550/arX5v
[65] Jelassi, D. Brandfonbrener, S. M. Kakade, and F. Malach. Repeat After Me: Transformers are Better than State Space Models at Copying. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
net formmm/rmmmm/ i RidduuRRGodRRRdRRGTTT
[66] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. L. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, Lavril, Wang, T. Lacroix, and W. E. Sayed. Mistral 7B, 2023. URI. https: / doi org/ org/10.0 harr///...55//aa. 4855008855i11 2310. 06825, arXiv: 2310.06825.
17

---

[67] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary. C. Bamford, D. S. Chaplot, d. L. Casas, E. B. Hanna, F. Bressand. G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, I. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang. Lacroix, and W. E. Sayed. Mixtral of Experts, 2024. URL https://doi. org/ 10. 48550/ arxiv. 2401. 04088, arXiv: 2401 104088.
[68] H. Jiang, Y. Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. Abdi, D. Li. C.- Y. Lin, Y. Yang. and L. Qiu. Minference 1.0: Accelerating Pre-filling for Long- Context LLMs via Dynamic Sparse Attention. In A. Globersons, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in Neu- ral Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http://papers wwtppprrcppppp.pppppppppppppp..p files/ patps///////////ssll/ AbSLRccC AoA.
[69] M. I. Jordan and D. E. Rumelhart. Internal World Models and Supervised Learning. In L. Birnbaum and G. Collins, editors, Proceedings of the Eighth International Workshop (ML91), Northwestern University Evanston, Illinois, USA, pages 70-74. Morgan Kaufmann. URL ittps://don org 10. 10167
1991. doi: 6978- 1- 500000 50018-0
[70] M. Joshi, E. Choi, S. Weld, and L. Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In R. Barzilay and M. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Van- couver, Canada, July 30 August 4, Volume 1: Long Papers, pages 1601-1611. Association for Computational Linguistics, 2017. URL https:t//d/aec //doi. org/ 10. 11100000166//////7-1112
[71] A. Kamath, J. Ferret, S. Pathak, N. Vicillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J.-B. Grill, S. Ramos, E. Yvinec, M. Cas- bon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Buse-Fekete, A. Feng. N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J.-T. Peter, D. Sinopalnikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Kolesnikov, A. Ben-
debury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang. A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. J. Carey, C. Brick, D. Deutsch, D. Eisen- bud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Plucinnss tor, and I. Nardini. Gemma 3 Technical Report. CoRR, abs/2503. 19786. 2025. doi: URL sttps://doi.org 48550/ arxiv. 2503. 19786,
H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, Szpek-
10.48550/ARXII.250...o0 arXiv: 2503.19786.
[72] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast Au- toregressive Transformers with Linear Attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156-5165. PMLR, 2020. URL http://proceedinng: mml..mmr..... wwtppp/wee/eepeeeeetepppp wwtpss/vv191kaaaaa http:////////aaaoooooooo//0 html.
[73] D. P. Kingma and Ba. Adam: A Method for Stochastic Optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: /arxivv. org/ abs/ 1412. 6980.
[74] Kitaev, L. Kaiser, and A. Levskaya. Reformer: The Efficient Transformer. In 8th In- ternational Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URI. https:://openreveew wttp://nwwweieeeeee//t/mrmmo net/forum?id TkgVKKHtVB.
18

---

[75] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet Classification with Deep Con- volutional Neural Networks. P. L. Bartlett, C. N. Pereira, C. C. Burges, L. Bot- tou, and K. Q. Weinherger, editors, Advances in Neural Information Processing Sys- tems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Pro- ceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106-1114, 2012. URL http:://rroeeedngg..ee.em. new.ips..urpp..... httpr/ppss.pppppp/22//////////// hlm.l
[76]
D. Lepikhinn H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Aus- tria, May 3-7, 2021. OpenReview.net, 2021. URL grwe/AHIme /XHMYD.
net/forum?id
[77] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-Augmented Generation for Knowledge-Intensiv NLP Tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Bal- can, and H.-T. Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12. 2020, virtual, 2020. URL
DDC /481 Abstract htm
[78] and D. Roth. Learning question classifiers. In 19th International Conference on Com- putational Linguistics, COLING 2002, Howard International House and Academia Sinica, Taipei, Taiwan, August 24 September 1, 2002, 2002. URL https://aclaantoolog. org/
CO2- 11507.
[79]
X. L. Li and P. Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In C. Zong. F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4582-4597 Association for Computational Linguistics, 2021. 1901:///00016633///22201
doi: 10. 1 URL.
oci-iong.35.
[80] H. Liu, M. Zaharia, and P. Abbeel. RingAttention with Blockwise Transformers for Near- Infinite Context. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openrevies.
[81] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the Middle: How Language Models Use Long Contexts. Trans. Assoc. Comput. Linguistics, 12: 157-173, 2024. doi: 10. 10.1162/TACL_A A_ 00638. URL https://doi. org/ 111.1...111/cc..../
00638,
[82] T. Liu, C. Xu, and J. J. McAuley. Repobench: Benchmarking repository-level code auto- completion systems. In The Twelfth International Conference on Learning Representa- tions. ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL Attps: //openreiie net www.:eee/ofoouoeee http://oormmiddp//iioo.t
[83] X. Liu, H. Yan, C. An, X. Qiu, and D. Lin. Scaling Laws of RoPE-based Extrapolation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: netr/foruu
hd 107
[84] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, and A. Roberts. The Flan Collection: Designing Data and Methods for Effec- tive Instruction Tuning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Muchine Learning, ICML 2023, 23- 29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22631-22643 PMIR, 2023. URI. sttps:// httpas///ooceediings.. www.eddiinnmmmm......mmmmm.. mlr. 7202/longpre html.
press/ press/
19

---

[85] I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL.
[86] P. Lu, S. Wang, M. Rezagholizadeh, B. Liu, and I. Kobyzev. Efficient classification of long documents via state-space models. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6559- 6565, Singapore, Dec. 2023. Association for Computational Linguisties. doi: 10.18653/v1/ 2023. 3023.emllp-main400k 404. URL
org./rrg//// 2023. emnlp-mmalme....e wwa.a.a............ 44047
[87] P. Lu, I. Kobyzev, M. Rezagholizadeh, Chen, and Langlais. Regla: Refining gated linear attention. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL 2025 Volume I: Long Papers. Albuquerque, New Mexico, USA, April 29 May 4, 2025, pages 2884-2898, Association for Computational Linguistics. 2025. doi: 10. 10.5 maa.m-lon.....4444
URL https://doi. org/
[88] S. Massaroli, M. Poli, D. Y. Fu, H. Kumbong, R. N. Parnichkun, D. W. Romero, A. Timalsina, Q. McIntyre, B. Chen, A. Rudra, C. Zhang, C. Ré, S. Ermon, and Y. Bengio. Laugh- ing Hyena Distillery: Extracting Compact Recurrences From Convolutions. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA. USA, December 10 16. 2023, 2023. URL http://papers. http://pppprr..pp.....pp.p cwppaperpppppppppppppppppppe..ep files/par hwwes/paperr/023/// pappp//022/aasa. 6 311hhbed42aaat 834 111.a883411116666000 463899/ Aontracc
lt
[89] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer Sentinel Mixture Models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: 7/ //openrevier mett/orrumiddBy:y220.
[90] T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. Kale, J. Love, P. Tafti, L. Hussenot, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, Ippolito, Reid, E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G.-C. Muraru, G. Rozhdestvenskly. H. Michalewski, I. Tenney, I. Grishchenko, J. Austin, Keeling, Labanowski, J.-B. Lespiau, Stanway, J. Brennan, J. Chen, J. Ferret, J. Chiu, and e. al. Gemma: Open Models Based on Gemini Research and Technology. CORR, 0bs/24403.08995 2024. doi: 10.48550/ARXIV.4400 08295. URL https://doi.oow htt.//di..org//// 10. 00l.48850/aaaa arx.v./rrvvv.... 2403. 08295, arXiv: 2403.08295.
[91] A. Mohtashami and M. Jaggi. Random-Access Infinite Context Length for Transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::/pppess..niiiie nips. wwtpppprrccppppppppppppp
abUsdGebt SebbtcGbb33aaaGGGG oe http://cco...ottttttaaaaooooooao htilles/paper//22///aaas
ctt
[92] OpenAL. GPT-4 Technical Report, 2023. URL https://doi.org 400///000.555//rx111 2303. 08774
arXiv: 2303.08774.
[93] A. Orvieto, S. L. Smith, A. Gu, A. Fernando, Ç. Gülçehre, R. Pascanu, and S. De. Res- urrecting Recurrent Neural Networks for Long Sequences. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 26670-26698, PMLR, 2023. URL
https: https://proceedingg mlr. press /atss/v222//a0 www.e///oviet2aaa.... html,
[94] J. Park, Park, Z. Xiong, N. Lee, Cho. S. Oymak, K. Lee, and D. Papailiopoulos. Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks. In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27. 2024. OpenReview.net, 2024. URL https: httttp:://opeereiie..t net wwtp://ooooo..dd0oooooooeo.eee..
20

---

[95] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, At- lanta, GA, USA, 16-21 2 June 2013, volume 28 of JMLR Workshop and Conference Proceed- ings, pages 1310-1318. JMLR.org. 2013. URL http://proceediings... htr.eisssssir.sssssss.ess. v228/
pascanu13. ntm.
[96] B. Peng, J. Quesnelle, H. Fan, and E. Shippole. YaRN: Efficient Context Window Exten- sion of Large Language Models. In The Twelfth International Conference on Learning Rep- rexentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
https://openrevient net/ feet/formm?? feeerummdd==llle Widd:didd=bhhhbe xhZulu.
[97] F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. H. Miller. Lan- guage Models as Knowledge Bases? In Inui, Jiang, V. Ng, and X. Wan, editors, Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNE 2019, Hong Kong, China, November 3-7, 2019, pages 2463-22222 2473.. Association for Computational Linguistics, 2019. doi: 10.18653/V 1/D19-120. URL https: http:://dooo.oo////// org/10. 18653/v1/ D19-1250.
[98] M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré. Hyena Hierarchy: Towards Larger Convolutional Language Models. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarleu, editors, International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28043-28078. PMLR, 2023, URL attps://proceedi
mlr. press 7202/poli23a. html
[99] M. Poli, A. W. Thomas, E. Nguyen, P. Ponnusamy, B. Deiseroth, K. Kersting, T. Suzuki, B. L. Hie, S. Ermon, C. Ré, C. Zhang, and S. Massaroli. Mechanistic Design and Scal- ing of Hybrid Architectures. In Forty-first International Conference Machine Learn- ing. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:
/openreview hett/forumiidd-Dp
[100] O. Press, N. A. Smith, and M. Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openrevien nww/ pet/forrum/id-R8s0ppooo
[101] Z. Qin, S. Yang, and Y. Zhong. Hierarchically Gated Recurrent Neural Network for Sequence Modeling. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, edi- tors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans. LA, USA, December 10 16. 2023, 2023. URL http://paperr: nips cC/ paper files/ willespapped http:/pppppp2222//222 Tateeeelee-aatatatc.cc
65489
[102] Z. Qin, S. Yang, W. Sun, X. Shen, D. Li, W. Sun, and Y. Zhong. HGRN2: Gated Lin- ear RNNs with State Expansion. In The First Conference Language Modeling, COLM 2024, Philadelphia, Pennsylvania, USA, October 7-9, 2024, Oct. 2024. URL https:
//openrevier.o wwwpenrereei.net/forunnticoo. www.nn/eoooumiidddssooooo.
[103] J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap. Compressive Trans- formers for Long-Range Sequence Modelling. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openrevie/ wwwpeeeeeeoooooeooeeee ferss//frummmae 1yy
[104] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Teas Transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL https://jmlr. orgppapers/v21/
20-074. htmll
[105] M. Rivière, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, Shahriari, A. Ramé, J. Ferret, P. Liu, Tafti, A. Friesen, M. Casbon, S. Ramos, R. Ku- mar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev,
21

---

M. Hoffman, S. Thakoor, J.-B. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Severyn, Parrish, A. Ahmad, A. Hutchison. A. Abdagic, A. Carl, A. Shen, A. Brock, A. Co- enen, A. Laforge, A. Paterson, B. Bastian, B. Piot, B. Wu, B. Royal, C. Chen, C. Kumar, C. Perry, C. Welty, C. A. Choquette-Choo Choo, D. Sinopalnikov, D. Weinberger, D. Vijaykumar, D. Rogozinska, D. Herbison, E. Bandy, E. Wang. E. Noland, E. Moreira, E. Senter, E. El- tyshev, F. Visin, G. Rasskin, G. Wei, G. Cameron, G. Martins, H. Hashemi, H. Klimczak- Plucinska, H. Batra, H. Dhand, I. Nardini, J. Mein, J. Zhou, J. Svensson, J. Stanway, J. Chan, J. P. Zhou, Carrasqueira, J. Njjzi, J. Becker, Fernandez, J. v. Amersfoort, J. Gordon, J. Lipschultz, J. Newlan, J.-y. Ji, K. Mohamed, K. Badola, K. Black, K. Millican, K. Mc- Donell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjösund, L. Usui, L. Sifre, L. Heuer- mann, L. Lago, and L. McNealus. Gemma 2: Improving Open Language Models Practical Size. CoRR, abs/2408 0bs/2400.00118, 2024. doi: 10,48550/ 105550/AXXX..2408.00002 18. URL https://dofs orggoogg/// 1t///10..8800 48550/ arriv. 2408.00118 arXiv: 2408.001 18.
[106] A. Roberts, C. Raffel, and N. Shazeer. How Much Knowledge Can You Pack Into the Parameters of a Language Model? In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing. EMNLP 2020, Online, November 16-20, 2020, pages 5418-5426. Association for Com- putational Linguisties, 2020. doi: 10. 10.188////II0 hTt.://2222222222222 EMNLP- URL https: 7/doi. org/ 10. 1t..186661122222 emn 1000mmann... 437.
[107] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, Evtimov, J. Bitton, M. Bhatt, C. Canton-Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L.. Martin, N. Usunier, T. Scialom, and G. Synnaeve. Code Llama: Open Foundation Models for Code, 2023. URL https: //doi www.://///..../////./////// arviv 2308. 12950.
arXiv: 12950.
[108] W. J. Rugh. Linear system theory. Prentice-Hie Hall, Inc., 1996.
[109] D. E. Rumelhart. Parallel distributed processing. 9th Edition. MIT Pr., 1989. ISBN 978-0-
262-68053-0. URL https: : //www. worldcat. ort/
[110] P. Sadegh and J. C. Spall. Optimal random perturbations for stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Trans. Autom. Control., 43(10);
1480-1484, 1998. doi: 10. 109999720511.
[111] R. Salakhutdinov and G. E. Hinton. Deep Boltzmann Machines. D. A. V. Dyk and M. Welling, editors, Proceedings of the Twelfth International Conference on Artificial In- telligence and Statistics, AISTATS 2009, Clearwater Beach, Florida, USA, April 16-18, 18, 2009, volume 5 of JMLR Proceedings, pages 448 455. JMLR.org, 2009. URL pttp: \/proceedings.. mlr. prwpsrreaeeaeee skkbsaaoooutttoooo htmll
[112] Salimans and D. P. Kingma. Weight Normalization: A Simple Reparameterizztion to Accel- erate Training of Deep Neural Networks. D. D. Lee, M. Sugiyama, U. v. Luxburg. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Con- ference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, page 901, 2016. URL httpa///ee0oaaaaoe///00...aa....
Abstract..hmm.
[113] I. Schlag, K. Irie, and J. Schmidhuber. Linear Transformers Are Secretly Fast Weight Pro- grammers. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Con- ference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9355-9366. PMLR, 2021. URL http: http://proceedingss mlr
html
[114] J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao. FlashAttention- 3: Fast and Accurate Attention with Asynchrony and Low-precision. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL. http::/tppp:ss...t.ette nips. cc/ wwwpapprpppppe.peecpppeeeee files/ wtteee/ppppr2222//haaaee SGG.:
hlmll
22

---

[115] U. Shaham, E. Segal, M. Ivgi, A. Efrat, O. Yoran, A. Haviv, A. Gupta, W. Xiong, M. Geva, J. Berant, and O. Levy. SCROLLS: Standardized CompaRison Over Long Language Se- quences. In Y. Goldberg, 7. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 12007-12021. Association for Computational Linguistics, 2022. doi: https://doi.org// 1rg.//..1653/v........ 18653/v1/202/ emn commp-mmiin 823.
URL
[116] U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy. ZEnSCROLLLS A Zero-Shot Benchmark for Long Text Understanding. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguisties: EMNLP 2023, Singapore, De- cember 6-10, 2023, pages 7977-7989. Association for Computational Linguistics, 2023, URL htpp:p/////01.../110001011
doi:
FTN.2......... 536.
2
[117] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song. M. Zhang, Y. K. Li, Y. Wu, and D. Guo. DeepSeek- Math: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024, URL https://doi... org/ 10. 48550/ arx.v. 2402. 03300, arXiv: 2402.03300.
[118] F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei. Language Models are Multilingual Chain-of-Thought Reasoners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https::/ het/forum?id=fRkkte
[119] C. Spall. Introduction to stochastic search and optimization estimation, simulation, and control. Wiley-Interscience series in discrete mathematics and optimization. Wiley, 2003. ISBN 977-0-471-33305--3 doi: 10. 10.000004777222138...
[120] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929- 1958, 2014. doi: 10.555526274335267033 URL https://al. acm. axg/dol/10.555 55557 2627435 2670313
[121] J. Su, M. H. M. Ahmed, Y. Lu, S. Pan, W. Bo, and Liu. RoFormer: Enhanced trans- former with Rotary Position Embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/ J.NEUCOM.200 202..12206.2 URL /ttp:::/da...ogg//1.00000.. neucom.. 2023. 127063,
10.
[122] Y. Sun, L. Dong, B. Patra, S. Ma, S. Huang, A. Benhaim, V. Chaudhary, X. Song, and F. Wei. A Exngth-EEtrappolttable Transformer. In A. Rogers, J. L. Boyd- and N. Okazaki, edi- tors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 14590-14600 Association for Computational Linguistics, 2023. doi: 10. N...1855//1/2223.A00 12101/102223..1LOOOOOO URL https:///oi...... http:///oo...gg///// 111131811110000 acl- 816
[123] I. Sutskever, Martens, and G. Hinton. Generating Text with Recurrent Neural Net- works. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 July 2011, pages 1017-1024, 2011. URL https: http:///mmcmccc///////ee//
imml paper. pdf.
[124] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Ken Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. T. Barron, and R. Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In H. Larochelle, M. Ranzato, R. Hadsell, M.- F. Balcan, and H.-T. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, Decem- ber 6-12, 2020, virtual, 2020. URL https:://rooceee:e www.eeeedddinnn....i..........ee cc/ pappp//ppr//200 Aww.eaA-aaatttt.eee html
[125] Tay, M. Dehghani, S. Abnar, Shen, D. Bahri, Pham, Rao, L. Yang, S. Ruder, and D. Metzler. Long Range Arena : A Benchmark for Efficient Transformers. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, CrCk
2021. OpenReview.net, 2021. URL https://openrreveew wtt.:errrrreeeeeeeooouuooooo.aad
23

---

[126] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. Efficient Transformers: A Survey. ACM Comput. Surv., 55(6):109:1-1099 2023. doi: 10.1 10./3353353011..
[127] Н. Touvron, T. Lavril, G. Izacard, Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and Efficient Foundation Language Models, 2023. URL https://doi.org: (8550/arXiv 2302. 13971
arXiv: 2302.13971.
[128] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Ba- tra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, Goyal, A. Hartshom, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. Koura, M.-A. Lachaux, Lavril, J. Lee,
Liskovich, Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Sal- adi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. URL https: //doi org/ 10. 405.44850/////// arviv.
arXiv: 0907..9288.
[129] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All you Need. In I. Guyon, U. Luxburg, S. Ben- gio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017. Long Beach, CA, USA, pages 5998 6008, 2017. URL https://proceeddngss neurips.com cttpppp.ccppppp////////aa 7Tb88243547G8691 whdnhnt c4854haa Abstract h.
[130] J. Wang, D. Paliotta, A. May, A. M. Rush, and T. Dao. The Mamba in the A. Globersons, Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 files/ paper/ htmll
Llama: Distilling and Accelerating Hybrid Models.
15, 2024, 2024. URL http://papers
files/paper.reee haaper/024/haaar 2024/haahh
[131] S. Wang. B. Z. Li, M. Khabsa, H. Fang. and H. Ma. Linformer: Self-Attention with Linear Complexity, 2020. URL https://arxil: org/ abs/2006. 04768, arXiv: 2006.04768.
[132] X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-Soonis Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL 1d=1PLINIMPIr..
[133] Wang. Ma. J. Huang, Lu, P. Parthasarathi, X.-W. Chang, B. Chen, and Y. Cui. Res- ona: Improving context copying in linear recurrence models with retrieval. In The Second Conference on Language Modeling, COLM 2025, Montréal, Canada, September 3-6, 2025, net net/forum hm=
Sept. 2025. URL
[134] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q V. Le. Finetuned Language Models are Scro-Shot Learners. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openrerviex www/ferruuunn www.ummdddggrrrroo LdogZZGGoozdqR
[135] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, Zhou, D. Metzler, H. Chi, T. Hashimoto, O. Vinyals, Liang, J. Dean, and W. Fedus. Emergent Abilities of Large Language Models. Trans. Mach. Learn. Res., 2022, 2022. URL https://openrevicc wwt.://eweeee.rmmmaaaaaassssssu
[136] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, Chi, Q. V. Le, and D. Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In
24

---

S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information P Processing Systems 2022, NeurIPS 2022, New Orleans, IA, USA, November 28 Decem- ber 9, 2022, 2022. URL http://papers.ntte nips wttp.cccppppe.. files/ Absaaaaaaa 2ttpp//eeeeee/apee/2/////////// pttpppppp2//aaaaa///ep
9d56U961
htm.l
[137] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue. A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. v. Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: othtc-of-the-Art Natural Language Processing. In Q. Liu and D. Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, EMNLP 2020 Demos, Online, November 16-20, 2020, pages 38-45. Association for Computational Linguistics, 2020. doi: https: wtt.://.///....///1.1111.. 18653/v1/20080 ort.:///oo..1111100000mmmaeaeeaa
URL
[138] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024, URL https:: het/forummid:NG7500
[139] M. Xu, X. Men, B. Wang. Q Zhang, H. Lin, X. Han, and W. Chen. Base of RoPE Bounds Context Length. In A. Globersons, L. Mackey, D. Belgrave, Fan, U. Paquet, M. Tomezak, and C. Zhang, editors, Advances in Neu- ral Information Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http://papers: nips wttps.ccpppper.e files/pars https//ppp///2//aaa//// 921203320522 http://wwww.aaaaaaauaeae........ hotp::/oooootttttaaaooooooooeee
htm
[140]
A. Yang, B. Yang. B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang. J. Wang, J. Yang. J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang. M. Li, M. Xue, N. Ni, P. Zhang, P. Wang,
R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge,
X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, X. Liu, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, 7. Cui, Z. Zhang, 7. Guo, and 7. Fan. Qwen2 Technical Report, 2024. 18850//a0ii..../// arw.v. 2407 10671. arXiv: 2407.10671
URL / org/ 10. 48550/
[141] A. Yang, B. Yang. B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang. H. Wei, H. Lin, J. Yang, J. Tu, Zhang. J. Yang, Yang. J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang. L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, 7. Zhang, and Z. Qiu. Qwen2.5 Technical Report, arXiv: 2412.15115.
2024. URL https:: //dtss://do..... org/ 10. 48550/ arxiv. 2412. 15115,
[142] A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, Tu, J. Zhou, Lin, K. Lu, M. Xue, Lin, T. Liu, X. Ren, and Z. Zhang. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via cell-Improvemeet, 2024. URL https: org/10 48550/ar52/ aa.iv. 2409. 12122
arXiv:
[143] S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated Linear Attention Transform- ers with Hurdwarr-Efficiee Training. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 2 21-27, 2024. OpenReview.net, 2024. URL https://openrevicce www.anrrvvee.nneffouuurllte..... net/forur//ddiiasxxZZ..
[144] S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim. Parallelizing Linear Trans- formers with the Delta Rule over Sequence Length. In A. Globersons. L. Mackey, D. Belgrave, A. Fan, U. Paquet, M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Anmual Conference on Neural Informa- tion Processing Systems 2024, NeurIPS 2024, Vancouver. BC, Canada, December 10 15, 2024, 2024. URL http://papers: nips cc/paper files/ / http:/aperr2/000 happr//022/hash/ 215.
htm.l.
[145] Z. Yang, Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In E. Riloff,
25

---

D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, Brussels, Belgium, October 31 Novem- ber 4, 2018, pages 2369- 2380. Association for Computational Linguistics, 2018. URI 1ttp:/00000000/111--222220
[146] Z. Ye, K. Xia, Y. Fu, X. Dong. J. Hong. X. Yuan, S. Diao, J. Kautz, P. Molchanov, and Y. C. Lin. LongMamba: Enhancing Mamba's Long-Context Capabiliitis via Training- Free Receptive Field Enlargement. In The Thirteenth International Conference on Learning Representationss ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openrevien net/Eorum?id--Moo.s.
[147] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontañón, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed. Big Bird: Transformers for Longer Sequences. In H. Larochelle, M. Ranzato, R. Hadsell, M.-.. Balcan, and H.-T. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceeddnngs newrips wtt.riis.ccppppppaaee hotpp//s....pcpppp///////////// c85 2TT.://4///411111111//1212211111
/
[148] Q. Zeng, J. Huang, P. Lu, G. Xu, B. Chen, C. Ling, and B. Wang. ZETA: Leveraging Z-- order Curves for Efficient Top-k Attention. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OvenReview.net, 2025. URL https://openreviii
net fett/forumidddjjWaaaa
[149] X. Zhang, Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang. Z. Liu, and M. Sun. nftyBench: Extending Long Context Eyaluation Beyond 100K To- kens. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 15262-15277. Association for Computational Linguistics, 2024. doi:
URL https:
//doi org 10. 11w53//1//002202 acl long. .844.
[150] Y. Zhang, S. Yang, R.-J. Zhu, Y. Zhang, L. Cui. Y. Wang. B. Wang, F. Shi, B. Wang. W. Bi. P. Zhou, and G. Fu. Gated Slot Attention for Efficient Linear- Time Sequence Modeling. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neu- ral Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: www.::sttttttrttt......oo Confrrrrracctttcaaaeccce... htm
nips. ct/ wwwpppppppppppececcccccccccc.eep files/ happ//apppr///// / Setaperr222//heee 2024/hash/
DKT3985 11r339511ht6241 1w1111111646000 wtC.k4Arbcccccccc8556300
[151] Y. Zhang, S. Yang, R.-J. Zhu, Y. Zhang, L. Cui, Y. Wang, B. Wang, F. Shi, B. Wang. W. Bi, P. Zhou, and G. Fu. Gated Slot Attention for Efficient Linear-Time Sequence Modeling. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neu- ral Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL 63t33e5 www.99111166644 18
nips. cc/pppercppppprepp files/pane (aaper/2024/ash))
htm
[152] 7. Zhang, Sheng, T. Zhou, Chen, I.. Zheng, Cai, 7. Song, Tian, C. Ré, C. W. Barrett, Z. Wang, and B. Chen. H20: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL
.tpsscc/papp....... paper wilesspapeet pttppppppp/////a/a/ppp h2hh/hhhh/h//// ort.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims come with experimental support well as theoretical proof's when necessary.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide this in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
We provide proof to all these in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all these details in full.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
We mention which code bases we base our experiments off of.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.cc/ppp /nips.cc/ppbllc/uuiddde Coweiedeee/ooSSeemmieeee http:///ddeeeeeeeeeeeaaaaaaaaaaa
www.iiooiiipyyyyo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specifyy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide these details in our experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We show these within our plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
29

---

is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes] We provide this in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics litpss:
Answer:
[Yes]
Justification:
We have complied with the NeurIPS Ethics Guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide this in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to
30

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not introduce anything within this work that would be interpreted as having possibility of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The authors cite the original paper that produced used code packages and dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the pack- age should be provided. For popular datasets, saperswithcode www. datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
31

---

Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowd-sourcing nor research with human sub-
jects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowd-sourcing research with human sub- jects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
32

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs in any non-standard or novel manner.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: octta:///eeerrrsseccoooooeeeeeee for what should or should not be described.
33

---

A Proofs
A.1 Proof of Lemma
Lemma A. Let B Rdxd be matrix with | dB, and let E I2 be a vector such that each entry of æ satisfies | S az. The upper bound for |Bx 12 is:
|a oB on vd.
Proof. For any vector R it follows that:
Bxxll S B |
Substituting
[B] JB, we obtain:
Bx 2 S OB |
And
||01|2 S
2 Vd dr.
Substituting the bound on |æ||| into the inequality for |xll2 we have the norm of logit vector Rd.
|u|l2 Bxel22 on ||0|| S oB vd o..
A.2 Proof of Theorem
Theorem
Assume the transition matrix A is diagonal with eigenvalues X Uniform Amins Amax] for < Amin < Amax < Suppose the system evolves as
h = Ah- + BEG,
(10)
where xt NN, 1) and B is a weight matrix whose rows are independently sampled as No, 1 Ia. Then, in the limit + co, the expected squared norm of the hidden state converges to
1
Xmin 12 max
log
B[[] Bar Ba ?]
(11)
2(Amax Amin)
1
Proof. We begin by unrolling the recurrence:
h
A Bari..
(12)
2
Assuming stationarity and independence of the inputs T., the expected squared norm steady state
ho 2]
A Bal
(13)
Consider the case of a single unit with eigenvalue X E Dmmin Amax]. The contribution of this unit is:
1B
=
2 B boe
X2
(14)
1=0
where bxl [2]
x² is the contribution from the corresponding row of B, and B is a weight matrix whose rows are independently sampled as b ~ ~N(0, -I). I).
Es[EE, (bb)
E,
||l
34

---

With ~ Uniform/ Amin, Amax,, where I Amin < Amax < 1, the expected contribution over all units
o² o²
E hoc 171
d. E,
dx.
(15)
X2
Amax
Amin
X2
Aman
Evaluating the integral:
Amnax
1
1 - X
dd log
(16)
Amin
1- X2
2
1
X
X2
Hence,
B ho [| odd.
log
min
(17)
2(Amax
Amin)
Since E Bz || d- o², we obtain the final expression:
12
ho0 11 2(Amax Amia)
log
min
B
Bx Bx]
(18)
1 maax
A.3 Proof of Corollary 9.3 and Corollary
Corollary I [Norm of Mamba State] Suppose the diagonal entries A are independently drawn from uniform distribution on [0, A], moderate discretized step value A and the system evolves as h Ah Bx, diag (exp(-Aa)) f ABx. Then the convergence rate p of the expected squared norm of the limiting state satisfies O
A 2X
log 1-xx))
Proof. Given Theorem 2 the convergence rate p of Mamba state can estimated as Amin -
h
h
1 1- X2
lim +
lim 10000
log
(19)
E
B
B
2X
Corollary 9. [Norm of Mamba2 State] Suppose A XI exp(-Aa) is scalar multiple of the identity matrix, where À E (0, 1), moderate discretized step value A and the system evolves as h; = Aht-1 + Bz, exp(-Ac @ h,-1 + Az, Then the convergence rate p of the expected squared norm of the limiting state can be estimated as ((A))
Proof. Given Theorem the convergence rate p of Mamba2 state can be estimated as = Dmmar Aminl
0:
|4 |
AE h
22 min
lim ++0 5
lim
lim 8-20
log
(20)
E
Ba:
Bx:
28
(Amin + 5)2
8-0
Let Amin X, Amna = X + 6,
0+0
Substitute into the expression:
- X2
- X2
p = lim log
lim
log
(21)
8-2 25
1 (A 8)2
5- 28
1 X2 -215 82
Let Amin, 8 Amax X, then:
35

---

X2
lim
log
(22)
5-+0 20
- (A + +862
1 - 22
lim
log
(23)
5-+0 20
- X2 - 226 82
2 + 8²
lim
log
1
(24)
28
1 X2
2λ0 + 82
lim
(25)
8 8 28
X2
AA
(26)
X
B Elaboration on Theorem 4.2
The divergent convergence behavior is irrespective of spectral assumptions. We provide an elabora- tion on Theorem 2 and corresponding convergence analysis without imposing any strict distribu- tion assumptions.
B.1 Setup and notation
Let d, m E N. For l..... (Amin Amax] C. (0, 1). Denote
d let A E (0, 1) be iid. samples from a density p()) supported on
A = diag(A Ad) E Rdddd
Consider the linear system
h, = Ah--1 + Bx,, t2 1,
(27)
where {xe} are i.i.d (0 )m) and B E _dx has i.i.d. rows bi...., b, each distributed as
b, ~ Mn,
with EB E TR. m a (given) covariance matrix.
Assume hg = 0 (or any initial condition that decays under A). We are interested in the steady-state expected squared norm
lim E [[]b]]]3] +00
where the expectation over the driving noise eigenvalues {A,).
(x+} and the random matrix B and the random
B.2
Theorem (Expected state norm under general spectral distribution)
Theorem B.I. Under the assumptions above, the limit [E[] h [1 exists and equals
Amax p())
b 112] = E [x||]
1 X2 d),
(28)
Amin
where x NO, Im]) independent of and X, and the expectation on the left of the product taken over B and x.
Proof. Because A is diagonal and x, A/10, Imn) i.i.d., the process equation 27 is Gaussian with zero mean for all f. Let
Σ, = E h. E I
be the (time-l) covariance of the hidden state. From equation 27 we have the Lyapunov-type recur- sion [108. 2
Σx = AΣ,
Bxx B
36

---

Since x has expectation I and is independent of B and A, the driving covariance is
Q = X B B
E.
Because A is diagonal, the steady-state covariance E == limp-soc E. also diagonal; denote its diagonal entries by s; = (2, L, ...... d. The scalar recurrence for each diagonal entry = 1,5; + Qu.
hence (since |d < 1)
8, Qa
X
Therefore the steady-state expected squared norm equals
Qa
[[(traceEE>)) Because the pairs (Qas A:) are i.idd across and rows of B are independent of the >₂ h,'s, have for a generic row b E Rm
E b [|
E
X
| X2
hoo d E
d B||||
X2
where is a generic draw from p(A) and independence between b and À was used to factor the expectation. Observe that
E Bx|| En [I [] = EB [trace(B ( B)]
EE
||b:
Combining the last two displayed equalities yields equation 28:
Amax p())
[2]
E Bx
X2 = E
[x
1 A2 dA.
Amin
This completes the proof.
B.3 Evaluation of [Bx| in the isotropic row case
each row b, ~ NO, of mm (i.id. across rows), then b|| (ra moi B:
E [x|
= d m
In the special (informal) normalization used in the statement above, where each row has covariance ΣB = (I/vd)I.. one has o 1l/Vd and hence
1
Bx [| = dm
m22/2
Vd
B.4 Asymptotic analysis of the integral
Define
Amnax
p(A)
I(Amin Amaxx)
dd
E
1- X2
1 X2
Amia
(1) As Amax + 1- Near À 1 we have the expansion - X2 = (1 - A)(1 + & 2(1 - A).
Suppose p is continuous and p(1) 0. For close 1,
p())
p(1)
1 1- X2
2
1-X
Hence for Amax sufficiently close to 1,
p(A)
p(1)
Auaux
dd p(1)
Amin
T I(Amin, Amax)
log
Jmmin 1 1
2
1-X
2
Amax
Amii
Therefore
p(1)
T(minn Amax)
log(1 Amax) as Amax + 17,
2
and in particular 1(Aminy )max) t00 with logarithmic divergence.
37

---

(2) As Amax + 0+. When Amax is small, 1- X² = 1, so the integrand is approximately p(A). If p continuous near with p(0) O. then
Amax
T( T(Amin; Amax)
p(A) R p(0) (Amax Amin)-
Amin
If we also take Amin 0 or consider the leading-order scaling in Amax, then
I 1(mmin Amax) ~ p(0) Amax
as Amax + 0+,
ie., I vanishes linearly with Amax.
Combining the prefactor Bx| from equation DX with the asymptotics above yields
p(1) 2
E Bx | log(1 Amax), Amax - 1 1-
hoo [1
p(0) [[|
Bx I?] Amax:
Amax - 0+
38

---

C
Additional Experimental Details and Results
C.1 Technical Details
All experiments were conducted on a single machine with 2 NVIDIA RTX4080 16GB GPUs. Ex- periments were run in an environment using CUDA version 12.6 and PyTorch 2.6.0.
C.2
Constant Scaling Language Modeling Perplexity
Mamba2-2.7B
Mamba-2.88
10
10
PPL
10a
102
10
10a
10a
0
25000
50000
75000 100000125000
25000 50000 75000 100000125000
baseline
A: 1, A: 1/3
A: 1/2, A: 1/2
A: 1/3, A:1 1/3, 4:1/2
1/3, A:1/3
1, A:
1/2, A:1
A: 1/3/3
Figure Language modeling perplexity on ProofPile after applying constant scaling factor to either A or A, The red line with "o" mark indicates the baseline, where neither A nor A, are scaled.
39

---

C.3 MambaExtend Calibration
Here, we give an overview of the calibration functions we use within our MambaExtenddbasee exper- iments. Each of the described methods replace the calibration function CF within Algorithm II In our explicit implementation for calibrating scaling factors for A, we use the same hyperparameters as Azizi et al. [3]]
Calibration via pack-propagation. To train the un-frozen calibration parameters on a calibration set, we apply a back poc-ppropgtttion algorithm to find the optimal scaling factors. This is described in Algorithm
Algorithm 2 Calibration via back- pack-propagation
Input: Frozen model M, calibration set C, initial scaling factors Learning rate 1, perturba- tion magnitude iterations K
2: Output: Learned scaling factors S = .9.....,rr.. E 1R4 x
3: optimizer Adam(S,
4: for k S K do
5: [ = eval (Mcx--.C)
6:
L.backward () optimizer.ster
7:
8:
S + clamp (S, 0.001)
9: end for
10: return
Calibration via zeroth-order optimization. Zeroth-order optimization offers an efficient yet nois- ier method for calibration, relies solely on forward passes approximate gradients. Specifi- cally, this is a multi-iteration process in which, at each iteration, the scaling factors are randomly perturbed using random variable ô sampled from a Rademacher distribution. The magnitude of and
the perturbation and the learning rate for the updates are controlled by the respectively. We employ the two-sided variant of the simultaneous perturbation stochastic approxi- mation method (SPSA) [[]]], which obtains gradient approximations by applying both positive and negative perturbations to the parameters simultaneously. The two-sided SPSA approach yields gra- dient estimates with lower variance than the one-sided version, thus enhancing accuracy, especially in noisy environments
[ This is described Algorithm
Algorithm Calibration via zeroth-order optimization
Input: Frozen model M, calibration set C, perturbation magnitude c, iterations K
2: Output: Learned scaling factors $ [s1...., 81) E Rd
3: for k < K do
4:
E [d,xL Radamacher()
5:
St
S S
6:
S S --cx6 8
7:
gt eval(MMxxt
8:
eval(Mexxs,CC)
9:
VS (e+ (2
10:
S S V
11: S + clamp (S, 0.001)
12: end for
13: return
40

---

C.3.1 Language Modeling Perplexity
Mamba2 Language Modeling Perplexity
GovReport 130M
GovReport 370M
GovReport 780M
GovReport 1.38
GovReport 2.7B
1e5
164
1c3
162
1e1
PG19
PG19
PG19 780M
PG19 1.3B
PG19 2.7B
130M
370M
1e5
104
1e3
1e2
161
PPL
Pile 130M
Pile 780M
Pile 1.38
Pile 2.7B
370M
1e5
164
1e3
1e2
1e1
ProofPile 130M
ProofPile 370M
ProofPile 780M
ProofPile 1.38
ProofPile 2.7B
165
184
103
1e2
1e1
22
2
2
2
2
2.5
21
2
2
2
24
2
2
2
2
Length
Calibrate
Delta
Figure 6: Language Model Perplexity performance of Mamba2 models by calibrating scaling factors for either log (A) (red lines) or A, (cyan lines). Perplexities are reported across various datasets (GovReport, PG19, ProofPile, Pile) well as model sizes.
41

---

Mamba Language Modeling Perplexity
GovReport 130M
GovReport 370M
GovReport 790M
GovReport 1.48
GovReport 2.8B
166
164
162
PG19
PG19 370M
PG19 790M
PG19 1.4B
PG19 2.8B
130M
166
184
1e2
PPL
Pile
Pile 370M
Pile 790M
Pile 1.4B
Pile 2.8B
130M
1c6
164
1e2
ProofPile 130M
ProofPile 370M
ProofPile 790M
ProofPile 1.48
ProofPile 2.8B
1e6
184
1e2
2
2
2
22
24
2
2
2
2
2
24
2
2
2
Length
Calibrate
Figure 7: Language Model Perplexity performance of Mamba models by calibrating scaling factors for either log (A) (red lines) A, (cyan lines). Perplexities reported across various datasets (GovReport, PG19, ProofPile, Pile) as well as model sizes.
42

---

C.3.2 Passkey Retrieval
Mamba2 Passkey Retrieval Performance
Baseline 130M
Baseline 370M
Baseline 780M
Baseline 1.3B
Baseline 2.7B
0%
0%
25%
25%
25%
25%
25%
50%
50%
50%
50%
50%
75%
75%
75%
75%
75%
100%
100%
100%
100%
100%
1K 2K
4K
8K 16K
1K 2K 4K 8K 16K
1K
2K 4K
8K 16K
1K 2K 4K 8K 16K
1K 2K 4K 8K 16K
Full 370M
Full
Full
130M
780M
1.38
2.7B
076
0%
0%
016
076
25%
25%
25%
25%
2.5%
50%
50%
30%
50%
50%
75%
75%
75%
75%
75%
100%
100%
100%
1.00
100%
1K 2K 4K 8K 16K
1K 2K 4K 8K 16K
1K 2K 4K 8K 16K
1KK 2K 4K 8K 16K
1K 2K 4K 8K 16K
Depth
Delta 130M
Delta 370M
Delta 780M
Delta 1.3B
Delta 2.7B
0%
0%
0%
0%
0%
25%
25%
25%
25%
25%
50%
50%
50%
50%
50%
75%
75%
75%
75%
75%
100%
100%
100%
100%
100%
1K 2K
16K
1K 4K BK 16K
2K 4K 8K 16K
1K 2K
16K
1K 4K BKK 1KK
A 130M
A 780M
A 1.38
A
370M
0%
0%
0%
25%
25%
25%
25%
25%
50%
50%
50%
50%
50%
100%
100%
2K
1K
4K
1K 2K
1K 8K 1KK
Context Length
Figure 8: Passkey Retrieval performance of Mamba2 models by calibrating scaling factors for either log (A) or A. Blue squares mean that the model was able to solve all examples of the given evaluation length/depth pair after tuning scaling factors, while red squares means that least one mistake was made, i.c. an incorrect passage was retrieved.
43

---

Mamba Passkey Retrieval Performance
Baseline 130M
Baseline 370M
Baseline 790M
Baseline 1.4B
Baseline 2.8B
0%
0%
0%
0%
0%
25%
25%
2.5%
25%
25%
50%
50%
50%
50%
50%
75%
75%
75%
75%
75%
100%
100%
100%
100%
100%
2K 4K BK16102K
1K
1K2K 6162K
2K
BK16192K
Full FT 130M
Full FT 370M
Full FT 790M
Full FT 1.4B
Full FT 2.8B
75%
75%
75%
100%
100%
100%
100%
100%
Depth
2K 4K BK16X02K
1K 4K BK1 6192K
1K 4K 6192K
2K 4K 8K1 6182K
1K 4K 8K16192K
Delta
Delta
Delta 790M
Delta
Delta
0%
0%
0%
0%
25%
25%
25%
25%
25%
50%
50%
50%
50%
50%
75%
79%
75%
75%
75%
100%
100%
100%
100%
100%
1K 4K 9K16182K
1K 2k 4K 8K1 6182K
1K 2k 4k G182K
1K 2K 4K 8K1 6/82X
1K 2K 4K 9K16122K
130M
370M
790M
1.4B
2.8B
0%
0%
0%
0%
0%
25%
25%
25%
25%
25%
50%
50%
50%
50%
50%
75%
75%
75%
75%
75%
100%
100%
100%
100%
100%
2K 4K 9K15102K
1K 8K1 6132K
1K 2K 4K 5I67K
2K 4K 8K1 6182K
1K 8K1
Context Length
Score
Figure 9: Passkey Retrieval performance of Mamba models by calibrating scaling factors for either log (A) or A. Blue squares mean that the model was able to solve all examples the given evaluation length/depth pair after tuning scaling factors, while red squares means that at least one mistake was made, i.c. an incorrect passage was retrieved.
44

---

C.3.3 LongBench
We evaluate the following tasks from LongBench (Table B). Due to our pre-training on an English dataset, we choose use only the English language tasks included the benchmark.
Table 4: Tasks from LongBench on which we evaluate.
Task
Context Type
Average Length
Metric
Data Samples
Q
P
Science
3619
F1 F1 Fl
200
H LES
Wikipedia
9151
200
2WIKIMULTIQA [54]]]
Wikipedia
4887
200
TREC [
Web Questions
5117
Accuracy
200
TRIVIAQA 770
Wikipedia/Web
8209
F1
200
LCC [
Github
1235
Edit Similarity
500
REPOBENCH-I [2
Github Repositories
4206
Edit Similarity
500
C.4 Pre-Trained Model Checkpoints Used
We use the official pre-trained model checkpoints of Mamba from the Hugging Face model Hub, found at https://hugglngfacc.cc co/
state-appace//maaaa-1100
state-spacee/nnabaa-370
state-spacee/mmaab--790
state-spacces/mamaa-1.44
state-ssaaces/mambb22888888
state-spaccee/mmmaba211000
state-spacces/aambba-77770
state-sspcces/bambb2288888
state-spacee/mmaaab211..8
state-spacee//aaaaa222.777
We use the original a implementations for adjusting scaling parameters.
45

---

D
Broader Impacts
This work explores novel method for length generalization of Mamba-based language models. While the direct usage of such models can entail potential broader risks within Al-based systems potentially trained scale, these risks do not stem directly from the methods and analysis pre- sented within the paper. As such, there are no risks that are deemed significant and worthy of further discussion.
E
Limitations
The primary limitation of our current work is that it is focused Mamba-style models; such, the methodology requires adaptation to similar models that utilize state- tratstrranitionn dynamics. However, many analogies exist between how information is written to the state and read out from the memory, presenting potential avenue for use in other such models.
Another potential limitation is the lack of instruction-tuned models that are available for direct use, limiting the set of experiments and evaluations on which can be adequately conducted.
46


Reviews:
[{"review_id": "xu58Dfc3iZ", "reviewer": "Reviewer_vqA3", "summary": "This paper identifies spectral instability in the state-space transition matrix A as the cause of weak length generalization of Mamba & Mamba-2.\nThe authors prove that extreme eigenvalues critically affect hidden-state divergence. To remedy this problem, this paper proposes uniformly rescaling A with calibrated values through backpropagation.\nThis scaling of A is training-free (using pretrained weights directly) without updating model weights (but it actually tunes A values through calibration).\nThis solution keeps reasonable perplexity upto 64 K–128 K tokens while boosting long-context accuracy on tasks such as LongBench and Passkey Retrieval.", "questions": "Does this calibration method actually perform better than fine-tuning only A or $\\Delta$t values (with other parameters fixed) directly to long context length?\n\nWhat happens if combining both $\\Delta$t and A scaling?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "The idea is simple, and the calibration requires very minimal training of multiplier values, which leads to good reproducibility. The results are competitive, maintaining good perplexity.\nHowever there are some weaknesses:\n- the contributions are limited : scaling a different parameter in the same training-free calibration framework has only subtle differences from MambaExtend.\n- the theoretical analysis assumes uniform, diagonal spectra, limiting real-world applicability.\n- it is not clear how section 4.2 directly leads to the motivation of controlling A only.\n- using one scalar per layer coarsely addresses spectral outliers. this leaves finer eigenvalue issues not fully resolved\n\n\nminor: figure 2 legend color should be corrected.", "was_revised": true, "final_justification": "I appreciate the authors' detailed response. After reading the reviews and the responses, I believe the experimental results added during the rebuttal make the paper more promising, thus I raise my score. As the authors mention in their response, the experimental results (e.g., scaling both dt and A) do not necessarily mean negative results, and they rather help readers understand the value of the work. So, I strongly recommend the authors to add discussions and results in the main paper.", "rebuttal": "Thank you for your valuable feedback. We greatly appreciate your recognition of **our framework's efficiency**, **good reproducibility** and **competitive results**. We nevertheless appreciate the additional points that have been mentioned as limiting factors and hope that the following response will to address these follows:\n\n---\n\n>C0: This scaling of A is training-free (using pre-trained weights directly) without updating model weights (but it actually tunes A values through calibration).\n\n**A0**: To clarify, our method introduces a lightweight set of scaling factors applied to the pre-trained model; the number of scaling factors is smaller than the number of A. While it is possible to tune the entire A, our method and experiments avoid doing so. The scaling set functions as a `plug-and-play` module that can be activated only when the input sequence length exceeds the model's original maximum context length during inference.\n\n---\n\n>C1: The contributions are limited : scaling a different parameter in the same training-free calibration framework has only subtle differences from MambaExtend.\n\n**A1**: We would like to highlight our contribution and distinguish our work from previous studies.\n\n**Novel Theoretical Analysis.** We are the first to provide a theoretical explanation for Mamba’s failure to generalize to long sequences, identifying the root cause as the divergent growth of the state norm due to spectral properties of the state transition matrix. This insight is not only novel but also foundational, as it enables a deeper understanding of the underlying dynamics rather than relying on heuristic modifications.\n\n**Strong Interpretability.** While we recognize the simplicity and practicality of the scaling methods in MambaExtend, our approach is theoretically motivated and analytically grounded. In particular, our analysis explains why certain empirical scaling factors (e.g., those greater than 1) observed in MambaExtend emerge despite appearing counterintuitive under their original motivation.\n\n---\n\n>C2: The theoretical analysis assumes uniform, diagonal spectra, limiting real-world applicability.\n\n**A2**: We could fully clarify your concerns regarding the uniform and diagonal assumption.\n\n**1. The premise that our theoretical analysis assumes diagonal spectra is a point of misunderstanding.** In fact, the diagonality is not a simplifying assumption—it is a structural characteristic of Mamba and Mamba-2. Their state transition matrices are inherently diagonal, which means the spectrum and the matrices of the form $\\text{diag}(\\exp(-A))$ naturally share identical entries. We don’t impose any additional assumptions on them.\n\n**2. Our choice to model the spectrum using a uniform distribution is empirically grounded.** As shown in Section 4.1, the eigenvalues of pretrained Mamba models consistently span the range from 0 to 1 across all layers. This observed behavior supports our decision to use a uniform distribution as a practical and realistic approximation.\n\n**3. The divergent convergence behavior is irrespective of spectral assumptions.** We appreciate your rigorous comment and accordingly extend our analysis to accommodate a more generalized case without imposing any strict assumptions on the distribution of $\\lambda$. Notably, the asymptotic behavior of the system remains unchanged. \n\n---\n\n**Elaboration on Theorem 4.2**: Expected State Norm under General Spectral Distribution\n\nAssume the transition matrix $\\mathbf{\\Lambda} = \\mathrm{diag}(\\lambda_1, \\dots, \\lambda_d)$ has diagonal entries $\\lambda_i \\in (0,1)$ drawn i.i.d. from a probability density function $p(\\lambda)$ supported on $[\\lambda_{\\min}, \\lambda_{\\max}] \\subset (0,1)$.\n\nSuppose the system evolves as: \n$\\mathbf{h_t} = \\mathbf{\\Lambda} \\mathbf{h}_{t-1} + \\mathbf{B} \\mathbf{x}_t,$\nwhere $\\mathbf{x}_t \\sim \\mathcal{N}(0, I)$ and each row of $\\mathbf{B}$ is independently sampled as $\\mathbf{b} \\sim \\mathcal{N}\\left(0, \\frac{1}{\\sqrt{d}} \\mathbf{I}\\right)$.\n\nThen, as $t \\to \\infty$, the expected squared norm of the hidden state converges to\n$\n\\boxed{\n\\mathbb{E}[\\|\\mathbf{h_\\infty}\\|^2] = \\mathbb{E}[\\|\\mathbf{B} \\mathbf{x}\\|^2] \\cdot \\int_{\\lambda_{\\min}}^{\\lambda_{\\max}} \\frac{p(\\lambda)}{1 - \\lambda^2} \\, d\\lambda.\n}\n$\n\n---\n\n**Asymptotic Analysis of $\\mathbb{E}[\\|\\mathbf{h}_\\infty\\|^2]$**\n\nDefine $\nI(\\lambda_{\\min}, \\lambda_{\\max}) := \\int_{\\lambda_{\\min}}^{\\lambda_{\\max}} \\frac{p(\\lambda)}{1 - \\lambda^2} \\, d\\lambda.\n$\n\nWe analyze the behavior of $I(\\lambda_{\\min}, \\lambda_{\\max})$ as $\\lambda_{\\max}$ approaches the edges of the domain.\n\n1. Behavior as $\\lambda_{\\max} \\to 1^{-}$:\n\n Near $\\lambda = 1$: , we have $ 1 - \\lambda^2 \\approx 2 (1 - \\lambda)$ via Taylor expansion.\n\n If $p(\\lambda)$ is continuous at $\\lambda=1$ and $p(1) > 0$, then for $\\lambda_{\\max}$ close to 1, $I(\\lambda_{\\min}, \\lambda_{\\max}) \\approx \\frac{p(1)}{2} \\int_{\\lambda_{\\min}}^{\\lambda_{\\max}} \\frac{1}{1 - \\lambda} \\, d\\lambda = \\frac{p(1)}{2} \\log\\left(\\frac{1 - \\lambda_{\\min}}{1 - \\lambda_{\\max}}\\right).$\n\n As $\\lambda_{\\max} \\to 1^{-}$, $\\log(1 - \\lambda_{\\max}) \\to -\\infty$, so\n$\\boxed{I(\\lambda_{\\min}, \\lambda_{\\max}) \\sim -\\frac{p(1)}{2} \\log(1 - \\lambda_{\\max}) \\to +\\infty.}$\n\nThe expected state norm diverges logarithmically as the largest eigenvalue approaches 1.\n\n 2. Behavior as $\\lambda_{\\max} \\to 0^{+}$\n\n When $\\lambda_{\\max}$ is close to 0,\n$\n\\frac{1}{1 - \\lambda^2} \\approx 1,\n$\nsince $\\lambda^2$ is very small.\n\n If $p(\\lambda)$ is continuous near zero with $p(0) > 0$, then\n$I(\\lambda_{\\min}, \\lambda_{\\max}) \\approx \\int_{\\lambda_{\\min}}^{\\lambda_{\\max}} p(\\lambda) \\, d\\lambda \\approx p(0) \\cdot \\lambda_{\\max}.$ Hence, $\\boxed{I(\\lambda_{\\min}, \\lambda_{\\max}) \\to 0 \\quad \\text{linearly as } \\lambda_{\\max} \\to 0^{+}}$.\n\nThe expected state norm vanishes linearly with $\\lambda_{\\max}$ approaching zero.\n`The expanded analysis still reveals the inherent tendencies of Mamba’s state dynamics toward explosion or vanishing, irrespective of spectral assumptions.`\n\n---\n\n> C3: It is not clear how section 4.2 directly leads to the motivation of controlling A only.\n\n**A3**: The eigenvalues of the transition matrix is $diag(\\exp(-A))$ , thus we can control the spectrum distribution by applying a scaling factor to control the magnitude of A accordingly.\n\n> C4: Using one scalar per layer coarsely addresses spectral outliers. \n\n**A4**: For fair comparison, we follow the default setting of MambaExtend for both language modeling and NIAH experiments. The experiments on the language modeling task are applied with per-layer-wise scaling. We could perform state-wise scaling accordingly; however, this will lead to more computation compared to MambaExtend methods, as it will introduce more trainable parameters.\n\nOn one note, the NIAH results do use state-wise scaling for each layer for MambaExtend and ours, with the end results similar to what we observe on other datasets. We contend that this serves as additional evidence to support the proposed method of scaling $A$ as opposed to $\\Delta$ is valid.\n\n> C5: Does this calibration method actually perform better than fine-tuning only A or t values (with other parameters fixed) directly to long context length? What happens if combining both t and A scaling?\n\n**A5**: We note that the goal of using zeroth order optimization method here is intended to make calibrating the model significantly simpler and faster, as it only requires forward passes and no explicit gradient. Scaling both $A$ and $\\Delta_t$ simultaneously can possibly lead to better results, but at the cost of greater complexity and potential instability as zeroth-order optimization often relies on tuning a small set of parameters.\n\nFurthermore, $\\Delta_t$ is input-dependent, not a model parameter, and therefore cannot be directly fine-tuned.\nWe manage to add fine-tuning $A$ directly and calibrate full $A$ for language modeling on PG19 with Mamba models. Enable more tuning freedom with additional scaling factors can improve the performance accordingly.\n| PPL | 2k | 4k | 8k | 16k | 32k | 64k |\n|-|-|-|-|-|-|-|\n| **mamba-1.4b** | | | | | | |\n| Base | 9.67   | 10.23  | 11.43  | 17.46  | 59.77   | 444.09   |\n| MambaExtend         | **4.69** | **3.89**  | **3.83**  | **3.55**  | 5.31   | 1648.00   |\n| Calibrated $A$ | 5.31 | 4.31 | 4.13 | 6.88 | 14.94 | 19.13 |\n| Finetuning $A$ (full)| 5.03 | 5.09 | 4.72 | 4.13 | 5.69 |  **6.45** |\n| Calibrated $A$ (full)| 5.28 | 4.38 | 4.42 | 4.63 | **5.16** |   6.48 |\n| **mamba2-1.3b** | | | | | | |\n| Base | 9.52   | 10.54  | 25.49  | 115.65 | 634.32  | 1479.45  |\n| MambaExtend | **4.34**  | 3.69  | 3.44  | 5.10   | 14.94   | 27.50 |\n| Calibrated $A$ | 4.38 | 3.78 | 3.44 | **3.28** | 4.03 | 4.72 |\n| Finetuning $A$ (full)  | 4.41 | 3.72 | **3.43** | 3.29 |  **3.23** |  **3.20** |\n| Calibrated $A$ (full)  | 4.38 | **3.68** | 3.52 | 3.44 |  **3.23** |  3.24 |\n\n\nThe following is a full fine-tuning of only $A$ for the NIAH task, which we observe does not significantly deviate from directly scaling $A$ but remains nearly exactly comparable to our method of simply adding scaling parameters. Here, `X` indicates the model correctly solved all examples.\n\n**Mamba-1.4B**\n\n|Depth/Length| 1K | 2K | 4K | 8K | 16K | 32K |\n|-|-|-|-|-|-|-|\n|0%| X | X | X | X | X | X | \n|25%| X | X | X | X | X | X |\n|50%| X | X | X | X | X | - |\n|75%| X | X | X | X | X | X |\n|100%| X | X | X | X | X | X |\n\nWe use the same evaluation criteria as described within the manuscript for all additional experiments.\n\n> Q: Figure 2 legend color should be corrected.\n\n**A**: Thank you for this note; the legend was intended to show that the changing $A$ scales were represented by different colors while the scaling of $\\Delta_t$ used different line styles. We can understand where the confusion came about and will fix this issue."}, {"review_id": "q2cLOkVbk7", "reviewer": "Reviewer_up1U", "summary": "This paper focuses on the sequence length extrapolation problem in Mamba-family models, identifies the spectrum of the transition matrix A as a key influencing factor, and proposes methods to adjust the spectrum of A to enhance the sequence extrapolation capability of Mamba-family models. The paper achieves promising experimental results.", "questions": "See the weaknesses mentioned above.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The problem addressed in this paper is important, and the proposed ideas are highly insightful.\n2. The method proposed in this paper is easy to understand and implement.\n3. The paper presents solid experimental results that offer valuable insights.\n\nWeaknesses:\n\n1. The paper argues that the instability (explosion or vanishing) of the state norm with increasing sequence length in Mamba-family models is the main cause of failure in length extrapolation. However, there is no direct experimental validation of the state norm behavior throughout the paper, which raises questions about the validity of the theoretical analysis.\n2. Prior works, such as DeciMamba, have also proposed explanations for the failure of Mamba models in handling length extrapolation. This paper does not sufficiently argue that its explanation is superior to those provided in prior work, nor does it point out any flaws or inaccuracies in the existing explanations. This weakens the persuasiveness of the theoretical analysis.", "was_revised": true, "final_justification": "The authors have provided detailed explanations addressing the two specific concerns I raised. I believe they have clarified these issues to a certain extent. However, since my initial rating was already positive and not negatively affected by these matters in the first place, I’m inclined to maintain my current positive rating. Furthermore, what prevents me from further elevating my rating is that the state norm validation experiments—which I consider quite significant (as failure to validate might risk undermining the entire work's interpretation) —were only supplemented after being prompted by reviewers. This affects my fundamental assessment of the paper's thoroughness and rigor.", "rebuttal": "We thank the reviewer for the time and effort they have placed towards providing a thorough evaluation of our work, particularly their comments about **the problem being important**, **the experiment being solid**, and **both the ideas and results being insightful**. We are also appreciative of the weaknesses that have been raised and hope the following additional details/results will clarify these points and resolve these existing doubts about work.\n\n>**C1**: There is no direct experimental validation of the state norm behavior throughout the paper, which raises questions about the validity of the theoretical analysis.\n\n**A1**: We can fully address this concern by conducting additional experiments that track the behavior of the `ssm_state` as the input sequence length increases.\n\nSpecifically, we randomly sampled data from ProofPile and PG19, and measured the maximum difference between the largest and smallest `ssm_state` norms across all layers of the Mamba2-1.3B model. The results, presented in the following table, clearly demonstrate a divergent trend in the `ssm_state`norm as sequence length increases. This empirical observation is consistent with our theoretical prediction and further validates our analysis.\n\n| ProofPile | 1k | 2k | 4k | 8k | 16k | 32k | 64k |\n|-|-|-|-|-|-|-|-|\n| State norm (max) | 131.4489 | 174.6176 | 892.7114 | 1330.5321 | 1141.5253 | 1130.6537 | 1157.5122 |\n| State norm (min) | 0.0046   | 0.0030   | 0.0007   | 0.0007    | 0.0004    | 0.0003    | 0.0005    |\n--------\n\n| PG19 | 1k | 2k | 4k | 8k | 16k | 32k | 64k |\n|-|-|-|-|-|-|-|-|\n| State norm (max) | 158.7656 | 160.8465 | 166.3788 | 876.0833 | 1241.3979 | 1201.5530 | 1242.7589 |\n| State norm (min) | 0.0033   | 0.0014   | 0.0050   | 0.0004   | 0.0005    | 0.0003    | 0.0002    |\n--------\n\n\nWe appreciate your suggestion and will incorporate this analysis into the final version of the paper, as we agree it provides a valuable supplement that reinforces the strength of our contributions.\n> **C2**: Prior works, such as DeciMamba, have also proposed explanations for the failure of Mamba models in handling length extrapolation. This paper does not sufficiently argue that its explanation is superior to those provided in prior work, nor does it point out any flaws or inaccuracies in the existing explanations. This weakens the persuasiveness of the theoretical analysis.\n\n**A2**: Thanks for your feedback. While prior works, including DeciMamba, LongMamba, and MambaExtend, provided useful explanations for Mamba’s failure in long-sequence generalization, we believe our theoretical analysis offers a more comprehensive and principled understanding.\n\nSpecifically, our contributions extend existing explanations in three key ways:\n\n- **New insight – divergent states due to large eigenvalues**: Our analysis reveals that large eigenvalues can lead to the tendency of exploding state norm, introducing instability that is not captured by existing explanations. This insight identifies an additional failure mode beyond vanishing memory and provides a more complete theoretical account of why Mamba models struggle with length extrapolation.\n- **Consistency with prior insights**: Our framework also recovers and formalizes the core idea shared by previous works: that small eigenvalues, when combined with accumulated discretization steps $\\sum\\Delta_{t}$, cause the decay term $\\exp(-A \\sum \\Delta_t)$ to vanish, resulting in fast memory loss. In this sense, our theory subsumes and strengthens existing intuitions about the vanishing states.\n- **Unified and explanatory framework**: Through Corollaries 4.3 and 4.4, we show how modifying the discretization step $\\Delta_{t}$ helps modulate convergence rates. However, we further clarify that such adjustments do not address the root cause for extremely long inputs, which lies in the spectral properties of the state transition matrix A. Our framework provides a principled explanation for why these heuristics sometimes succeed—and when they may fall short.\n\nIn DeciMamba, the claims are quite complementary to our analysis. For example, they also make the claim about the cumulative product of the per-timestep transitions converging to 0; they do not provide any additional intuition as to why this may be the case. Our analysis, meanwhile, delves deeper into this phenomenon. While we don’t provide a subjective quantitative measure as they do, our inspection of the eigenvalue spectrum is meant as a way of highlighting that our claims have grounding and serves to motivate the rest of our analysis in Section 4. We hope this clarification addresses your concern and strengthens the persuasiveness of our theoretical contribution."}, {"review_id": "b9DOvH6kCd", "reviewer": "Reviewer_ZsCU", "summary": "In Mamba styled models (Mamba and Mamba2), there are 2 parameters which have so far been controlled for to enable length generalisation. One a state transition matrix - $A$, the other being $\\Delta$, a parameter used for discretising the other main parameters in the SSM core equation. Prior work has focussed for controlling for $\\Delta$, this paper argues why controlling $A$ would make much more sense. There are some theoretical results that the paper contributes that point to the same and the authors also provide quite strong experimental results to validate the claim.", "questions": "**Questions**\n- Why does section 4.1 exist, what’s the point of that analysis ? This just lists the graph and says a generic statement, is there something I missed ? I currently think that not much will be lost if that section just never existed. \n- Although 6.1 is very convincing, just maybe add a remark about why there could still be cases like GovReport Mamba - 1.4B where perplexity of scaling Delta is better. Acknowledge it or just share intuition. \n- Based on the graphs, it seems like initially controlling for $\\Delta$ starts out as being better, and then controlling for $A$ becomes better. Is there any scope of controlling for both, or is that stupid ? \n- What do you think about the suggestion to rewrite the paper to be tighter / shorter to communicate the core idea quicker and more intuitively, or do you think this long paper format of NeurIPS makes most sense ? \n\n**Suggestions**\n- MambaExtend is by Azizi et al, that wasn’t clear from the main paper when it was mentioned, maybe add the citation in the section as well, instead of it just being in the Appendix.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- There is very strong empirical evidence and the experiments are all very comprehensive. \n- The core idea communicated in the paper is quite actionable and is thus quite relevant for LLM practitioners who use Mamba based foundation models. \n\n**Weaknesses**\n- The core idea communicated in the paper is quite simple, and I somewhere feel the writing of the paper currently has been optimised to fit the full 9 pages somehow. I think the paper could have been written in a much more succinct manner. \n\nI am a bit conflicted about the paper. While I think the paper is quite solid and makes a very specific important point very convincingly, I do think that the full 9 pages are not required to put this point across. Thus, probably there is scope to think about a more appropriate format / venue for the paper. Although I wouldn't be opposed to this paper being accepted if others as well as the authors disagree. Especially because I don't think there is anything else that can make the paper better. Thus, I would be willing to raise my score further, in case the authors engage and communicate why they think the current format and version of the paper is a good choice to communicate this idea.", "was_revised": true, "final_justification": "I think the paper makes a minor albeit important point quite convincingly with quite extensive experimentation. My only concern before the rebuttal was regarding the length of the manuscript and the choice of a venue like NeurIPS. But the new experiments the authors have posted during the rebuttal as well as the discussions they have promised to include in the main paper would constitute a good paper in my view, and therefore I have increased my score to 5.", "rebuttal": "We appreciate that the author acknowledges our work in terms of the evidence we provide for **our claims are convincing**, that **our paper is solid** and **the actionable nature of our ideas**. We also appreciate the various questions and suggestions they make, which we hope are resolved through the rebuttal that follows.\n\n>**C1**: I think the paper could have been written in a much more succinct manner. Do you think this long paper format of NeurIPS makes most sense?\n\n**A1**:  We understand the concern regarding the length and format, and we would like to take this opportunity to explain why we believe the current 9-page version is appropriate and necessary to fully communicate the significance of our contribution.\n\n**Shorter page limits may reduce persuasiveness.** There is a fine line between communicating an idea succinctly/effectively, while also providing sufficient justification through analysis and experiments. Thus, it is difficult to conclude regarding a shorter page limit, as we wish to maintain the same level of detail and persuasiveness. \n\n**A heavier appendix may reduce readability.** We already carefully offload proofs and more ancillary experimental and implementation details to appendix sections, allowing the main text to stay focused and readable. Condensing more main content could risk shifting essential arguments into the appendix, impacting overall readability and accessibility.\n\n**Additional analysis provided.** We appreciate your feedback and would like to add more analysis to enhance our work. We can provide the norm analysis of the `ssm_state` to enhance section 4.1 by conducting more experiments to record the trend of the `ssm_state` as the input sequence length increases as we replied to reviewer up1U. We randomly sampled data from ProofPile and PG19, and measured the maximum difference between the largest and smallest `ssm_state` norms across all layers of the Mamba2-1.3B model.  The results, presented in the following table, clearly demonstrate a divergent trend in the ssm_state norm as sequence length increases. This empirical observation is consistent with our theoretical prediction and further validates our analysis. \n\nWe feel the depth of analysis offered by the current format aligns well with the standards and expectations of the NeurIPS long paper. At the same time, we are also happy to discuss and willing to make changes based on this interaction period.\n\n| proof-pile | 1k | 2k | 4k | 8k | 16k | 32k | 64k |\n|-|-|-|-|-|-|-|-|\n| State norm (max) | 131.4489 | 174.6176 | 892.7114 | 1330.5321 | 1141.5253 | 1130.6537 | 1157.5122 |\n| State norm (min) | 0.0046   | 0.0030   | 0.0007   | 0.0007    | 0.0004    | 0.0003    | 0.0005    |\n--------\n| PG19 | 1k | 2k | 4k | 8k | 16k | 32k | 64k |\n|-|-|-|-|-|-|-|-|\n| State norm (max) | 158.7656 | 160.8465 | 166.3788 | 876.0833 | 1241.3979 | 1201.5530 | 1242.7589 |\n| State norm (min) | 0.0033   | 0.0014   | 0.0050   | 0.0004   | 0.0005    | 0.0003    | 0.0002    |\n--------\n\n> **C2**: Why does section 4.1 exist, what’s the point of that analysis?\n\n**A2**: Section 4.1 motivates the theoretical developments in Section 4.2 by examining the numerical structure of the Mamba transition matrix.\n\n**Section 4.1 provides essential empirical motivation for the theory in Section 4.2.** The heatmap in Figure 1 reveals a consistent spectral pattern across layers: the coexistence of both large (near 1) and small (near 0) eigenvalues. This observation is crucial, as it sets the foundation for the theoretical analysis that follows—namely, the identification of two distinct dynamical regimes that govern the behavior of Mamba states.\n\n**Without this empirical grounding, the theory would lack connection to real model behavior.**  As mentioned above, we will add the state norm analysis to further ground the following theoretical section 4.2. This will make the connection between empirical trends and theoretical insights even more concrete in the final version of the paper.\n\n> **C3**: Although 6.1 is very convincing, just maybe add a remark about why there could still be cases like GovReport Mamba-1.4B where the perplexity of scaling Delta is better. Acknowledge it or just share intuition.\n\n**A3**: One way of interpreting this behavior is that at shorter lengths (ex. <= 16k), simply being able to scale either factor is sufficient, as the differences in perplexity are marginal. This may be because at these lengths, the compounded decay may approximate rather well; however, as the lengths keep increasing, this eventually leads to a significant difference between them, which may be the case of why at shorter lengths both methods appear to work rather well.\n\nOn one note, we can observe that on Mamba (first generation) models, scaling $\\Delta$ usually works slightly better on shorter lengths; we hypothesize that this might be due to how the $A$ matrix is constructed as a diagonal matrix rather than a scalar-times-identity like in Mamba2, which may make scaling $A$ with a single layer-specific scaling factor more appropriate in that scenario.\n\nOne specific outlier is on Mamba-1.4b with GovReport, at a sequence length of 32k; scaling $\\Delta$ for this example works significantly better than $A$ but it is the only case. We believe that this is more of a point of acknowledgement given the lack of other settings where this occurs (either with 1.4B models, first-generation Mamba models or just with other models on GovReport) and will mention this accordingly.\n\n> **C4**: It seems like initially controlling for $\\Delta$ starts out as being better, and then controlling for $A$ becomes better. Is there any scope of controlling for both, or is that stupid?\n\n**A4**: There are manners in which this is possible; for example, if one is to introduce scaling parameters for each factor and train them in the same manner as we do, then by all accounts it is possible to control them simultaneously.\n\n**Characteristics of zeroth-order optimization**: One reason why this isn’t done explicitly is the use of zeroth-order optimization, which is known to become more unstable as the number of parameters that need to be tuned increases. Alternatively, one way would be to directly fine-tune everything through gradient descent, but this is a much more time- and resource-consuming process compared to the proposed design.\n\n**Multi-stage optimization**: Another method would be to conduct a type of multi-stage optimization, where perhaps one subset of scaling factors is tuned, followed by another. However, there are some known potential issues with such types of methods in deep learning research that make this somewhat difficult.\n\n> **C5**: MambaExtend is by Azizi et al, that wasn’t clear from the main paper when it was mentioned, maybe add the citation in the section as well, instead of it just being in the Appendix.\n\n**A5**: We thank the reviewer for raising this detail; we did mention Azizi et al. at multiple times within the preceding sections but did not explicitly mention MambaExtend; we will make this very clear by mentioning them together."}, {"review_id": "0lLSvoDKLK", "reviewer": "Reviewer_z8Ky", "summary": "This paper explores the length generalization capabilities of Mamba. The paper provides theoretical arguments tying the spectrum of the transition matrix to the convergence behavior of the hidden state when applied to infinitely long sequences. The paper demonstrates that scaling the spectrum of the transition matrix directly is favorable compared to previous works that scale the discretization step.\nSupporting experiments are provided.", "questions": "- The sentence in lines 30-33 asserts that the primary motivation for mamba is length generalization capabilities. Can you please provide grounds for this claim? To the best of my knowledge Mamba shows superior performance compared to previous SSMs on long sequences (not necessarily in the extrapolation setup).\n\n- What is $s$ in line 151? it isn't defined before it is used.\n\n- Figure 1 isn't clear from its description - Specifically, this is the spectrum per layer of a specific model. What does it mean that certain layers have fast decay while others decay slowly?\n\n- Why is Lemma 4.1. state as a formal lemma? this is trivial and does not require a dedicated proof.\n\n- In Theorem 4.2, Corollary 4.3 & Corollary 4.4 - why is the assumption on the eigenvalue distribution different per statement? In general the statements are not very clear when written verbally, I would suggest writing the statement formally (in lines 192-193 and 196-197).", "limitations": "The empirical benefit of scaling $\\mathbf{A}$ instead of $\\mathbf{\\Delta}$ isn't clear. Results seem to be marginally better in certain setups and worse in others. The authors fail to explicitly discuss this in the paper. It would be good to discuss in a dedicated section the tradeoffs and benefits of the different scaling methods, as it stands the method presented seems to offer incremental novelty compared to MambaExtend.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n* The paper discusses a very relevant and vibrant topic of length generalization.\n* The paper is written well and structured in an intuitive flow.\n* The presented method for scaling the eigenvalues (and improving length generalization) is simple and clear.\n\n**Weaknesses**:\n* Results in the paper are trivial and the significance seems very limited.\n* The paper does not discuss known results on other architectures or distinguishes the results and the novelty brought forth in the paper.\n* Related work is very lacking - missing works on extrapolation and length generalization in different setups. I would expect a paper discussing length generalization of Mamba to discuss:\n   - Works related to extrapolation in transformers:\n      - Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. Press et al. ICLR 2022.\n      - Unveiling transformers with lego: a synthetic reasoning task. Zhang et al. Arxiv 2022.\n      - etc.\n   - Works discussing length generalization and extrapolation in RNNs:\n      - Implicit Bias of Linear RNNs. Emami et al. ICML 2021.\n      - On the implicit bias of gradient descent for temporal extrapolation. Cohen-Karlik et al. AISTATS 2022.\n      - Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Network. Cohen-Karlik et al. ICLR 2023.\n      - etc.", "was_revised": true, "final_justification": "The authors have conducted additional experiments. After reviewing other reviewers’ comments, I am persuaded that the paper’s significance is greater than I initially assessed.", "rebuttal": "First, we thank the reviewer for acknowledging the **relevant and vibrant topic** of our work, that it is **written well and structured** and the presented method is **simple and clear**. We also acknowledge the various points of concern and questions they have raised, which we hope will be resolved with the following response.\n\n-----\n\n> W1. Results in the paper are trivial and the significance seems very limited.\n\n**A1**: We understand these concerns, but highlight that our method offers **significant/consistent improvements** over the original pre-trained models.\n\n| PPL (**mamba-1.4b**)| 2k | 4k | 8k | 16k | 32k | 64k |\n|-|-|-|-|-|-|-|\n| Base | 9.67   | 10.23  | 11.43  | 17.46  | 59.77   | 444.09   |\n| Ours | **5.31** | **4.31** | **4.13** | **6.88** | **14.94** | **19.13** |\n\n| PPL (**mamba2-1.3b**)| 2k | 4k | 8k | 16k | 32k | 64k |\n|-|-|-|-|-|-|-|\n| Base | 9.52   | 10.54  | 25.49  | 115.65 | 634.32  | 1479.45  |\n| Ours | **4.38** | **3.78** | **3.44** | **3.28** | **4.03** | **4.72** |\n\nWe posit that generalization to longer inputs remains a challenge for Mamba models. Our proposed scaling method demonstrates improvements under such conditions, offering evidence to support our theorems. This insight is also novel, both explaining why existing methods fall short in extrapolating and providing a principled criterion for designing more resilient architectures.\n\n> W2. The paper does not discuss known results on other architectures or distinguishes the results and the novelty brought forth in the paper.\n\n**A2**: Mamba models present a promising alternative to the well-established Transformer architecture. However, despite their rising popularity, they remain comparatively under-explored, particularly when it comes to understanding their limitations.\n\nWe address this gap by uncovering the fundamental causes behind Mamba’s failure to generalize across sequence lengths, which previously has not been explained. Our analysis exposes key structural differences from self-attention mechanisms that contribute to this behaviour, providing valuable insight that deepens the field’s understanding and paves the way for more robust model design.\n\n> W3. Related work is very lacking - missing works on extrapolation and length generalization in different setups.\n\n**A**: We will add and discuss the mentioned references regarding length generalization in transformers and RNNs in the final version of our paper as follows.\n\nThe literature on transformers highlights positional encoding and attention biases as critical for length extrapolation:\nPress et al. introduced ALiBi (Attention with Linear Biases), demonstrating that linear distance-based attention biases enable effective extrapolation beyond training lengths. Zhang et al. used synthetic reasoning tasks to analyze transformers' generalization capabilities, revealing how attention patterns and pretraining affect robustness to longer sequences.\nEmami et al. analyzed the implicit bias of linear RNNs in sequence modeling tasks. Cohen-Karlik et al. investigated gradient descent dynamics for temporal extrapolation and showed how overparameterized RNNs learn low-dimensional state spaces.\n\nWhile Mamba's efficiency is well-established, its extrapolation behavior requires systematic comparison to transformer approaches and integration of insights from RNN theoretical analyses.\n\n\n1. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. Press et al. ICLR 2022. \n2. Unveiling transformers with lego: a synthetic reasoning task. Zhang et al. Arxiv 2022. \n3. Implicit Bias of Linear RNNs. Emami et al. ICML 2021.\n4. On the implicit bias of gradient descent for temporal extrapolation. Cohen-Karlik et al. AISTATS 2022.\n5. Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets. Cohen-Karlik et al. ICLR 2023.\n\n> Q1. Can you please provide grounds for [sentence in lines 30-33]?\n\n**A**: We acknowledge some potential confusion that may have stemmed from a poor choice of phrasing.\n\nThe primary motivation of Mamba is with regards to the ability to have models address the computational inefficiency of models on long sequences, which is shown by results in the original paper. However, Mamba does claim extrapolation abilities, namely on synthetic tasks such as selective copying and induction heads, but the same experiments have not been tested on language data, which is why we believe this claim prompts further investigation.\n\n> Q2. What is s in line 151? it isn't defined before it is used.\n\n**A**: ‘s’ simply appears after its definition:  ‘scalar values’, refers to the scaling factors used by Azizi et al. to multiply with $\\Delta_t$.\n\n> Q3. Figure 1 isn't clear from its description. What does it mean that certain layers have fast decay while others decay slowly?\n\n**A**: Figure 1 visualizes the spectrum of each layer to motivate our theoretical analysis. It reveals a consistent pattern across layers: the presence of both large (near 1) and small (near 0) eigenvalues. This is not incidental; it underpins Section 4.2, where the convergence behaviour is shown to depend critically on the spectral distribution. If the largest eigenvalues of different states in a layer are concentrated close to zero, then the information contained in the states of this layer decays fast. Conversely, layers with slower spectral decay (i.e., eigenvalues closer to 1) preserve state information for longer durations.\n\n> Q4. Lemma 4.1. is trivial and does not require a dedicated proof.\n\n**A**: This Lemma ensures that we do not need to impose any explicit assumption on the norm of the input. By presenting it as a lemma, we establish a clear and self-contained result that supports downstream arguments without relying on unstated constraints.\n\n> Q5. In Theorem 4.2, Corollary 4.3 & Corollary 4.4 - why is the assumption on the eigenvalue distribution different per statement?\n\n**A**: The assumptions in Corollary 4.3 and Corollary 4.4 are specific instantiations of the more general condition stated in Theorem 4.2. We provided detailed derivations in Appendix A.3, showing how each corollary follows from the general theorem under simplified eigenvalue distributions.\n\nIn particular, the convergence behaviour for the Mamba state is obtained by taking the limit $\\lambda_{\\min}⁡ \\to 0$. For Mamba2, we consider the regime where $\\lambda_{\\max} \\to \\lambda_{\\min}$​. These are chosen to reflect the parameterizations used in each model. We will clarify this by presenting the assumptions in formal mathematical notation and revise the manuscript accordingly.\n\n> Limitations: The empirical benefit of scaling instead of isn't clear. [...] It would be good to discuss [...] the tradeoffs and benefits of the different scaling methods.\n\n**A**: In future revisions, we will compare the tradeoffs of different scaling strategies and clarify their benefits, especially in comparison to MambaExtend. Corollaries 4.3/4.4 suggest that both eigenvalue adjustment and discretization step scaling can influence the convergence rate. While performance can vary depending on the model and sequence length, our results indicate that the scaling method becomes increasingly important as sequence length grows. We posit that scaling $A$ can be a dominant factor for extremely long sequences, where numerical stability and convergence speed become critical. \n\n> Paper Formatting Concerns: The paper has multiple flaws in the form of unclear sentences and other formatting issues, below are a few:\n\n**A**: Thank you for the detailed feedback, we will fix these typos in the final version."}]

Author rebuttal:

Meta-review: This paper proposes a method for improving the length generalization of Mamba models by scaling the spectrum of transition matrices. Reviewers found the approach simple, well-motivated, and supported by strong experiments and theoretical analysis. While some viewed the contribution as incremental relative to existing calibration methods, the rebuttal provided additional clarifications and results that addressed most concerns. Overall, the contribution is clear and well-supported, and the consensus is that the paper is suitable for acceptance.
Decision: Accept (poster)