From Collapse to Control: Understanding and Extending Context Length in Emerging Hybrid Models via Universal Position Interpolation
Authors: Haochen Shen
Davis Wertheimer
Zheng Wang
Garrett Goon
Derrick Liu
Naigang Wang
Mudhakar Srivatsa
Raghu K. Ganti
Minjia Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Hybrid Models
Mamba
Transformer
Long Context
Primary area: foundation or frontier models, including LLMs

Abstract: Hybrid Mamba-Transformer models have emerged as promising alternatives to pure Transformers, offering efficiency and competitive performance. However, they struggle to generalize beyond their training context windows, collapsing on long-context tasks. We provide the first systematic analysis of this failure, showing that it arises from uncontrolled state growth and uneven receptive field contributions across the hybrid architecture. Guided by this understanding, we introduce Universal Position Interpolation (UPI), a closed-form, training-free scaling method that unifies Mamba's cumulative decay with Transformer rotary frequency scaling. UPI selectively stabilizes unstable Mamba dynamics while rescaling Transformer encodings, controlling state growth and enabling reliable long-context generalization, with only a few auxiliary forward passes.  Evaluation shows that UPI extends multiple state-of-the-art hybrid and pure Mamba models from 4K to up to 64K tokens on PG-19 perplexity, LongBench and RULER benchmarks, without sacrificing short-context accuracy. These findings establish the first principled bridge between Transformers and state-space models and open a new direction for training-free context extension methods for emerging hybrid models.

Published as a conference paper at ICLR 2026
FROM COLLAPSE TO CONTROL: UNDERSTANDING AND EXTENDING CONTEXT LENGTH IN EMERGING HYBRID MODELS VIA UNIVERSAL POSITION INTERPOLATION
Haochen Shen', Davis Wertheimer", Zheng Wang', Garrett Goon', Derrick Liu", Naigang Wang", Mudhakar Srivatsa Raghu K. Ganti Minjia Zhang' "SSAIL Lab, University of Illinois Obbaa-Chaappiiee 'IBM Research, USA
(hshen14, 4, zhengw10, minjiaa e Billineis.. .ddu (davis. wertheimer, goon, d1 iu uu) @ eibm.cco com (nwang, msrivats, rganti @ss ibm. com
ABSTRACT
Hybrid Mamba-Transfone Transformer models have emerged as promising alternatives to Transformers, offering efficiency and competitive performance. However, they struggle to generalize beyond their training context windows, collapsing on long- context tasks. We provide the first systematic analysis of this failure, showing that it arises from uncontrolled state growth and uneven receptive field contributions across the hybrid architecture. Guided by this understanding, we introduce Univer- sal Position Interpolation (UPI, lightweight. training-free scaling method that unifies Mamba's 's cumulative decay with Transformer rotary frequency scaling. UPI selectively stabilizes unstable Mamba dynamics while rescaling Transformer en- codings, controlling state growth and enabling reliable long-context generalization, with only few auxiliary forward passes. Evaluation shows that UPI extends multi- ple state-of-the-arl hybrid and pure Mamba models from 4K to up to 64K tokens PG-19 perplexity, LongBench and RULER benchmarks, without sacrificing short-context accuracy. These findings establish the first principled bridge between context length extension on Transformers and state-space models and open a new direction for training-free context extension methods for emerging hybrid models.
INTRODUCTION
Large language models (LLMs) increasingly rely on long-context reasoning for applications such as document understanding (Liu et al. 2024a Hsieh et al.] 2024b Bai et al.) (024). retrieval (Mo- htashami Jaggi 2023 Kamradt 2023] Hsieh et al.) 2024a), and multi-turn dialogue (Li al. 2017; et al. (2024). However, scaling to long contexts remains a fundamental challenge. Canonical transformer models capture long-range dependencies through attention, but incur quadratic computation and memory complexity, limiting their scalability very long sequence lengths.
Hybrid Mamba- Transformer models have gained ground as a promising alternative Lenz et al. 2025 Ren et al.) 2024: Zuo et al. (2025.. Interleaved transformer and selective state-space models (SSMs) (Gu & Dao 2023) Dao & (0244 achieve strong performance at a fraction of computational cost. Models like Nemotron-H (NVIDIA al. 2025b), Jet-Nemotron (Gu et al. 2025). ) Nemotron- Nano-2 (NVIDIA et al. 2025a), and Bamba (Chu et al.] 2024: Ganti et al.) demonstrate strong performance against public Transformer LLMs (e.g.. LLaMA-3. 1) at substantially lower inference cost, making hybrid models a compelling design choice for next-generation long-sequence modeling.
While hybrid models deliver promising performance, their ability to generalize beyond the training context length remains largely unexplored. Our preliminary results suggest that hybrid models collapse when inputs exceed the training window. Bamba-9B-v2 (Ganti et al.), for example. is state-of-the-ar hybrid model with comparable performance to Llama-3.1 1..1-881 alongside 2-2.5x faster speed Yet as shown in Fig. I perplexity spikes when context length surpasses the 4K Kokeo training length. Making hybrid models practical therefore requires solving the context length extension (CLE) problem: enabling models trained on shorter windows to generalize to longer contexts.

---

Published as a conference paper at ICLR 2026
CLE methods exist for Transformers (e.g. g.. Pl (Chen jet al.; 2023), YaRN
25
100
(Peng et al.] (024). Lon-
gRoPE (Ding et al.
2024 Shang al. 2025
Perplexity
20
80
Accuracy (%)
13
60
and for Mamba (e.g.. DeciMamba (Ben-Kish et al.
2025), , StuffedMaaaa Chen et al. 2025). Mambalx-
10
40
lend (Azizi et al. (0255) ). but to our knowledge none have been tested on hybrid models. LongMamba (Ye et al. 2025), the sole exception, only tests on a 1.2B scale model and provides no analysis into its failure. Thus, it remains unclear why hybrid CLE fails, or how to solve A second challenge lies in scalability. Many CLE methods rely on fine-grained token-wise control BBen-Kish et al.) 2025] Ye et al. 2025) or hours-long, model-specific, search-based rescaling of positional embeddings (Chen et al. 2023 LocalL.
Bamba v2
20
Bembee
2048
4096
8192
16384
32768
Context Length (token)
Figure 1:
Performance of state-of-the-ard
hybrid model on long- congecontext tasks. Blue lines denote perplexity()) on PG-19 valida- tion set; red lines show retrieval accuracy(t) on
Keyge- task
from RULER (Hsich et al., 2024a). Model: Bamba v2 (Ganti et al.), with 4k training con- text. YaRN applied without tuning.
[aMA|20233 b] Ding et al.]]] Shang et al. 20225)
and often require fine-tuning on long sequences. This
particularly costly, involving massively inflated activations, memory overhead and time. Meanwhile, training- fareing-free&ae CLE remains largely unexplored for hybrid models.
Motivated by these challenges, we introduce Universal Position Interpolation (UPI), a elosed-form CLE technique for scaling hybrid Mamba-Trane Trans---ran models without retraining or inference overhead. We conduct the first systematic analysis of hybrid model scaling behavior, identifying uneven receptive field contributions and uncontrolled state growth as key factors behind long-context failure. Via selective rescaling, UPI prevents state explosion and restores balanced contributions. Most importantly, UPI requires no additional training, hyperparameter tuning, or modification of highly-fused kernels. By unifying Transformer position encoding with Mamba gate dynamics, UPI establishes the first principled, efficient, training- free CLE method tailored to hybrid models.
We evaluate UPI on PG-19, LongBench, and RULER benchmarks using sfate-of-the-rt Mamba and hybrid models such as Mamba2, Bamba and Nemotron-H. UPI extends usable context length from 4K up to 64K tokens, delivering substantially lower perplexity and higher accuracy. Relative to trchitecture-speiffi CLE baselines, UPI achieves comparable superior performance without training. Moreover, UPI requires substantially less effort to identify unstable heads than prior methods (e.g. Mambaa and adds inference-time overhead or model modifications.
2 BACKGROUND AND RELATED WORK
The Emergence of Hy brid Mamba-Transformee Models. Improving efficiency and scalability has been central focus of LLM research. Beyond optimizing Transformer attention with approxima- Kitaev et al.) 2020; Yuan et al. 2025) or high-performance kernels (Dao) et al. 2022 Dao 2024), alternative architectures such as RWKV (Bo 2021; [un et al., 2023), and
tions (Wang et al., 2020
S4 (Fu et al. 2023 Gu et al. 2022) achieve linear-time sequence modeling with reduced memory footprint and compute cost, but often struggle to match Transformers in accuracy when scaled to
large tasks. More recently. Mamba (Gu & Dao 2023 Dao & Gu 2024) introduces a GPU-friendly linear recurrence that enables efficient modeling ling of long sequences while offering competitive accu- racy (Wang et al. 2024. Shams et al.] 2024: Zhu et al. 20244 Liu et al.) 2024b) Schiff et al.) [024]] Pióro et al. 2024). At the tt token X, I. Mamba's s state update recurrence can be simplified as: h, = T. h, 1 + B,x, yt = C.h,, Forget gates ā. := exp(-oA.)
(0, 1) and input gates B, = A,B, E Raxd are discretized using Zero-Order Hold with a token-wiss step size of A, E R.. A, governs update magnitude, by modulating the input and forget gates. Output gate C, E R projects the updated hidden state to the output y: gd. This linear recurrence is computed via parallel scan (c.g.. Hillis & Stecle (1986)) on modern GPUs, requiring only O(N) compute and Ollg N) parallel depth over sequence length N. More detail can be found in Appendix B
Although Mamba offer both a structured approximation to attention and highly parallel recurrence,
pure Mamba models still lag behind Transformers in accuracy across a range of tasks. This perfor-
mance gap has motivated hybrid architectures, such as Jamba (Lenz et al., 2025 2024). Nemotron-
H NVIIDI al. 2025b), Bamba (Chu et al. 2024), Nemotron (Gu al. 2025). with compu-
2

---

Published as a conference paper at ICLR 2026
tationally efficient Mamba blocks interleaved with expressive Transformer layers. These hybrids achieve competitive performance within their training context (e.g., 4K tokens), but their performance beyond training length remains largely untested. Several recent works further explore architectural modifications to extend the context length of such hybrid models, often requiring retraining from scratch. For example, Hybma (Dong et al. 2025) mixes SSM and attention at the head level, while Samba (Ren et al., 2024) replaces full attention with sliding- window attention. Because their long- context behavior primarily determined by architectural design and training-time context windows, they are not directly comparable to post-hoc context extension methods that operate purely at inference time. In contrast, UPI designed extend the context length of existing hybrid Mamba-Transee Transformer models without modifying model architecture or requiring additional training. UPI operates solely at inference time and is therefore directly applicable to off-the-shelf hybrid models.
Context Length Extension. We refer to the problem of generalizing models trained on short contexts, to longer sequences as context length extension (CLE). In Transformer models, position encoding plays a central role in CLE. Early approaches use additive positional bias (e.g.. T5 (Raffel et al.
2019),
ALiBi (Press et al.] (022)), while more recent LLMs (Touvron et al., 2023 Al 20244] Bai
et al. 2023).
2023 Qwen et al.] 2024) adopt multiplicative positional embeddings such as RoPE (Su et al. CLE in Transformers has been widely studied through RoPE modifications, including position
interpolation (PI) (Chen al. 2023). NTK-aware scaling [LocalLLaMM]] 2023a by, YaRN (Peng) et al.) 2024), and LongRoPE: (Ding et al. 2024 [Shang et al. 2025). These methods rescale positional embeddings across RoPE dimensions to enable interpolation or extrapolation of position information.
CLE methods also exist for Mamba models, such as DeciMamba (Ben-Kish et al. 2025), Long- Mamba (Ye al. 2025), StuffedMamba (Chen et al.) 2025). and MambaExtend (Azizi et al. 2025). These approaches often rely on discrete mechanisms such as token filtering or time step clamping, introducing inference- time overhead or requiring costly tuning on long sequences. Prior work has also focused exclusively on pure Transformer or Mamba architectures. To our knowledge, training free CLE for hybrid models remains largely unexplored. This gap motivates our work: identifying hybrid model failure modes and proposing lightweight, closed-form scaling method to address them.
UNDERSTANDING CONTEXT LIMITATIONS IN HYBRID MODELS
Hybrid Hamba-Transfore models perform strongly within their training contexts, but degrade sharply beyond them. As shown in Fig. of-te-of-the-ar hybrid models consistently struggle when evaluated on context lengths unseen during training. Bamba v2 and Nemotron- H (base) experience substantial performance degradation (solid lines): perplexity explodes while retrieval accuracy drops precipitously. common assumption, seen in works like Hymba (Dong et al. 2025) and Jamba (Lenz) et al. 2025). is that Transformer and Mamba layers play complementary roles: attention handles precise retrieval, while SSM summarizes local context. Under this view, hybrid model CLE seems as simple scaling RoPE in the Transformer layers. However, the dashed lines in Fig. reveal this approach falls short. Applying YaRN yields mild gains, but fails to prevent performance collapse. This motivates deeper look into why hybrid models fail on long contexts.
3.1 EFFECTIVE RECEPTIVE FIELD (ERF) ANALYSIS
We start by extending effective receptive field (ERF) analysis (Ben-Kish al. 2025 Ye al. 2025) to hybrid models, covering both Mamba and Transformer layers. ERF quantifies the influence of each input token on a given output position, illustrating how information propagates across sequence length. A detailed description of ERF is given in Appendix C We find that within the training context, Mamba layers with the highest average ERF perform comparably to Transformer layers. However, as sequence length increases, ERF of Mamba layers saturates, whereas Transformer layers do not, revealing a weakness in Mamba's ability to scale. Fig. (left) shows highly non-uniform ERF patterns: while the largest values by far come from the transformer layers (L9/18/27), some Mamba layers attend several hundred tokens away in expectation. Averaging across heads also obscures significant variation between them. A fine-grained head- and layer-wise ERF analysis also reveals substantial variability between individual layers and heads. Fig. (right) shows that Mamba heads highly heterogeneous, with the highest-ERR ERF heads rivaling their transformer counterparts, despite sharing the same update recurrence with heads that saturate quickly. What accounts for this disparity? In & 3.2. we analyze and connect this behavior to hybrid model failure modes on long inputs.
3

---

Published as a conference paper at ICLR 2026
8 (Mamba2)
Layer 21 (Mamba2)
12000
Langth
Layer Type
2000
60000
2k
MambeZ
Transformer
1500
1000
8000
20k
20000
ERF
60k
Layer 9 (Transformer)
Layer 18 (Transformer)
4000
40000
2000
20000
20000
0
10
155 20
25
30
Layer Index
Sequence Length
Sequence
Figure 2: Left: ERF distribution across layers of Bamba-9B-v2. For each layer, the ERF is averaged across heads. Yellow highlighted layers (L9, L18, L27) are Transformer layers. Right: ERF as function of context length for all heads in layer. Each curve represents ERF change w.r.t. sequence length in one head. Transformer heads depict consistent near-linear ERF growth, while most Mamba heads have saturated ERF once sequence length exceeds training context. A few Mamba heads continue to expand ERF comparably to Transformer heads.
3.2
STATE MAGNITUDE EXPLOSION IN MAMBA LAYERS
The ERF analysis in 3.1 reveals that Transformer receptive fields expand reliably with input length, while only some Mamba heads do the same. This imbalance stems from the dynamics of Mamba states themselves: certain heads accumulate state aggressively, leading to uncontrolled growth at longer contexts. To understand this phenomenon, we analyze the Mamba's recurrent update rule.
As introduced in section the update rule for Mamba layer state can be simplified to: h activated. Similarly,
h. + B, Note that V' 0 S < 1, as āe, the forget gate, is B, and x, are mostly positive, since X, is Silu-activated, and B, is the product of softplus- and Silu-activated values. For simplicity, let B, and x₁ be always positive. we assume that ae B, x are roughly constant over time ((T,, B,, xx) (a, B, x)), it can be easily shown that the sequence bo... forms successive terms of geometric series:
hp Bx, h a(ho)
Bx aBx Bx,
Bx
h; Σ
B
Σ
Bx
1-a
All values are positive and a 1, this series converges asymptotically in the limit finite upper bound Bx/(1- α). Thus the Frobenius norm of matrix- matrix-valued state h would also converge to finite upper bound, since every matrix element converges to its own positive supremum individually. In practice, these terms not constant consistently positive, but still observe this predicted behavior in Mamba layers, as shown in Fig. 3 State magnitude varies noisily due to Mamba's dynamic gating, but it grows and saturates alongside ERF within each head.
This pattern becomes problematic when the forget gate is consistently close to 1. Returning to our simplified model, the asymptotic upper bound for scalar h is given by Bx/ a). When ā = 1, the series converges extremely slowly: upper bound Bx/ (1 a.) is much larger than the initial (and greatest) summand Bx, and the individual summands Bx converge to zero only for very large One easily envision Mamba head with forget gate consistently close to 1 that its state magnitude takes more time to approach its supremum than is available in the training sequence length. And this indeed occurs in practice. Heads in Fig, with low state magnitude reach their upper bounds quickly, then hover as sequence length extends beyond the training length. Meanwhile, a high-ERF head grows far beyond any scale seen during training, it has yet converge its supremum.
Such out-of-distee antoo--distibution growth produces feature collapse in Mamba layers. The output of the SSM operation is given by yt C,h, + Dx,, where yt, C,, x,, are all vector-valued. Here, h, represents the state matrix for every head concatenated in the horizontal dimension. Thus when head
(n) has state h.")) with very large magnitude, the first term C,h overwhelms the second. Likewise,
4

---

Published as a conference paper at ICLR 2026
Layer 14
Layer 2.1
Layer 1.6
Layer 31
60000
1.5000
23000
2000
1.0000
5000
20000
10000
3000
1000
20k 40k
State Magnitude [h.]
2000
400
State Megnituree
300
2000
2000
200
300
2000
1000
0
20k
100k
2k 20k 40k: 6ok D6k 100k
2k. 20k 40k 6ük Bok 100k
2k
20k 4dk G08 Bok 106K
Sequence Length
Sequence Length
Sequence Length
Sequence Length
Bamba- 9b-v2
mamba2-2 - 7bb
Layer 14, head 26 - Layer 21, head 6
Layer 16, head 26 - Layer 31, 6
Layer 14, head
Layer 21, head 27
Layer 16, head
Layer 31, head 27
Figure 3: Frobenius norm of hidden state over sequence length for Bamba (Left) and Mamba2 (Right), using a sample from PG-19 validation set. Each color represents a different head. ERF and state magnitude grow together, with high-ERF heads overwhelming saturated heads long inputs.
the submatrix product C, (n) dominates the concatenated vectors coming from the other heads. The next step in the Mamba layer a GroupNorm across heads, which scales the large vector terms back into distribution, but in doing suppresses all terms coming from other heads inputs. This leads to sparse activations, and the vast majority of layer weights contributing to dead neurons. We claim this feature collapse that is responsible for Mamba and hybrid model failure long contexts.
In order to scale Mamba and hybrid model inputs, we must then control the state magnitude of unconverged heads. One simple approach would be reduce further the contribution of individual tokens in such heads: for an input n times longer than the training length, we would want each token to shift the state matrix 1/n times much it otherwise would have. This idea forms the backbone of our approach, which we formalize in the next section.
4 UNIVERSAL POSITION INTERPOLATION
4.1 MOTIVATION AND INTUITION
The analysis in $ 3 reveals that collapse arises from unstable state magnitudes in a subset of Mamba heads. These heads increment their states far beyond the training distribution, suppressing contribu- tions from other heads. A natural question is how to prevent this extra drift. Our intuition is simple: when sequence length is scaled up by factor n. the hidden state contribution of each token should be scaled down by 1/n. Otherwise, the cumulative state grows disproportionately, producing the observed runaway heads. Phrased differently, the magnitude of unconverged head states can be kept in-distribution by "slowing down" the state's 's rate of change proportional to the increased input length. This view motivates Universal Position Interpolation (UPI), a training-free, closed-form method that rescales token contributions maintain stable state dynamics sequence length grows.
UPI is "universal" two senses. First, it applics to both Mamba and Transformer components of hybrid models, ensuring consistent treatment of positional dynamics across architectures. Second, requires no retraining, additional parameters, or structural changes: purely a closed- foore rescaling that can be inserted at inference time. This makes UPI especially well-suited for hybrid models, where only a subset of heads require correction.
4.2 CLOSED- FORM DERIVATION
We now formalize the intuition behind UPI. We start with Mamba's original state update rule: h = exp (-aAt) h + A,
Let the state update rule on the expanded sequence with scaling factor n take the following form: h k+1 = [a, A,, h),4 + g(,, A., n)x,, VO < k < n,
5

---

Published as a conference paper at ICLR 2026
where f(a, Ae, n) and g(B,, Ae, n) are adjusted forget and input gates. Unrolling the recurrence, and setting the gates equal to the ones from the original update rule, yields adjusted formulas: (Forget Consistency)
exp (-a.) f(a, n)
A, = g( A. n)
Σ f(a, D). k=0
f(a, A,,
= [B, n)
(Input Consisteney)
Here we are enforcing two consistency conditions: (i) forget consistency: after n small steps, the total forgetting should equal one big step; and (ii) input consistency: input contributions accumulated over n small steps should equal one big step. Solving these produces the following gate functions,
{ f(a, ((a,,,,1) A,, = exp -.)
cxp g (B,,
A,B, 1 1-exp( 1-exp(- =..))
For the forget gate, this amounts to taking the nth root, scaling it closer to 1, while the input gate receives an adjusted fraction between 1 and 1//n based on the forget gate, theory, for an input times longer than the original, the above formula scales the forget- and input-gates of unconverged heads, bringing state magnitudes back into distribution. Unfortunately, this does not actually work in practice. While the provided adjustments scale down state magnitude growth by in expectation, find that data-dependent input gate scaling behaves too erratically when A, varies aggressively in practice. Thankfully, this can be solved by simple tweak to the input gate scale factor.
Recall that these adjustments are applied specifically to Mamba heads with forget gate close to 1 in expectation. From the Mamba update rule, the forget gate approaches 1 as A, approaches O. While the data-dependent input gate scaling factor is undefined for A. O, we can still take the limit:
lim 1 exp A
A. exp (-ad,)
The input gate scaling factor can thus be replaced by constant 1/n. This reduces further to the original Mamba update rule. with A, divided by n. Thus for an input n times longer than the training length. we can simply divide A, by for unconverged heads and attain good performance, shown in $ 5.1
4.3 STATE SPACE DUALITY AND POSITIONAL ENCODING
So far we have motivated UPI empirically: it stabilizes unstable heads and restores balanced context scaling. But is this simply a heuristic, or does it connect to deeper principles of sequence modeling? To answer this, we revisit the structured state space duality between Mamba and Transformers (Dao & Gu 2024), which posits component-wis correspondence across the two architectures: Transformer's query, key, and value correspond to Mamba's output gate C., input gate B, and input token x respectively. Similar to multi-head attention, input and output matrices B, and C, are shared across heads, while forget gate and step size use head-specific parameters a and A.
Building on this foundation, we make a further observation: Mamba's compounding forget gates and Transformer's 's RoPE encodings both act as mechanisms for injecting position into the model. From this lens, UPI can be viewed as a principled extension of position interpolation: both determine how representations advance with token position, and gate scaling in Mamba is mathematically analogous to frequency scaling in RoPE (Su et al. 2023). RoPE, interpolation methods (Chen et al. 2023) such as YaRN rescale the lowest frequencies to enable longer contexts; in Mamba, UPI similarly rescales small A,. This conceptual bridge explains why UPI succeeds on hybrid models, and why can be seen the first training-free free CLE tailored to them.
We provide a proof sketch of this equivalence in Appendix E Both RoPE and Mamba maintain a bank of keys, and multiply key for input at time n. by exp Σi= A.). with A, varying per-channel(hea)) and position. In ROPE, A is a static complex constant (w.r.t. f); in Mamba, A is real-valued and input-dependent. Both can be manipulated analogously: YaRN scales small A, (low frequencies) by 1/s, skips large ones, and interpolates between. UPI is the state-space analogue, scaling small Mamba A by 1/s and leaving the rest untouched. Consistent initializations and similar long-context failure modes further support this RoPF- duality, with further details in Appendix and

---

Published as a conference paper at ICLR 2026
These intriguing links provide avenues for further exploration, but we leave this for future work and simply indicate the duality as additional justification for UPI. In this sense, our work establishes a principled bridge between Transformer position encoding and Mamba gate dynamics.
4.4 APPLYING UPI ACROSS HYBRID MODELS
Having established both the intuition and theoretical grounding of UPI, the last outstanding question is how to identify unconverged heads in model. Unlike RoPE, Mamba A values are dynamically generated, not static, and thus unknown a priori. While it is possible to analyze state growth per head in detail, and define thresholds for when state magnitude has approached its supremum sufficiently closely within the training sequence length, we find it easier to re-use existing code calculating ERE, and select the top-K% % of highest-ERF heads for adjustment" As shown in section 3.2 heads with large ERF tend to have large final state magnitude, given that both are determined by the average forget gate value. ERF profiles are highly stable, shown in $5.2 and only few parallel forward passes on a small calibration dataset are required to reliably identify the top- K% heads by ERF.
The final UPI procedure is lightweight and general. Broadly, UPI works by identifying components prone to long-context instability, and applying the appropriate closed-form scaling rule from $ 4.2
UPI Pipeline
Step 1:
Choose existing RoPE frequency adjustment technique for Transformer layers
Step 2:
Take few forward passes to identify unconverged Mamba heads 1% by ERF)
Step 3:
Adjust Transformer layer RoPE frequencies, and selected Mamba heads, to match the length of each given input
The result is robust, tuning-free approach context length extension that incurs no additional inference time overhead, minimal architectural intervention, and little extra computation to perform model adjustment (a few forward passes to find the unconverged heads). In the following section, we validate and analyze these claims via multiple long-context benchmarks and experiments.
0.25
0.20
0.15-
Density
0.15
0.10
0.10
0.05
0.00
0.00
0.00
log(ERF)
log(ERF)
log(ERF)
Bamba-9B-a
Nemotroo -B-8
8k
mamba2-2 7b,
Top 10%
Top 30%
Data
Top 20% -- Top 40
Mixture PDF
Figure 4: Distribution of Log-ERF across Mamba heads different models at their training context length: Bamba-9B-v2 at 4k, Nemotron-H-SB at 8k, and Mamba2-2.7B 2k. The distributions of all three models show bimodal characteristic, motivating our head-wise selective mechanism.
To justify our head-wise selective mechanism, we examine the distribution of the profiled ERF on each heads for all three models, Bamba-9B-v2, Nemotron-H-SSh and Mamba2-2.78 7B, presented in Fig. 4 The top-K% % cutoff is motivated by the empirical distribution of log-ERF scores across heads. In all evaluated models, the log-ERF distribution is bimodal: minority subset of high-ERF heads (approximately 10- ~ 30%, depending on layer and sequence length) separates cleanly from the majority of low-ERF heads. Exact percentages vary by model, but we adopt 20% as a general and effective cutoff because it closely aligns with the average split point of the bimodal log-ERF distribution and further supported by the empirical results Table
Altemattvee metries for head selection discussed in Appendix

---

Published as a conference paper at ICLR 2026
Table Perplexity on PG-19 at different sequence lengths across different hybrid models.
Method
Buse
8.35
8.78
9.37
14.23
32.76
127.90
Bamba-9B-v2
Mambaa YaRN
8.42
8.51
8.83
13.02
23.68
53.14
UP[(ours) + YaRN
8.13
8,30
9.01
9.82
14,60
18.59
Basc
7.53
7.82
7.59
46.57
210.30
530.31
Nemotron-EH
Mombaa
7.82
7.71
7.13
23.24
46.42
78.51
UPl(ours)
7.46
7:91
7.39
15.58
25.72
44,01
16.73
478.21
Mamba22-277
LongMamba
7.31
7.94
9.16
14.82
23.59
42.20
UP[(our))
7.40
8.21
8.75
13.69
17.58
22.24
EXPERIMENTS
Models and Benchmarks. (mamba2-2.. 7B (Dao & Gu 2024)) and two hybrid models (Bamba- Bambb-9B--ve 9B-v2 (Ganti et al.) and Nemot ron- ron-H-8B. 1a--8---Be---- 8K (NVIDIA al.) (025b)). The Mamba2 model is trained with 2k context length, Bamba with 4k, and Nemotron- with 8k. For language modeling capability, we evaluate perplexity on the validation set PG-19 (Rae al. 2019).
Our main experiments evaluate UPI Mamba2 model
ceatiee--ta--expeeteeee sequences
are concatenated until they exceed the target length, and overflow is discarded. LongBench-E (Bai al. 2024) and RULER (Hsieh et al. 2024a) are used to evaluate long-range understanding.
Baselines. Since no existing tuning-free CLE methods have been developed for hybrid models, we compare against component- -ise CLE baselines, applying separate strategies Mamba and Transformer layers. For Transformer layers with RoPE, we adopt YaRN (Peng et al. (024). tuning- free position interpolation method. For Mamba layers, we compare our proposed UPI method against LongMamba (Ye et al. 2025). strong CLE baseline that, while tuning-free in architecture, requires parameter search for optimal performance. Hyperparameters for LongMamba are optimized over several rounds of LongBench-E, which take roughly 10 hours on 8 A100 GPUs for each model. UPI calibrated by running forward passes on samples from the ArXiv subset of proof-pile (Computer) 2023), which takes under 3 minutes. The cost during inference is negligible for both methods. Further implementation details are provided in Appendix A
5.1
MAIN RESULTS: EVALUATING HYBRID MODELS ON LONG-CONTEXT TASKS
PG-19 Perplexity, As shown in Table I UPI excels at extending hybrid models to long contexts without fine-tuning, It outperforms not only base models that degrade rapidly, but also the strong LongMamba + YaRN baseline, which still struggles to maintain low perplexity at extended lengths. The gains are substantial: at 64k on Bamba, UPI reaches perplexity of 18.59 an 85.5% relative reduction vs. the base model; the relative reductions on Nemotron-H and Mamba-2 are 91.7% and 95.3%, respectively. This result reflects UPI's theoretical grounding: state behavior is kept in-distribution as promised, even when inputs are much longer than expected.
RULER. We further evaluate UPI on the RULER benchmark for long-context retrieval and reasoning tasks across various domains, As shown in Table 2 UPI + YaRN consistently outperforms both base hybrid models and the LongMamba YaRN baseline for all extended context lengths. The method excels at retrieval (S1-S3, M1 MM,, MV, MQ), especially on tasks like S2 where it outperforms competitors by a substantial margin. While is not immune the expected performance degradation on complex reasoning tasks (VT, CWE, FWE, QAI, QA2), UPI YaRN mitigates this drop effectively. Moreover, UPI also demonstrates similar superior performance when applied to the pure Mamba2 model. proving its robustness over both hybrid and pure Mamba architectures.
We also report results on LongBench-E in Table which targets long-range dependency, UPI + YaRN consistently surpasses both baselines in both hybrid models on single/multi-docunen QA and summarization, and overall, with a small dip on few-shot tasks. We note that while UPI trails LongMamba when applied to Mamba2, this is likely feature of LongMamba rather than UPL LongMamba is optimized end-to- end-to-end directly for this task family, while UPI represents a general closed-form solution. In hybrid models, there is no LongMamba equivalent for transformer layers,
LongBench.
8

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of performance (%) on RULER with extended context lengths.
Model
Len
Method
S1
52
53
M1
M2
M3
MV
MQ
VI
CWE
FWE
QAT
QA2
Avg.
Banbu- 9B-v2
LongMamba YaRN
25.6
19.6
67.0
63.0
38.6
22.0
25.6
16.8
16.6
1.6
27.5
27.6
11.2
7.1
0.8
1.2
8.7
24.0
18.8
17.6
26.0
UPlrours) + YaRN
55.8
68.0
49.2
27.0
15.4
0.6
31.9
31.8
4.7
3.3
26.4
30.8
22.8
28.3
Base
43.6
16.2
24.8
11.6
0.0
0.0
8.0
3.9
1.3
9.7
0.9
16.4
20
12.0
16k
LongMamba YaKN
72.8
13.2
0.2
0.0
13.0
11.4
UPl(ours) YaRN
60.6
40.6
34.8
21.8
0.6
0.0
19.1
15.0
5.0
6.2
0.3
22.6
20.8
19.0
Rase
54.8
63.1
31.8
38.8
45.1
12.0
54.8
33.5
78.2
57.4
42.1
4.2
13.4
40.7
Nemotoo-H
16k
Mangaan
67.3
30.0
84.4
24.9
40
UPl(ours)
65.8
67.2
36.9
24.8
52.0
4.3
47.9
43.8
82.0
88.5
46.5
15.7
8.7
45.0
Base
26.3
17.2
11.1
16.1
4,9
0.0
44.1
8.7
63.9
43.2
18.6
0.0
17.8
20.9
32k
LongMamba
34.7
12.0
3.8
8.5
01.4
0.0
45.3
11.8
59.0
40.5
8.8
0.9
14.3
18.5
UPl(ours)
61.4
46.6
Mambs 2277
Base
100.0
0.0
0.0
0.0
0.6
0.4
0.0
48.7
3.9
66.6
37.6
19.0
21.3
4k
Mangh
100.0
3.4
1.8
0..2
0.0
0.0
1.7
1.3
4.7
9.0
31.2
36.6
27.5
16.7
UPl(ours)
37.2
17.7
0.2
Manga UPl(ours)
100.0
2.3
1.6
4.3
0.0
0.0
0.8
0.6
2.3
3.5
1.3
8.8
13.0
10.7
100.0
5.8
2.2
5.2
0.0
0.0
2.0
1.3
31.4
0.5
11.6
8.0
9.4
13.6
so Mamba and transformer scaling cannot be optimized jointly, and the benefits of Long evaporate. UPI, meanwhile, is consistent in boosting performance for long contexts.
Table 3: LongBench-E results on different category of tasks.
Model
Method
SQA
MQA
Summary
Few-shot
Synthetic
Code
Avg.
Rase
11.68
10.09
18.69
49.19
6,64
38.69
21.71
Bamba 9B-v2
LongMamba + YaRN
12.16
8.43
17.49
56.57
2.53
46.85
22.85
UPl YaRN
12.20
13.25
20.94
52.71
5.85
41.68
23.68
14.87
19.74
Nemotron-H-S0
LongMamha
16.47
12.29
17.54
$1.92
7.65
37.03
23.14
UP[(ours)
16.81
14.48
21.86
48.8
8.47
46.85
25.15
Base
8.79
3.95
9.04
10.78
2.32
22.89
9.72
Mambu2-2.78
LongMamba
15.77
UPl(our))
13.60
6.47
13.23
15.58
8.50
29.21
14.52
5.2 ANALYSIS RESULTS
How important is scaling both Mamba and Transformer layers? We perform an ablation study of UPI on PG-19 perplexity by selectively disabling UPI in either the Transformer or Mamba layers of Bamba-v2. As shown in Table removing either part degrades performance. Notably, disabling UPI at Mamba layers produces higher perplexity than disabling YaRN, which aligns with the ERF analysis in 3 showing that Mamba layers the bottleneck for context scaling of hybrid models. Moreover, we interpolate all Mamba heads, performance likewise drops dramatically, echoing our findings that particular unconverged heads are the issue.
Table 4: Ablation study UPI components.
Table 5: Ablation study the Top-K setting.
Method
8K
16K
32K
64K
Top-K
8K
16K
32K
64k
UPI YaRN
9.82
14.60
18.59
10%
24.21
33.69
w/o YaRN 9.23
12.54
16.87
25.79
20%
9.01
9.82
14.60
18.59
UPI
8.97
14.85
27.29
98.01
30%
12.18
21.09
20.14
27.02
w/o Selective head
9.47
15.73
23.85
65.92
40%
16.24
22.76
22.53
34.14
Is it always beneficial to scale more heads? We perform further ablations on PG-19 perplexity by testing different filtering thresholds for unconverged heads in Bamba-v2. As shown in Table setting the threshold 20% yields the best performance across context lengths. Conducting interpolation on more than 30% of heads causes performance to degrade noticeably, indicating that aggressive interpolation disrupts short-term heads, harming ability to model local dependencies.
Is ERF-based head selection susceptible to dataset change? We compare the top-K head selections calculated from the validation split of three calibration datasets: proof-pile ArXiv subset (Computer) 2023) (D1), PG-19 (Rae et al. (019) (D2), and wikitext2-vl (Merity et al. 2016) (D3). As shown in Table 6 the overlap increases the top-K percentage decreases, indicating that the heads with

---

Published as a conference paper at ICLR 2026
the highest ERF are consistent across datasets. For our top-20% filter, the overlap between datasets consistently larger than 80%, indicating fairly robust selections. Thus, using a reasonably small calibration dataset (such as the proof-pile ArXiv subset, with only 100 samples) we rank the ERF of the heads with confidence.
UPI achieves fast calibration process. As decribed in sec- tion [4.4] for per-model parameter selection, our pipeline (i) applies YaRN to Transformer layers with RoPE, (ii) profiles ERF for Mamba heads, and (iii) applies a constant scaling to the top 20% heads by ERF. Stages (i) and (ii) add negligi- ble overhead. For stage (ii), we calibrate per-head ERF on 100 16k-token sequences that completes <3 minutes on 8 A100 GPUs. By contrast, LongMamba tunes hyperparame- ters via multiple LongBench runs, taking 10.5 hours on the same hardware. This gap underscores the lightweightness of our approach. At inference, both methods apply element-wise operations the A array, yielding extra overhead. See Appendix for details.
Table 6: Ablation study on head selection overlap percentage (%)
across different datasets.
Top-K
D2-D3
10%
91.8
95.2
90.4
20%
30%
75.2
80.9
68.1
67.3
Scalability and Generalizability To demonstrate the generalizability of our method to larger models and diverse tasks, evaluate Nemotron-H-44 H-47B ----as--Sk on the Dialogue History QA and Code Repo QA tasks from LongBench v2. The results are shown in Table 7 Our method consistently improves performance retrieval, dialogue understanding, and code repository understanding for the 47B model, indicating strong generalization.
Passkey Retrieval
Dialogue History QA
Code Repo QA
Model
Method
8k
16k
32k
Nemotron-He
Baseline
100.0
9.6
0.0
28.00 33.28
31.58
-BB--Base-8k
UPI
100.0
16.3
8.5
34.15
Table Evaluation of Nemotron---4775 Base-BB- on Passkey Retrieval from RULER, and Dialogue History QA, and code Repo QA from LongBench v2.
Complementarity with Fine- Tee--TTnnnnn-aaan Extrapolation To illustrate complementarity of UPI and extrapolation methods, we fine-tuned Bamba-9B-v2 to 32k and 128k context lengths. As shown in Table applying UPI to the 32k model improves long-context performance beyond 32k, closing part of the gap to the 128k fine-tuned checkpoint. Additionally, UPI also helps improve the 128k checkpoint long contexts. This demonstrates that UPI can still provide benefits even when fine-tuning- based extrapolation available.
Model
Method
4k
8k
16k
32k
64k
128k
Base
84.74
78.05
70.62
62.10
35.49
18.04
Bamba-9B-v2 32k
UPI + YaRN
85.57
79.70
71.56
63.79
40.90
28.42
Base
83.93
77.30
74.12
69.35
64.36
61.57
Bamba-9B-v2 128k
UPI+ YaRN
81.85
76.57
75.62
71.43
66.91
62.17
Table 8: Evaluation of Mambu2-2.7B on PG-19 perplexity, comparing the effectiveness of Deci- Mamba and UPI. DeciMamba's token token-prunin pruning scheme shows better performance on short context (2k to 8k), while UPT's state growth stabilization improve performance on even longer context.
6 CONCLUSION
We present the first in-depth analysis of hybrid Tranba-Tranffomerr model scaling, showing that long-context failures arise from a small subset of unstable heads. By viewing this through the lens of position encoding and structured state space duality, we introduce Universal Position Interpolation (UPI), a closed- -oosed-form and tuning-free method for context scaling. UPI extends hybrid models far beyond their training context lengths while preserving short-context accuracy, establishing the first principled bridge between Transformer encodings and Mamba gate dynamics. We hope this work inspires further exploration of training-free CLE methods for hybrid models.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We sincerely appreciate the anonymous reviewers. Their insightful feedback helps significantly improve the quality of the paper. This research was supported by the National Science Foundation (NSF) under Grant No. 2441601. The work utilized the Delta and DeltaAl system at the National Center for Supercomputing Applications (NCSA) and Jetstream2 at Indiana University through allocation CIS240055 from the Advanced Cyherinfrastruucure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. The Delta advanced computing resource collaborative effort between the University of Illinois Urbana- Ahbana-Chammaign and NCSA, supported by the NSF (award OAC 2005572) and the State of Illinois. UIUC SSAIL Lab supported by research funding and gift from IBM, Google, Amazon, and AMD.
REFERENCES
Meta AL. Introducing Meta LLaMA-3. https://ai, http://aaamaaaa..eaeemm mottaccom/bloo/mmttaall amma 2024.
amma- -//
Seyedarmin Azizi, Souvik Kundu, Mohammad Erfan Sadeghi, and Massoud Pedram. Mambaextend: A training-free approach to improve long context extension of mamba. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Forum? firbmmidd-ggzooo R
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint 23Xiv:2309.1660 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1119-3137 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18 2024.acl-long. 172. URL htt/hhtts///aae http///caaaatooooooooooo waanhhooogy..oo wtt.gyorrg/22000 www.:244..111o1oo11111..
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025. URL /ettps://openrviiiie
net.
Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL
htw.lpperviiieeeettteeeeeeee net
PENG Bo. Blinkd//rwky-Im: 0.01, August URL. 5196577
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. CoRR, abs/2306.15595, 2023.
Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Stuffed mamba: State collapse and state capacity of RNN-based long-context modeling, 2025. URL
httts:://penneet hwt.:/ee//oooee otw.

---

Published as a conference paper at ICLR 2026
Linsong Chu, Divya Kumari, Tri Dao, Albert Gu, Raghu Ganti, Dakshi Agrawal, Mudhakar Srivatsa, Davis Wertheimer, Yu Chin Fabian Lim, Antoni Viros, Nelson Gonzalez. Tuan HoangTrong, Ofir Arviv, Yotam Perlitz. Michal Shmueli, Haochen Shen, Minjia Zhang, Gabe Goodhart, Naigang Wang, Nick Hill, Joshua Rosenkranz, Chi-Chun Liu, Adnan Hoque, Chih- Chieh Yang. Sukriti Sharma, Anh Uong, Jay Gala, Syed Zawad, and Ryan Gordon. Bamba: Inference-efficient hybrid mamba2 model. https: https:///tttggggtggt/eeeeeee co/blg/// bamba inference- efficient hybrid bybrid-mamba2 model $ 9008FF998888 Decem- ber 2024. Accessed: 2025-05-06.
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, 2023.
URL https: http://wwww....ooteeeeee/eeeteea
Data
Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning, In International Conference Learning Representations (ICI.R) 2024.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory--fficient exact attention with IO-awareness. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens, 2024.
Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, ZIJI CHEN, Ameya Sumil Mahabaleshwarkat Shih- Yang Liu, Matthijs Van keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Celine Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head architecture for small language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https
//openreview.oo net aieenrevveee.net//ommmm.......
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry Hippos: Towards language modeling with state space models. In International Conference on Learning Representations, 2023.
Raghu Ganti, Lingsong Chu, Divya Kumari, Mudhakar Srivatsa, Antoni Viros, Davis Wertheimer, Naigang Wang. Nick Hill, Yu Chin Fabian Lim, Thomas Parnell, Joshua Rosenkranz, Chi- Chun (Charlie) Liu, and Garrett Goon. Bamba-9B-v2 Fast and powerful! https t/huggingfaace cw/blog/////oocccccccccc. ibm ai wwwppaoooo//eeeeeeeeeaeapp bamrraaaaabbbaaaa 9b-v2 Accessed: 10-May- 2025.
Albert Gu and Tri Dao. Mamba: Linear- time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00052, 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference Learning Representations (ICLR), 2022.
Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, and Han Cai. Jet- nemotron: Efficient language model with post neural architecture search, 2025.
W. Daniel Hillis and Guy L. Steele. Data parallel algorithms. Commun. December 1986. ISSN 0001-0782, doi: 10. 1411147992..993. URL htt.tttst///.o.ooooo http://///o.oo////////// org/10
1145/7903 7903.
Cheng-Pine Hsieh, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint 4441::2404.06554 2024a.
12

---

Published as a conference paper at ICLR 2026
Cheng- Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen- Yu Lee, Ranjay Krishna, and Tomas Pfister. Found in the middle: Calibrating positional attention bias improves long context utilization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACI. 2024, pp. 14982-14995 Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10. 10.18653/v/2224 http:///22222..nnaaaaaaa.a.030 findings-acl.890, URL
httpss:/acllantboooooo.orr///// Findings wwlgggss-cl..... acl. 8907
Gregory Kamradt. Needle in a haystack - com/ gwwmraattLLWWestttt.eeee htw.:/eeest_.eeeeeeeeellliaaaaa
pressure testing Ims,, 2023. URL htL.httss///ithhu
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In
International Conference on Learning Representations, 2020. URR.tttps://ppereviii
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))227--2324, 1998. doi: 10.1109/5.726991.
Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M. Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zusman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Opher Lieber, Or Dagan, Orit Cohavi, Raz Alon, Ro'i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shaked Meirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Ychoshua Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, and Yoav Shoham. Jamba-1.5; Hybrid transformer- mamba models at scale. CoRR, abs/2408. 12570, 2024. URLbttps / org/ 0 055008550///0 axivv. 2408 12570
Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M. Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz. Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zusman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Or Dagan, Orit Cohavi, Raz Alon, Ro'i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shai Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Josh Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, and Yoav Shoham. Jamba: Hybrid transformer-mante language models. In The Thirteenth International Conference on Learning Representations, 2025. URL httpps://oge http::///peneevieww net
Shaked Haim Meirom, Tal Delbari, Tal Ness, Tomer
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Greg Kondrak and Taro Watanabe (eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 986-995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL httt:s:://aclanthooog.oo org/ Ittrg//I171000 1099/
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. Transactions of the Association for Computational Linguistics, 12:1571173, 2024a.
doi: 10. 1621ta//taclaa a_aeclla__0333 URL
https: pettps://aclantaae 1 logy. org/ 202 tacttallll.. tacl-11.97
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprini arXiv:2401.10116 10166, 2024b.
Loca Dynamically scaled rope further increases performance of long context LLaMA with reddit com/:/LocalLLa/// comments / 4mr
zero fine-tuning, httpps://wwt http:///wwwwww.d.tcooo
Etynamically scaled ralerroureeeeeeee wttesrrinneeeereeeeeeeeee 2023a. Reddit post, LL
LaMA; accessed 2025-05-13.
13

---

Published as a conference paper at ICLR 2026
LocaILLaMA. atk-aware scaled rope allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degration. https: cowments/ 7/www. reddit. wttp://////////aaateeta rope_ a aal_alllooeeo lows_ 11
j5/ttkwwrte scaled
s_tto_avet 2023b. Reddit post, r/ accessed
2025-05-13.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard S. Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4898 4906, 2016.
Adyasha Maharana, Dong-Ho Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. Lun-Wat Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 13851 13851-1387 113870, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/211//////222.alllllll URL
org/ 2024 wttp:////22....1.....eo.......
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
NVIDIA, :. Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Ren- duchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, Aleksander Ficek. Alex Kondratenko, Alex Shaposhnikov, Alexander Bukharin, Ali Taghibakhshi, Amelia Barton, Ameya Sunil Mahabaleshwarkar, Amy Shen, Andrew Tao, Ann Guan, Anna Shors, Anubhav Mandarwal, Arham Mehta, Arun Venkatesan, Ashton Sharabiani, Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Buddharaju, Banghua Zhu, Barnaby Simkin, Bilal Kartal, Bita Darvish Rouhani, Bobby Chen, Boris Ginsburg, Brandon Norick, Brian Yu, Bryan Catanzaro. Charles Wang, Charlie Truong. Chetan Mungekar, Chintan Patel. Chris Alexiuk, Christian Munley, Christopher Parisien, Dan Su. Daniel Afrimi. Daniel Korzekwa, Daniel Rohrer, Daria Gitman, David Mosal- lanezhad, Deepak Narayanan, Dima Rekesh, Dina Yared, Dmytro Pykhtar, Dong Ahn, Duncan Riach, Fileen Long, Elliont Ning, Eric Chung, Erick Galinkin, Evelina Bakhturina, Gargi Prasad, Gerald Shen, Haifeng Qian, Haim Elisha, Harsh Sharma, Hayley Ross, Helen Ngo. Herman Sahota, Hexin Wang. Hoo Chang Shin, Hua Huang, Iain Cunningham, Igor Gitman, Ivan Moshkov, Jachun Jung, Jan Kautz, Jane Polak Scowcroft, Jared Casper, Jian Zhang, Jiaqi Zeng, Jimmy Zhang. Jinze Xue, Jocelyn Huang, Joey Conway, John Kamalu, Jonathan Cohen, Joseph Jennings, Julien Veron Vialard, Junkeun Yi, Jupinder Parmar, Kari Briski, Katherine Cheung. Katherine Luna, Keith Wyss, Keshav Santhanam, Kezhi Kong, Krzysztof Pawelec, Kumar Anik, Kunlun Li, Kushan Ahma- dian, Lawrence McAfee, Laya Sleiman, Leon Derczynski, Luis Vega, Maer Rodrigues de Melo, Makesh Narsimhan Sreedhar, Marcin Chochowski, Mark Cai, Markus Klicgl, Marta Stepniewska- Dziubinska, Matvei Novikov, Mehrzad Samadi, Meredith Price, Meriem Boubdir, Michael Boone, Michael Evans, Michal Bien, Michal Zawalski, Miguel Martinez, Mike Chrzanowski, Mohammad Shoeybi, Mostofa Patwary, Namit Dhameja, Nave Assaf, Negar Habibi, Nidhi Bhatia, Nikki Pope, Nima Tajbakhsh, Nirmal Kumar Juluru, Oleg Rybakov, Oleksii Hrinchuk, Oleksii Kuchaiev, Oluwatobi Olabiyi, Pablo Ribalta, Padmavathy Subramanian, Parth Chadha, Pavlo Molchanov, Peter Dykas, Peter Jin, Piotr Bialecki, Piotr Jamuszewski, Pradeep Thalasta, Prashant Gaikwad, Prasoon Varshney, Pritam Gundecha, Przemek Tredak, Rabeeh Karimi Mahabadi, Rajen Patel, Ran EI- Eanyy, Ranjit Rajan, Ria Cheruvu, Rima Shahbazyan, Ritika Borkar, Ritu Gala, Roger Waleffe, Ruoxi Zhang, Russell Hewett, Ryan Prenger, Sahil Jain, Samuel Kriman, Sanjeev Satheesh, Saori Kaji, Sarah Yurick, Saurav Muralidharan, Sean Narenthiran, Seonmyeong Bak, Sepehr Sameni, Seungju Han, Shanmugam Ramasamy, Shaona Ghosh, Sharath Turuvekere Sreenivas, Shelby Thomas, Shizhe Diao, Shreya Gopal, Shrimai Prabhumoye, Shubham Toshniwal. Shuoyang Ding, Siddharth Singh, Siddhartha Jain, Somshubra Majumdar, Soumye Singhal, Stefania Alborghetti, Syeda Nahida Akter, Terry Kong, Tim Moon, Tomasz Hliwiak, Tomer Asida, Tony Wang, Tugrul Konuk, Twinkle Vashishth, Tyler Poon, Udi Karpas, Vahid Noroozi, Venkat Srinivasan, Vijay Korthikanti, Vikram Fugro, Vineeth Kalluru, Vitaly Kurin, Vitaly Lavrukhin, Wasi Uddin Ahmad,
14

---

Published as a conference paper at ICLR 2026
Wei Du, Wonmin Byeon, Ximing Lu, Xin Dong, Yashaswi Karnati, Yejin Choi, Yian Zhang, Ying Lin. Yonggan Fu, Yoshi Suhara, Zhen Dong. Zhiyu Li, Zhongbo Zhu, and Zijia Chen. Nvidia nemotron nano 2: An accurate and efficient hybrid
reasoning model, 2025a.
NVIDIA, :, Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi. Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, Andrew Tao, Anna Shors. Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Buddharaju, Bobby Chen, Boris Ginsburg, Boxin Wang. Brandon Norick, Brian Butterfield, Bryan Catanzaro, Carlo del Mundo, Chengyu Dong. Christine Harvey, Christopher Parisien, Dan Su, Daniel Korzekwa, Danny Yin, Daria Gitman, David Mosallanezhad. Deepak Narayanan, Denys Fridman, Dima Rekesh, Ding Ma. Dmytro Pykhtar, Dong Ahn, Duncan Riach, Dusan Stosic, Eileen Long, Elad Segal, Ellie Evans, Eric Chung, Erick Galinkin, Evelina Bakhturina, Ewa Dobrowolska, Fei Jia, Fuxiao Liu, Gargi Prasad, Gerald Shen. Guilin Liu, Guo Chen, Haifeng Qian, Helen Ngo, Hongbin Liu, Hui Li, Igor Gitman, Ilia Karmanov, Ivan Moshkov, Izik Golan, Jan Kautz, Jane Polak Scowcroft, Jared Casper, Jarno Seppanen, Jason Lu, Jason Se- wall, Jiaqi Zeng. Jiaxuan You, Jimmy Zhang, Jing Zhang. Jining Huang, Jinze Xue, Jocelyn Huang, Joey Conway, John Kamalu, Jon Barker, Jonathan Cohen, Joseph Jennings, Jupinder Parmar, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Luna, Keshav Santhanam, Kezhi Kong, Kirthi Sivamani, Krzysztof Pawelec, Kumar Anik, Kunlun Li, Lawrence McAfee, Leon Derezynski, Lindsey Pavao, Luis Vega, Lukas Voegtle, Maciej Bala, Maer Rodrigues de Melo, Makesh Nar- simhan Sreedhar, Marcin Chochowski, Markus Kliegl, Marta Srepiewska-Dziubinss
Matthieu Le, Matvei Novikov, Mehrzad Samadi, Michael Andersch, Michael Evans, Miguel Martinez, Mike Chrzanowski, Mike Ranzinger, Mikolaj Blaz, Misha Smelyanskiy, Mohamed Fawzy, Mohammad Shoeybi, Mostofa Patwafry, Naycon Lee, Nima Tajbakhsh, Ning Xu, Oleg Rybakov, Olcksïi Kuchaiev, Olivier Delalleau, Osvald Nitski, Parth Chadha, Pasha Shamis, Paulius Micikevicius, Pavlo Molchanov, Peter Dykas, Philipp Fischer, everre-yees, Aquilanti, Piotr Bialecki, Prasoon Varshney, Pritam Gundecha, Przemek Tredak, Rabeeh Karimi, Rahul Kandu, Ran EI- Yaniv, Raviraj Joshi, Roger Waleffe, Ruoxi Zhang, Sabrina Kavanaugh, Sahil Jain, Samuel Kriman, Sangkug Lym, Sanjeev Satheesh, Saurav Muralidharan, Sean Narenthiran, Selvaraj Anandaraj, Seonmyeong Bak, Sergey Kashirsky, Seungju Han, Shantanu Acharya, Shaona Ghosh, Sharath Turuvekere Sreenivas, Sharon Clay, Shelby Thomas, Shrimai Prabhumoye, Shubham Pachori, Shubbam Toshniwal, Shyamala Prayaga. Siddhartha Jain, Sirshak Das, Slawek Kierat, Somshubra Majumdar, Song Han, Soumye Singhal, Sriharsha Niverty, Stefania Alborghetti, Suseella Panguluri, Swetha Bhendigeri, Syeda Nahida Akter, Szymon Migaez, Tal Shiri, Terry Kong, Timo Roman, Tomer Ronen, Trisha Saar, Tugrul Konuk, Tuomas Rintamaki, Tyler Poon, Ushnish De, Vahid Noroozi, Varun Singh, Vijay Korthikanti, Vitaly Kurin, Wasi Uddin Ahmad, Wei Du, Wei Ping, Wenliang Dai, Wonmin Byeon, Xiaowei Ren, Yao Xu, Yejin Choi, Yian Zhang, Ying Lin, Yoshi Suhara, Zhiding Yu, Zhiqi Li. Zhiyu Li, Zhongbo Zhu, Zhuolin Yang, and Zijia Chen. Nemotron-h h: A family of accurate and efficient hybrid transforrrr models, 2025b.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL httts::///ppessee
net htaee//foummiim wite
wHBfxh2ul.
Maciej Pióro, Kamil Ciebiera, Krystian Król, Jan Ludziejewski, and Sebastian Jaszezur. Moe-mamba: Efficient selective state space models with mixture of experts, 2024.
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
arxiv org. Barrg//bs//00 /200 .tt.//22111119919991
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2024.
Jack W Rae, Anna Potapenko. Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint, 2019. URL https://arxxis www./ogg///o/// 05507
15

---

Published as a conference paper at ICLR 2026
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text org/ abs 910 10683
transformer. CoRR, abs/1910.1066. 2019. http::///
Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hy- brid state space models for efficient unlimited context language modeling. ArXiv, abs/2406.07522, 2024. [Rttps::://api
ow..or///oooooo Corpus I 270380294
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024.
Siavash Shams, Sukru Samet Dindar, Xilin Jiang, and Nima Mesgarani. Ssamba: Self-supervised audio representation learning with mamba state space model. arXiv preprint 2rXiv:2405.1/83/ 2024.
Ning Shang, Li Lyna Zhang, Siyuan Wang, Gaokai Zhang, Gilsinia Lopez, Fan Yang, Weizhu Chen, and Mao Yang. Longrope2: Near-lossless Ilm context window scaling, 2025.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL httpttss//aaaa..oo///aaaaa 09864
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor transformer for large language models, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. ILaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302. 13971, 2023.
Junxiong Wang, Tushaar Gangavarapu. Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. arXiv preprint 2rX1v:2401.13666 2024.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020.
Zhifan Ye, Kejing Xia, Yonggan Fu, Xin Dong, Jihoon Hong, Xiangchi Yuan, Shizhe Diao, Jan Kautz, Pavlo Molchanov, and Yingyan Celine Lin. Longmamba: Enhancing mamba's 's long- context capabilities via training-free receptive field enlargement. In The Thirteenth International Conference Learning Representations, 2025.
1d= LddfMbLszV018
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X.X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention. 2025. PRtp::ts: http:////app.saaaiiaiiaaccee www.entiiccnoooeeeeeeeee.ee orw.o/ruuuuucuouuuooeeee
org/ Corpus : 276408811
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: efficient visual representation learning with bidirectional state space model. In Proceedings of the 41st International Conference on Machine Learning, ICML'24 24. JMLR.org, 2024.
Jingwei Zuo, Maksim Velikanov, Ilyas Chahed, Younes Belkada, Dhia Eddine Rhayem, Guillaume Kunsch, Hakim Hacid, Hamza Yous, Brahim Farhat, Ibrahim Khadraoui, Mugariya Farooq, Giulia Campesan, Ruxandra Cojocaru, Yasser Djilali, Shi Hu, Iheb Chaabane, Puneesh Khanna, Mohamed El Amine Seddik, Ngoc Dung Huynh, Phuc Le Khac, Leen AlQadi, Billel Mokeddem, Mohamed Chami, Abdalgader Abubaker, Mikhail Lubinets, Kacper Piskorski, and Slim Frikha. Falcon-h1: A family of hy bhaddddhead language models redefining efficiency and performance, 2025.
16

---

Published as a conference paper at ICLR 2026
A EXPERIMENT DETAILS
Model Checkpoints. The model checkpoints we experimented on, as listed in HuggingFace, are:
bm- ai-p -pl at form Bar--Bama-BBBae (Ganti et al.): trained on 4k context length, consists of 29 Mamba2 layers and Transformer layers with RoPE.
avidia/ a/eaaotetttt.. ron- --n----BE B---a------------ 8K (NVIDIA et al. 2025b): trained on 8k context length, consists of 24 Mamha2 layers and 4 Transformer layers without RoPE. Note that we are using the 8k base model instead of the one fine-tuned 128k context length.
state spaces/ paace///mmaaaa/// mamma2-aaa 2. 7b (Dao & Gu. 2024): trained 2k context length, con- sists of Mamba2 layers.
Evaluation. To evaluate the model's long context behavior, we adopt the following three widely used benchmarks
Perplexity on PG-19 (Rae et al. 2019) test set: consists of 100 books with an average length 69k tokens.
RULER (Hsich et al. 2024a): consists of 8 variants of needle-in-a-haystieee retrieval task, and 5 other tasks that test the model's context understanding ability. Note that S1 is equivalent to the passkey retrieval task (Mohtashami & Jaggi (023).
LongBench-E (Bai et al.) 2024): consists of 4 tasks on single-document QA, 4 tasks on multi-document QA, 4 tasks on summarization, 4 tasks few-shot learning, 3 tasks
synthetic tasks, and 2 tasks on code completion. In section $ 5.
we reported the average
score under each of the categories, well the overall average.
Baselines. At the time of writing, the LongMamba YYe et al. 2025) codebase has not published their pipeline for parameter searching, so we followed the description implement their two-stage method.
1. Parameter search: Conduct 16 runs on LongBench-E E and choose the hyperparameter setting (long-term head threshold 0, and large clipping percentage C') that maximizes the overall average score. This procedure yields superior performance LongBench-E but incurs substantial search cost (~ 10 hours on A100 GPUs) and must repeated for each new model. We also observe that performance other benchmarks is sensitive to this choice, making the search overhead necessary.
2. Small A clipping value look-up table: Build a look-up table for the clipping value per 1000 tokens, where each entry is computed via taking inference on 5 samples. We only compute the clipping value for the sequence length experimented on, the cost for this phase is 30 forward passes, which negligible comparing to the first phase.
During inference, Mambaa is efficient as the running time is still dominated by the state space model. In practice, the overall running time of LongMamba's pipeline takes 10.5 hrs on A100 GPUs.
Universal Position Interpolation (our method). As descrihed in section 4.4 our method consists of three steps, where the first and the third step does not require any computation. For the second step, we utilize the validation of the proof-pile dataset's ArXiv subset (Computer) 2023). consisting of 100 samples. We concatenate and truncate sequences to make sure all inputs contain 16k tokens. The model performs inference on all samples to acquire the average ERF of each Mamba head. This process takes 100 forward passes that can be conducted in parallel, which is less than 3 minutes on BxA100
B STATE SPACE MODELS AND MAMBA
State Space Models (SSMs) offer promising alternative to attention- based architectures for process- ing long sequences efficiently. An SSM is typically defined by a continuous time dynamical system that captures temporal dependencies through set of linear differential equations. Formally, SSM
17

---

Published as a conference paper at ICLR 2026
can be represented follows:
dh(k) Ah(k) (x(k),
dk
y (Ak) Ch(k) + Dx(k),
(4)
where hkk) represents the hidden state at time k. x(k) the input signal, and y(k) the output signal. Matrices A, C, parameterize the system dynamies and are learned during training. To accom- modate the model with discrete signals such as natural language, this continuous-time formulation is discretized into a linear recurrent equation:
h
Ah, + Bx.
(5)
Ch. + Dx:
(6)
with discrete-time system matrices A, B, C, D. This system can process long sequences efficiently, as they avoid attention's 's quadratic computation complexity and rely on parallelizable recurrence updates.
Mamba builds on SSMs but adds data-dependent gates enhance flexibility and expressiveness. The detailed design of Mamba and Mamba2 model can be found (Gu Dao 2023 Dao & Gu 2024). In this work, mainly invesitgate context extension methods without modifications to the model architecture.
The basic design of Mamba starts from an SSM discretized by zeroth-order hold (ZOH), while making the system matrices context-aware:
h h, + B
where A, exp(ArA),
(8)
B: (A,A) -1 (exp (AA) - I) A,B.,
(9)
A, SofPPlus( (WIssWWAx + ba),
(10)
B: WBX;
(11)
where all subscripts denote the dependency the current input x2, and A, denotes the discretization time step of it. By letting the matrices vary with the current input xt, Mamba implements a selective gating mechanism akin aain RNN forget and input gates that dynamically filters and propagates
content along the sequence.
In Mamba2, the design is further simplified as the following.
A := aI: The continuous forget gate matrix is simplified to a single parameter for each head.
B, R A,B, The ZOH discretization rule for the input gate is simplified to first-order approximation that effectively reduces computation.
By substituting A, in the expressions of ā. and B,, we can obtain the new expressions as:
a Sigmoid (-ooidddWWWW) -WWAx + ba))", B, = SoftPlus( Cetuus(WAA WAX; + ba) (WBx:),
(12)
Thanks the linear updating rules of the Mamba layer, we can unroll the recurrence into the following multiplication,
Y= = MX
(13)
B.
if = j.
M
C [Mlma41 0k
if i j. Bj,
(14)
0.
otherwise.
where a; is the forget gate, controlling the proportion of information retained the next step, while B, serves as inpul gate, determining how much information is added by the new input.
M can unrolled into a structured lower triangular matrix,
C
B
0
0
C a B
C
B
O
M
(15)
C
[
Bo
C
ah
B
CT
B
18

---

Published as a conference paper at ICLR 2026
which mimics the causal attention map of transforerr layer. The diagonal entries reflect direct influence from current input, and the lower triangle indicates cumulative influence of prior inputs decayed by forget gates. The outputs are then expressed weighted sum of all inputs via this pseudo-attention map.
C EFFECTIVE RECEPTIVE FIELD
Elfective Receptive Field (ERF) was first introduced for analyzing CNNs (Luo et al. 2016), and later adapted to Transformers (Dosovitskiy et al. 202T) and very recently Mamba (Ben- Kish-Kishea et al. (025). Although modern LLMs have theoretical full-context receptive fields (Lecun et al. 1998), the ERF captures the practical extent of this influence. To estimate ERF in our analysis, we adopt the Mamba Mean Distance (MMD) metric introduced by (Ben-Kish et al. 2025) given as:
MMD = Eyga [d(i = E 1) E
,=1
This measures the expected distance (in token positions) between the last token and the weighted influence earlier tokens, derived from the unrolled recurrence matrix M. high MMD indicates long-term memory, i.e., the last token is influenced by tokens far back in the sequence, while low value indicates short-range behavior, i.e., the model is mostly influenced by recent tokens.
noticeable advantage of using MMD) our ERF metric is that operates the attention score, which means it also applicable to analyze the behavior of Transformer models. This duality allows fair comparison between Mamba and Transformer layers, aligning with our goal of investigating hybrid models.
D ROPE AND POSITION INTERPOLATION
Rotary Position Embedding (RoPE, Su et al. (2023) is an increasingly popular method provide
position information into transformer models. Given a position index m, an array of frequencies O
(0g, 01.... 02/2-1] and an embedding vector x == [ro, #o,..1,.... ;.a.1 J where d the dimension of the attention head, RoPE defines a vector-valued complex function r(x, m, +) follows
Td-1]
r(x, m, +) = [r + iæ m. m,00): (o), r(x2 + iæ3, m, 01), r( + i m, 0d/2-1))
= [1ro + iæ i imoo, (I2 + ix im0; ,..., (a + i 1 imfa//2
where i == 1-1 is the imaginary unit and 8, fr f-2i/d 2i/d with fixed predefined base frequency fo that is commonly set to 10,000 or 500,000.
Position Interpolation (PI, Chen et al. 2023) defines an attention score aiming for interpolation, whose RoPE is replaced by the following
mL (x, m, θ) = (.,,))==<x(x,/ (7,,77.0)
where the trained context length and L' is the length to be extended to. Such simple scaling guarantees equal impact among all tokens, such that each token now takes L/L' of its original position. Since the transformation merely a change in rotation frequency but not attention magnitude, it won't 't harm the original attention pattern, while reducing the speed of decay.
E DUALITY BETWEEN ROPE AND MAMBA POSITIONAL ENCODING
We can establish duality between RoPE and Mamba positional encoding mechanisms by showing that both schemes are equivalent to maintaining a bank of keys. They both multiply each channel of the keys by exp Eieey An) where f is the current token position (time step in Mamba's context). j is the position associated with a given key, and A,, is a frequency (discretization per step in Mamba's context) term that vary per-channel and per-position.
19

---

Published as a conference paper at ICLR 2026
RoPE case: By construction, the dot product between given Qi and Kj with RoPE function r(-) is: r(Qu, 1, 0) Q2 exp (i10), r.
j. = K2 exp (ijØ),
+ (r(0,,,,,)r(K,,,, (Qr. Kj exp (-i0 - 1)))
t
(Q Kj) exp
Σ A
where (with some slight abuse of terminology) is vector, and exp (0) is diagonal matrix containing exp (0z-) for each coordinate 0₂ in 0. Here we keep the computation of RoPE within the complex plane and let A := -i0, where i = V-I. In practice we can take the magnitude of the real and imaginary part for the cosine and sine values.
Mamba case: The recurrence operation for Mamba can be simplified as: h = exp (-aA,) h, A B, (see for more details). Unrolling this recurrence gives:
n-1
h
Σ B,x;
a
B, B exp
aAk
k=i+1
we let C. Q:: Bj = K,, Xj = Vj, and 4 -ad; then the mamba output C.h,, equivalent to linear attention over Q, K, V, with K, scaled by exp (Ei=sti A.). matching our desired formula up to trivial off-by-one adjustment.
Further circumstantial evidence that RoPE frequencies and Mamba timesteps are equivalent is the fact that they are, in practice, initialized the same way. RoPE @ values are spread evenly in log-space between and the inverse of the base frequency, while Mamba's ba values are initialized uniformly in log-space between max and min gating thresholds. The failure modes are also the same: slowly-rotating RoPE channel pairs rotate out of distribution when input lengths exceed the training length, while high-frequency channels have completed several rotations and remain in-distribution. Similarly, slowly-converging Mamba heads grow their state magnitude out of distribution, while guickly-converging heads attain their upper bound and then stay stable in perpetuity. These intriguing links may provide avenues for further refinement of Mamba gate-scaling, but we leave this for future work.
F EXTENDED DUALITY TABLE AND DISCUSSIONS
The duality between Mamba's 's decay pattern and RoPE reveals deeper connection: both Mamba and Transformer encode position by modulating the influence of past inputs, but they do so through different mechanisms. We formalize this connection by extending the Structured State Space Duality (SSD) from Mamba2 (Dao & Gu 2024) from a position encoding perspective:
Structured State Space Duality A Position Information Encoding Perspective
The cumulative decay in Mamba is functionally dual to the explicit position encodings used in Transformers, with both introducing long-range dependencies in language sequence processing.
biases that progressively de-emphasizing
The detailed extended duality is listed in Table We include new rows on how Transformer's positional embeddings correspond to Mamba's 's decaying pattern cumulative forget gates, and identify the corresponding out-of-domain problem that causes bad generalization capability for the model.
Out-of-Domain (OOD) Problem Position Encoding. Mamba's long- long-context degradation can be interpreted form of dom-in- (OOD) failures in its implicit position encoding mecha- nism. This similar to observations RoPE-basee basedd transformers, where extrapolating sinusoidal embeddings beyond the training range leads distorted token interactions and degraded perfor-
20

---

Published as a conference paper at ICLR 2026
Table 9: Extended structured state space duality from a context scaling perspective.
SSM (Mamba)
Transformer
Interpretation
C (contraction matrix)
Q (queries)
Latent-to-output projection query
B (expansion matrix)
K (keys)
Input-to-laten projection key
X (input sequence)
V (valucs)
Projected input tokens value
1,
A
II
Bjs
<
Cumulative forget product
1=++1
0,
otherwise.
lower-triangular shape
P (causal attention mask) RoPE ALiBi
Causal bias
off-diagonal decay pattern
Position-dependent bias
Mr., C. A. Yr E. Mc, X.
T Softmax (K" PP)
Structured attention matrices
yr E.= Te, V.
Weighted sums over the input
Per-head SSM keree
Per head attention module
Head-s flead-speciaalzed feature Non-uniform domain scaling OOD problem
Head-wise gate scaling (ours)
RoPE dimension-wei scaling low-frequency dimension
high-ERF heads
mance
Shang et al. 2025
Barbero et al. 2025).
While RoPE's OOD issue arises from low-frequency
Field
12000
0:30
8000
0.00
Effective
4000
-0.50
RoPE
freq = 0.0001
freq = 0.0009
2000
-1.00
0.0039
10000
50000
00000
100000
Context Length (# tokens)
Context Length (# token)
Figure 5: Left: RoPE at different frequency. Right: Bamba's 's head-wise ERF change layer 14, collected from PG-19. The vertical lines show the training context length. For ROPE, low frequency dimensions (dark green) tend to only cover small portion (1 0.95) within trained length. For Mamba, heads with high ERF (pink and brown) that tends to grow much faster than others break the distribution of head magnitude.
rotations in unseen position ranges, Mamba's gated recurrence exhibits a similar failure mode: the cumulative decay function A, under-trained beyond the context window, leading to uncontrolled ERF expansion in some heads; some heads' ERFs plateau, while others grow unchecked; over time, the runaway heads overwhelm the rest. As shown Fig. 5 both Mamba and RoPE-based Transformers exhibit these pathological behaviors when inputs exceeds their training lengths.
Connecting Transformer and Mamba CLE methods. With the extended duality established, we may also give description the context extension methods, that applied to one type of the models, being projected to the other type. Some examples are listed below.
Project Transformer methods to Mamba models:
YaRN (Peng et al. 2024y Discriminate heads within different sets of ERF values, and their input/forget gates with different scaling parameters determined by the input sequence length. Our selective UPI can be categorized as special case of this type of methods.
NTK-based methods LocalLLaMA 2023a b)
21

---

Published as a conference paper at ICLR 2026
LongRoPE2 (Shang et al., 2025 Perform heuristic search for the optimal combinaiton of the critical ERF to start the scaling, and the scaling parameters for each head with ERF larger than the critical one.
Project Mamba methods to Transformer models:
Mamba-style Position Embedding (Gu & Dao 2023 Dao & Gu 2024):
Most current position embedding used by transformers are context agnostic, which is not the case in Mamba. We may build similar activation- context-aware position embedding, for more fine-grained position information.
DeciMamba (ien-Kish et al., 2025): Decimate portion of tokens, and only compute smaller attention map on the rest of the tokens, with the decimated entries in the attention map set to 0 for more concentrated attention.
LongMamba (Ye et al. 2025): Skip portion of tokens by copying the closest previous un-skipped token's positional embedding, which effectively keeps all positional embeddingg inside the trained domain.
We leave exploring these projected methods as an interesting future work for extending both types of models.
G ALTERNATIVE METRICS FOR HEAD SELECTION
Aside from ERF-based top-K% head selection, we also explored alternative metrics including average A,, [. and the average state growth rate. These metrics are also profiled on the same dataset (Proof-pill ArXiv) used for ERF. A, can be interpreted as a per-token discretization step controlling
700
600
700-
Bottom RON
500-
500
Top 20%
Count
500
400-
300
200+
200
100
100
100
0.0
A
Bamba-9EB 9B-v2, 4k
Nemotron-Be
mamba2-2 7b, 2k
Figure Distribution of average AS across Mamba heads different models at their training context length: (Left) Bamba-9B-v 9B-v2 at 4k, (Middle) Nemotron-Hl - H-8B at 8k, (Right) Mamba2-2.7 7B at 2k. Across all three models, we observe a consistent pattern: heads with high ERF tend to have average A, > 0, in line with our assumption in 4.2
the effective decay rate the state. Small A, corresponds to slow state decay and long memory retention, whereas large A, induces stronger decay and more rapidly changing state dynamics. The distribution of average A, values for the top-20% ERF heads versus the bottom-80% ERF heads for all three models are shown in Fig. 6 The results show clear pattern: high-ERF heads exhibit average A; values tightly concentrated near zero, while low-ERF heads display substantially larger A, magnitudes. This empirical behavior is consistent with our underlying assumption.
We have also explored using more direct gate-based metrics, such as the average forget gate E[T,] or the average state growth rate, identify the unstable heads. Their distributions are shown in Fig. and Fig. g In practice, these quantities do not reliably distinguish the problematic heads. For example, many low-ERF heads also exhibit average forget gate values close to yet do not display the large receptive fields or unconverged state behaviors that characterize the truly unstable heads. This suggests that instability governed by the joint dynamics both input and output gates, rather than by either gate in isolation. As for the average state magnitude growth, while the majority of the low-ERF heads gather around 1, it does not cleanly separate the two types of heads, and thus requires similar empirical threshold. We indeed validate that the heads with average large state magnitude growth correspond large ERF heads, so setting a threshold could result in a similar effect as using ERF.
22

---

Published as a conference paper at ICLR 2026
500-
Count
300-
300-
200
200-
100
200
0.0
0.2
0.4
0.6
(a)
Average forget gate
600
1000
Bottom 80%
Count
400
800-
300-
600
200
400-
200+
100
200
3.00
0.02 0.00 0.06
0.00 0.57 0.04 0.06
0.14
0.00
0.08 0.30 0.32
(b) Variance forget gate
Bamba-9B-Ve 4x
Nemotron-Ho B--n 8k
mamba2-2. 7b, 2k
Figure Mean (a) and variance (b) of forget gates across Mamba heads in different models at their training context length: (Left) Bamba-9B-v2 at 4k, (Middle) Nemotron-H-a8 H-8B at 8k, (Right)
Mamba2-2.7B at 2k. Across all three models, heads with high ERF tend to have average forget gate concentrating at with close-to-O variance.
3000
1750
Bottom 80%
2500
1500
Count
2007
1250
1000
750
750
500
500
250
250
2.0
1.1
1.2
1.0
3.2
1.3
1.0
1.3
Average state magnitude growth per step
Bamba-9B-v2,
Nemotron-B-Ba
7b,
Figure 8: Distribution of average state magnitude growth rate across Mamba heads in different models at their training context length: (Left) Bamba-9B-v2 at 4k, (Middle) Nemotron-H-8B at 8k, (Right) Mamba2-2.7B at 2k. State growth rate needs similar empirical threshold value to conduct head selection.
The variance of the forget gate can also be an important indicator of state stability. Consider an extreme toy example with two heads: in the first, all gates are except every n-th gate O; in the second, the gate value is constantly (n 1)/n. Although these heads share the same average gate value, the first has an ERF upper-boundn bounded by n, whereas the second can achieve much larger ERF. The histogram the variance of forget gates over each head confirms that high ERF heads identified indeed have smaller variance than the rest.
H COMPARISON WITH DECIMAMBA
DeciMamba (Ben- Kish et al., 2025) is, knowledge, the first method targeting context extension for Mamba models. While its token- token-pruning scheme based on SSM step size is quite inspiring, the relatively long training horizon (roughly 1 steps), and is designed in a task desk-specific manner.
original DeciMamba pipeline is tailored to Mambal (Gu & Dao 2023). operates
For fair comparison, adapted the DeciMamba pipeline to Mamba2-2.77 and fine-tuned the model on the PG-19 training split for 500 steps before evaluating perplexity. As shown in Table TO UPI is comparable DeciMamba at shorter contexts for sequence lengths below 16k, but surpasses
23

---

Published as a conference paper at ICLR 2026
Model
Method
4k
8k
16k
32k
64k
DeciMamba
6.83
7.47
8.33
15.42
34.72
67.88
Mamba2-2.7B
UPI
7.40
8.21
8.75
13.69
17.58
22.24
Table 10: Evaluation of Mamba2-2.7B on PG-19 perplexity, comparing the effectiveness Deci- Mamba and UPI. DeciMamba's 's token-pruning scheme shows better performance on short context (2k 8k), while UPT' state growth stabilization improve performance on even longer context.
it at longer contexts, achieving up to 45.64 perplexity points improvement 64k. We believe this reflects a key difference in design: while DeciMaaba achieves strong performance at moderate lengths, its token-pruning scheme does not address the unstable state growth issue that dominates at very long contexts, an issue UPI directly mitigates. We will include this comparison the final version for completeness.
I
LLM USAGE
Large language models were used solely for writing assistance. No LLMs were involved in developing the methodology, running experiments, or drawing conclusions.
24


Reviews:
[{"review_id": "oyVwjtRJSV", "reviewer": "Reviewer_iUod", "summary": "- This paper addresses the Context Length Extension (CLE) problem for hybrid Mamba-Transformer models. While these hybrid models are computationally efficient and perform well within their training context window, the authors show they \"collapse\" catastrophically when presented with sequences longer than what they were trained on (e.g., perplexity spikes when a 4K-trained model sees 8K tokens).  \n- The failure is not just due to the Transformer components, but primarily stems from a subset of \"unstable\" or \"unconverged\" heads within the Mamba layers. These heads have forget gates consistently close to 1, causing their internal state to grow uncontrollably beyond the training length. This \"state magnitude explosion\" overwhelms the outputs of other, stable heads, leading to a \"feature collapse\" where most of the model's neurons become inactive.  \n- The authors introduce UPI, a training-free, closed-form method to scale the context window of hybrid models. UPI works by:  \n  - Using a small calibration dataset (100 samples, <3 minutes) to find the top 20% of Mamba heads with the largest Effective Receptive Field (ERF), which are the unstable ones.  \n  - For these selected heads, it scales their discretization step size $\\Delta_t$ by a factor of $\\frac{1}{n}$ (where $n$ is the context scaling factor). This effectively \"slows down\" the state updates, preventing explosive growth.  \n  - It seamlessly integrates with existing Transformer CLE methods like YaRN to handle the positional encodings in the Transformer layers.\n- UPI is evaluated on models like Bamba and Nemotron-H, extending their effective context from 4K/8K up to 64K tokens. It achieves substantial reductions in perplexity and improves performance on long-context benchmarks (RULER, LongBench) without any fine-tuning, outperforming baselines that combine methods like LongMamba and YaRN.", "questions": "- How well does this method scale to larger models?  \n- Could you please include a comparison with DeciMamba?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is well written and easy to follow.  \n- The paper provides the first systematic analysis of why hybrid models fail on long contexts.  \n- UPI is a simple, lightweight, and highly effective. Its training-free nature and minimal computational overhead for calibration.  \n- The experiments are comprehensive, using multiple benchmarks (PG-19, LongBench, RULER) and models. The results are convincing, showing dramatic improvements in perplexity and task accuracy at extended contexts.", "weaknesses": "- The choice of the \"top-20% of heads by ERF\" is shown to be effective but is ultimately a heuristic. While the paper shows this selection is robust across datasets, a more theoretical or analytical justification for this specific threshold would strengthen the method.  \n- The largest model tested is 9B parameters. While this is a reasonable scale, validating the method on larger hybrid models would be important to confirm its scalability.  \n- The results on the pure Mamba2 model show that UPI is outperformed by the LongMamba baseline on the LongBench benchmark.  \n- The tables do not include comparison to DeciMamba [1].  \n\n[1] DeciMamba: Exploring the Length Extrapolation Potential of Mamba", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kVf3UhPe0S", "reviewer": "Reviewer_5ThG", "summary": "This paper addresses the long-context generalization failure of hybrid Mamba-Transformer models and proposes a training-free context length extension (CLE) method called Universal Position Interpolation (UPI). UPI unifies Mamba’s cumulative decay and Transformer’s rotary positional encoding (RoPE) scaling, enabling hybrid/pure Mamba models to extend context length from 4K/8K to up to 64K tokens without sacrificing short-context accuracy.", "questions": "1. Rationale for the 20% Top-K Threshold for Mamba Heads. \nWhat is the systematic basis for choosing 20% specifically? Is it derived from theoretical analysis (e.g., the proportion of unstable heads in hybrid models) or just empirical tuning on the tested benchmarks? Does this threshold generalize across different hybrid model architectures (e.g., models with more Transformer layers than Mamba layers) or task types (e.g., code completion vs. multi-document QA)? If the threshold needs adjustment for new models/tasks, what guidelines can be provided?\n\n\n2. Generalization to Longer Contexts and Unseen Model Configurations\nThe paper extends context lengths from 2K/4K/8K to 64K, but key gaps remain: Have the authors tested UPI on longer contexts beyond 64K (e.g., 128K or 256K tokens)? If not, does the authors anticipate performance degradation, and what mechanisms might limit scalability?\n\n3. Comparison with More Relevant Baselines for Hybrid Models. \nThe baseline comparison uses LongMamba (for pure Mamba) + YaRN (for Transformer), but this is a \"component-wise combination\" rather than a dedicated hybrid CLE method. Have the authors considered baselines that jointly optimize Transformer and Mamba layers for hybrid models?\n\n4. Applicability to Pure Transformer Models and Adaptive Head Selection. The paper frames UPI as \"universal,\" but its utility for pure Transformer models is unaddressed. Can UPI be adapted to pure Transformer models? Since UPI unifies RoPE scaling and Mamba gate dynamics, would applying its core logic (e.g., selective dimension-wise scaling based on ERF-like metrics) outperform existing Transformer CLE methods (e.g., YaRN, LongRoPE)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper provides the first systematic investigation of long-context failure in hybrid Mamba-Transformer models, addressing a critical gap, prior work focused exclusively on pure Transformer or Mamba architectures, ignoring the unique challenges of hybrid designs. The identification of \"uncontrolled state growth\" and \"uneven receptive field contributions\" as root causes is a foundational insight.  \n\n2. It balances mathematical rigor (e.g., state magnitude explosion derivation, UPI closed-form proof) and empirical validation. ERF analysis and state growth characterization provide concrete evidence for failure modes, while ablation studies (selective head adjustment, Top-K thresholding) validate the method’s components.  \n\n\n3. Hybrid Mamba-Transformer models are emerging as efficient alternatives to Transformers, but their long-context limitation hinders real-world use (e.g., document understanding, multi-turn dialogue). UPI extends their context window from 4K to 64K tokens without performance loss, making these models viable for long-sequence tasks.", "weaknesses": "1. Subjectivity and Non-Generality of the Top-20% High ERF Head Threshold: UPI relies on the rule of \"selecting the Top-20% of high ERF heads for adjustment,\" but this threshold is determined solely based on experimental experience from benchmarks such as PG-19 and RULER, lacking systematic validation. The selection logic for \"20%\" is not explained, nor is its adaptability to different model architectures (e.g., models with different ratios of Mamba/Transformer layers) and task types (e.g., code generation, multi-turn dialogue) verified.\n\n2. Insufficiently Comprehensive Baselines: Only \"LongMamba + YaRN\" was selected as the baseline. However, LongMamba is a non-specific baseline designed for pure Mamba and is not compared with \"CLE methods specifically optimized for hybrid models\" (e.g., Hymba's hybrid head adjustment logic, Samba's state space co-extension). Furthermore, \"fine-tuning CLE methods\" (e.g., long context fine-tuning for hybrid models) are not compared, making it impossible to quantify the actual value of UPI's \"no-training\" advantage (e.g., fine-tuning methods may have higher performance; is UPI's deployment advantage sufficient to compensate for the performance gap?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YuUg5xVGRo", "reviewer": "Reviewer_dePa", "summary": "This paper addresses context length extension (CLE) for hybrid Mamba-Transformer models, which combine SSM layers with attention layers for efficiency. The authors identify that these models fail beyond training context lengths due to uncontrolled state growth in certain Mamba heads and uneven receptive field contributions. They propose Universal Position Interpolation (UPI), a training-free method that selectively rescales unstable Mamba heads (top 20% by ERF) while applying YaRN to Transformer layers. The method is evaluated on Bamba, Nemotron-H, and Mamba2 models across PG-19, RULER, and LongBench benchmarks.", "questions": "1.  The \"top-20% ERF\" heuristic is practical and effective. Have you explored a more direct method for identifying the unstable heads during calibration? For example, could one directly measure the average forget gate $\\overline{a}$ or the state growth rate per head and set a threshold based on that?\n2.  How did authors select the 20% threshold for Nemotron-H and Mamba2-2.7B? Was the 20% value from the Bamba-v2 ablation (Table 5) found to be universally optimal, or did this hyperparameter require tuning for each model?\n3.  Can authors provide an empirical analysis (e.g., a histogram or density plot) of the $\\Delta_t$ distributions for the selected top-20% heads versus the remaining 80%? This would directly validate the core assumption used in the derivation of the simplified input gate scaling (Eq 2).\n4.  In Table 3, LongMamba outperforms UPI on the pure Mamba2 model. Authors plausibly argue this is due to LongMamba's task-specific tuning. Could this result also suggest a potential limitation of UPI's \"universal\" $\\Delta_t / n$ scaling when compared to LongMamba's more complex mechanism, especially in a pure-SSM setting?\n5.  The RoPE-Mamba duality is a fascinating conceptual bridge. Does this duality suggest that other, more advanced RoPE-scaling techniques (e.g., NTK-aware scaling) could be \"translated\" to Mamba gate-scaling? What do authors see as the most promising future work based on this \"principled bridge\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 - First systematic analysis of context length limitations in hybrid Mamba-Transformer models, addressing a practically relevant issue.\n\n2 - The ERF analysis (Section 3.1) and state magnitude explosion analysis (Section 3.2) provide valuable insights into why hybrid models fail on long contexts.\n\n3 - UPI requires no fine-tuning, making it practically appealing with minimal computational overhead (<3 minutes calibration).", "weaknesses": "1.  **Heuristic Head Selection:** The method identifies \"unconverged\" heads using a proxy: the \"top-20% by ERF\". While the authors justify this correlation and show its robustness across datasets (Table 6), it remains a heuristic. A more direct, analytical method to identify heads where $E[\\overline{a}] \\approx 1$ would be more theoretically satisfying.\n2.  **The \"20%\" Threshold:** The 20% threshold is presented as a robust choice, but the ablation (Table 5) is only shown for Bamba-v2. It's unclear if 20% is a \"magic number\" that generalizes to all models, or if it was re-tuned for Nemotron-H and Mamba2. If it requires per-model tuning, it slightly weakens the \"tuning-free\" claim (though it's still far cheaper than the baseline).\n3.  **Missing Empirical Validation of $\\Delta_t$:** The key derivation (Eq 2) relies on the fact that the unstable heads have $\\Delta_t \\rightarrow 0$. This is a reasonable assumption, but the paper would be strengthened by an empirical validation. A histogram of $\\Delta_t$ values for the top-20% ERF heads versus the bottom-80% would provide direct evidence to confirm this crucial assumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)