Title: Mamba Modulation: On the Length Generalization of Mamba Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Length Generalization

Mamba

Efficient Method

State Space Models

Length Extrapolation

Calibration
Abstract: The quadratic complexity of the attention mechanism in Transformer models has motivated the development of alternative architectures with sub-quadratic scaling, such as state-space models. Among these, Mamba has emerged as a leading architecture, achieving state-of-the-art results across a range of language modeling tasks. However, Mamba’s performance significantly deteriorates when applied to contexts longer than those seen during pre-training, revealing a sharp sensitivity to context length extension. Through detailed analysis, we attribute this limitation to the out-of-distribution behavior of its state-space dynamics, particularly within the parameterization of the state transition matrix $A$. Unlike recent works which attribute this sensitivity to the vanished accumulation of discretization time steps, $\exp(-\sum_{t=1}^N{\Delta}_t)$, we establish a connection between state convergence behavior as the input length approaches infinity and the spectrum of the transition matrix $A$, offering a well-founded explanation of its role in length extension. Next, to overcome this challenge, we propose an approach that applies spectrum scaling to pre-trained Mamba models to enable robust long-context generalization by selectively modulating the spectrum of $A$ matrices in each layer. We show that this can significantly improve performance in settings where simply modulating ${\Delta}_t$ fails, validating our insights and providing avenues for better length generalization of state-space models with structured transition matrices.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the length generalization capabilities of Mamba. The paper provides theoretical arguments tying the spectrum of the transition matrix to the convergence behavior of the hidden state when applied to infinitely long sequences. The paper demonstrates that scaling the spectrum of the transition matrix directly is favorable compared to previous works that scale the discretization step.\nSupporting experiments are provided.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n* The paper discusses a very relevant and vibrant topic of length generalization.\n* The paper is written well and structured in an intuitive flow.\n* The presented method for scaling the eigenvalues (and improving length generalization) is simple and clear.\n\n**Weaknesses**:\n* Results in the paper are trivial and the significance seems very limited.\n* The paper does not discuss known results on other architectures or distinguishes the results and the novelty brought forth in the paper.\n* Related work is very lacking - missing works on extrapolation and length generalization in different setups. I would expect a paper discussing length generalization of Mamba to discuss:\n   - Works related to extrapolation in transformers:\n      - Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. Press et al. ICLR 2022.\n      - Unveiling transformers with lego: a synthetic reasoning task. Zhang et al. Arxiv 2022.\n      - etc.\n   - Works discussing length generalization and extrapolation in RNNs:\n      - Implicit Bias of Linear RNNs. Emami et al. ICML 2021.\n      - On the implicit bias of gradient descent for temporal extrapolation. Cohen-Karlik et al. AISTATS 2022.\n      - Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Network. Cohen-Karlik et al. ICLR 2023.\n      - etc.\n\nQUESTIONS:\n- The sentence in lines 30-33 asserts that the primary motivation for mamba is length generalization capabilities. Can you please provide grounds for this claim? To the best of my knowledge Mamba shows superior performance compared to previous SSMs on long sequences (not necessarily in the extrapolation setup).\n\n- What is $s$ in line 151? it isn't defined before it is used.\n\n- Figure 1 isn't clear from its description - Specifically, this is the spectrum per layer of a specific model. What does it mean that certain layers have fast decay while others decay slowly?\n\n- Why is Lemma 4.1. state as a formal lemma? this is trivial and does not require a dedicated proof.\n\n- In Theorem 4.2, Corollary 4.3 & Corollary 4.4 - why is the assumption on the eigenvalue distribution different per statement? In general the statements are not very clear when written verbally, I would suggest writing the statement formally (in lines 192-193 and 196-197).\n\nLIMITATIONS:\nThe empirical benefit of scaling $\\mathbf{A}$ instead of $\\mathbf{\\Delta}$ isn't clear. Results seem to be marginally better in certain setups and worse in others. The authors fail to explicitly discuss this in the paper. It would be good to discuss in a dedicated section the tradeoffs and benefits of the different scaling methods, as it stands the method presented seems to offer incremental novelty compared to MambaExtend.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn Mamba styled models (Mamba and Mamba2), there are 2 parameters which have so far been controlled for to enable length generalisation. One a state transition matrix - $A$, the other being $\\Delta$, a parameter used for discretising the other main parameters in the SSM core equation. Prior work has focussed for controlling for $\\Delta$, this paper argues why controlling $A$ would make much more sense. There are some theoretical results that the paper contributes that point to the same and the authors also provide quite strong experimental results to validate the claim.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- There is very strong empirical evidence and the experiments are all very comprehensive. \n- The core idea communicated in the paper is quite actionable and is thus quite relevant for LLM practitioners who use Mamba based foundation models. \n\n**Weaknesses**\n- The core idea communicated in the paper is quite simple, and I somewhere feel the writing of the paper currently has been optimised to fit the full 9 pages somehow. I think the paper could have been written in a much more succinct manner. \n\nI am a bit conflicted about the paper. While I think the paper is quite solid and makes a very specific important point very convincingly, I do think that the full 9 pages are not required to put this point across. Thus, probably there is scope to think about a more appropriate format / venue for the paper. Although I wouldn't be opposed to this paper being accepted if others as well as the authors disagree. Especially because I don't think there is anything else that can make the paper better. Thus, I would be willing to raise my score further, in case the authors engage and communicate why they think the current format and version of the paper is a good choice to communicate this idea.\n\nQUESTIONS:\n**Questions**\n- Why does section 4.1 exist, what’s the point of that analysis ? This just lists the graph and says a generic statement, is there something I missed ? I currently think that not much will be lost if that section just never existed. \n- Although 6.1 is very convincing, just maybe add a remark about why there could still be cases like GovReport Mamba - 1.4B where perplexity of scaling Delta is better. Acknowledge it or just share intuition. \n- Based on the graphs, it seems like initially controlling for $\\Delta$ starts out as being better, and then controlling for $A$ becomes better. Is there any scope of controlling for both, or is that stupid ? \n- What do you think about the suggestion to rewrite the paper to be tighter / shorter to communicate the core idea quicker and more intuitively, or do you think this long paper format of NeurIPS makes most sense ? \n\n**Suggestions**\n- MambaExtend is by Azizi et al, that wasn’t clear from the main paper when it was mentioned, maybe add the citation in the section as well, instead of it just being in the Appendix.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

comments: SUMMARY:
This paper focuses on the sequence length extrapolation problem in Mamba-family models, identifies the spectrum of the transition matrix A as a key influencing factor, and proposes methods to adjust the spectrum of A to enhance the sequence extrapolation capability of Mamba-family models. The paper achieves promising experimental results.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The problem addressed in this paper is important, and the proposed ideas are highly insightful.
2. The method proposed in this paper is easy to understand and implement.
3. The paper presents solid experimental results that offer valuable insights.

Weaknesses:

1. The paper argues that the instability (explosion or vanishing) of the state norm with increasing sequence length in Mamba-family models is the main cause of failure in length extrapolation. However, there is no direct experimental validation of the state norm behavior throughout the paper, which raises questions about the validity of the theoretical analysis.
2. Prior works, such as DeciMamba, have also proposed explanations for the failure of Mamba models in handling length extrapolation. This paper does not sufficiently argue that its explanation is superior to those provided in prior work, nor does it point out any flaws or inaccuracies in the existing explanations. This weakens the persuasiveness of the theoretical analysis.

QUESTIONS:
See the weaknesses mentioned above.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper identifies spectral instability in the state-space transition matrix A as the cause of weak length generalization of Mamba & Mamba-2.
The authors prove that extreme eigenvalues critically affect hidden-state divergence. To remedy this problem, this paper proposes uniformly rescaling A with calibrated values through backpropagation.
This scaling of A is training-free (using pretrained weights directly) without updating model weights (but it actually tunes A values through calibration).
This solution keeps reasonable perplexity upto 64 K–128 K tokens while boosting long-context accuracy on tasks such as LongBench and Passkey Retrieval.

STRENGTHS AND WEAKNESSES:
The idea is simple, and the calibration requires very minimal training of multiplier values, which leads to good reproducibility. The results are competitive, maintaining good perplexity.
However there are some weaknesses:
- the contributions are limited : scaling a different parameter in the same training-free calibration framework has only subtle differences from MambaExtend.
- the theoretical analysis assumes uniform, diagonal spectra, limiting real-world applicability.
- it is not clear how section 4.2 directly leads to the motivation of controlling A only.
- using one scalar per layer coarsely addresses spectral outliers. this leaves finer eigenvalue issues not fully resolved


minor: figure 2 legend color should be corrected.

QUESTIONS:
Does this calibration method actually perform better than fine-tuning only A or $\Delta$t values (with other parameters fixed) directly to long context length?

What happens if combining both $\Delta$t and A scaling?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 1
DATE: 2025-07-03