Title: Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
Venue: NeurIPS 2025
Accepted: yes
Keywords: State Space Model; Mixture of Experts; Mamba
Abstract: State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant per-step inference-time computation and memory complexity. Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations, positioning them as strong alternatives to Transformers for long sequence modeling. However, efficiently scaling the expressive power of SSMs, particularly with Mixture of Experts (MoE), remains challenging, as naive integration attempts often falter or degrade performance. In this work, we introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts. By sharing routing decisions between projection layers and lightweight sub-modules within Mamba across experts, RoM leverages synergies among linear projection experts for effective and efficient sparse scaling of Mamba layers. At a scale of 1.3B active parameters (10B total) and 16K training sequence length, RoM achieves language modeling performance equivalent to a dense Mamba model requiring over 2.3$\times$ more active parameters, and demonstrates consistent perplexity across context lengths. Experimental results further show RoM effectively scales hybrid language models, yielding a 23% FLOPS saving compared to dense Mamba scaling for similar performance. We release our training codebase at https://github.com/zhanzheng8585/Routing-Mamba.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Routing Mamba (RoM) to address the challenge of scaling the expressive power of Linear State Space Models (LSSMs) for long-sequence modeling. While LSSMs are gaining attention as efficient alternatives to Transformers, the naïve application of Mixture-of-Experts (MoE) often results in performance degradation or training failure. RoM mitigates this by sharing routing decisions across the projection layer and lightweight submodules within each Mamba block, leveraging synergy among experts. The model matches the performance of a much larger dense Mamba with only 1.3B active parameters—2.3× fewer—and achieves a 23% reduction in FLOPs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe paper clearly demonstrates that applying MoE naïvely to Mamba can degrade performance and proposes a concrete method to resolve this. Through comprehensive experiments, the authors show that RoM consistently improves performance across various context lengths. It also outperforms alternative strategies like MoA, Switch Transformer heads, and even naive Mamba layer expansion.\n\nWeaknesses\nAlthough the proposed method is empirically effective, the paper somewaht lacks a clear explanation of why the naïve MoE approach fails and why the proposed design works. The motivation behind the architectural choices is somewhat opaque, and statements like \"focusing on “he most impactful layers” or \"reducing router learning difficulty” remain abstract. A deeper discussion of the underlying failure modes and the rationale behind each design decision would help readers better understand and generalize the approach.\n\nQUESTIONS:\n- The motivation behind several design choices in RoM is unclear. In Section 4.1, the paper shows that naïvely applying MoE to Mamba leads to degraded performance, and argues that simply attaching MoE modules independently to each unit is insufficient. However, in Section 4.2, the explanation jumps directly to the proposed solution without a clear analysis of why the naïve approach fails. Phrases like “focusing on the most impactful layers” or “lowering router learning difficulty” are used, but remain abstract without further justification. It would strengthen the paper if the rationale behind these claims were clarified and better supported.\n\n- In particular, the claim that reusing routing decisions helps reduce the difficulty of router learning could be made more convincing. A comparison between independent and shared routers—for example, showing PPL vs. training steps or identifying early plateau points—would help support this argument empirically.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents a careful study on applying Mixture-of-Experts techniques to State Space Models (SSMs). It demonstrates that naively routing different parts of the model independently does not yield strong performance. Instead, sharing routing strategies via carefully chosen projection matrices results in better outcomes. The proposed method, RoM, shows promise across multiple configurations and tasks.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. The paper conducts a thorough and well-scoped investigation into where and how sparse structures can be applied within SSMs. It clearly demonstrates that straightforward sparsification is not sufficient.

2. The authors identify effective strategies for layer selection and routing design, leading to a generalizable approach that applies across multiple model sizes and context lengths. On smaller model scales, RoM achieves significant gains over larger dense baselines.

3. The paper includes thoughtful design choices—such as eliminating the need for a load-balancing loss—while still achieving stable training, which highlights the practicality and robustness of the method.

**Weaknesses**

The paper does not extensively analyze the trade-offs between routing overhead and performance gains, especially as the number of experts increases, or selecting at least 2 experts per-token. A deeper cost-benefit discussion would strengthen the practical relevance.

QUESTIONS:
See `Weaknesses`

LIMITATIONS:
See `Weaknesses`
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This paper proposes a new method to scale State Space Models (SSMs), like Mamba, with Mixture-of-Experts. The authors indicate that naive integration of MoE into Mamba's architecture is ineffective and can degrade performance. The primary contribution is a novel method called Routing Mamba (RoM), which applies a sparse mixture of experts to the linear projection layers of Mamba. Through extensive experiments, the paper demonstrates that RoM outperforms standard dense Mamba models..

STRENGTHS AND WEAKNESSES:
Strengths:
- The authors conducted a comprehensive set of experiments. The authors validate their approach across multiple model scales, demonstrating consistent and significant performance gains. 
- The paper is well-written, clear, and easy to follow. 

Weaknesses:
- The contribution, while effective, could be viewed as incremental. The central novel idea is the shared routing strategy, which could be seen as a well-executed "engineering trick" rather than a novel algorithm since it lacks theoretical results to support the method. The explanation is a bit heuristic. There is no formal analysis to prove why this approach should lead to more stable training or better expert specialization.

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Routing Mamba (RoM), a novel framework that integrates Mixture-of-Experts (MoE) into State Space Models (SSMs), specifically Mamba, to enhance scalability and efficiency in sequence modeling.\n\nNaive MoE integration into Mamba layers degrades performance due to the complex interplay of projection layers (Conv, Gate, Output).\n\nTherefore, RoM solves this issue by employing a shared routing strategy across the projection layers, selectively scaling impactful components while reusing routing decisions to reduce overhead and improve coherence. In RoM, only the top $K$ experts corresponding to the highest gating weigths $\\\\mathcal{R}_i(\\\\mathbf{X}_t)$ are selected. Where the gating is performed through a routing projection layer $\\\\mathbf{W}_r$, followed by a SoftMax for obtaining probabilities. \n\nThe method is evaluated on Mamba with parameters varying from 115M to 1.3B, where 1 out of 8 expert networks for each token at each layer is activated. The models are trained on 20B tokens of SlimPajama, and evaluated on its validation set. Additionally, the approach is tested on several other zero-shot datasets: LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, and WinoGrande. The results show clear dominance of RoM over the baseline Mamba, under the same number of active parameters.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper is well written.  \n- The approach is properly tested on multiple datasets.  \n- The contribution of each block is studied.\n- RoM matches the performance of dense Mamba models with 2.3× fewer active parameters.  \n- Achieves 23% FLOPs reduction in hybrid SSM-attention models (e.g., Samba).  \n- Maintains consistent perplexity across varying context lengths (4K–16K).  \n\n**Weaknesses:**\n- While RoM reduces active parameter usage and FLOPs, training throughput still suffers.  \n- Although shared, the routing mechanism introduces new design challenges and could affect training stability in less controlled settings.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}