Title: Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
Primary area: foundation or frontier models, including LLMs
Keywords: Hybrid Architecture, Transformer, Mamba, Language Modeling, Efficiency

Abstract: Recent progress in large language models demonstrates that hybrid architectures--combining self-attention mechanisms with selective state space models such as Mamba--can achieve a compelling balance between modeling quality and computational efficiency, particularly for long-context tasks. While these hybrid models show promising performance, systematic comparisons of hybridization strategies and analyses on the key factors behind their effectiveness have not been clearly shared to the community. In this work, we present a holistic evaluation of these hybrid architectures via both inter-layer (sequential) and intra-layer (parallel) fusion. We evaluate these designs from a variety of perspectives: language modeling performance, long-context capabilities, scaling analysis, and training and inference efficiency. By investigating the core characteristics of each computational primitive, we identify the most critical elements for each hybridization strategy and further propose optimal design recipes for both hybrid models. Our comprehensive analysis provides practical guidance and valuable insights for developing hybrid language models, facilitating the optimization of architectural configurations.