Hybrid Architectures for Language Models: Systematic Analysis and Design
  Insights
Authors: name: Sangmin Bae, name: Bilge Acun, name: Haroun Habeeb, name: Seungyeon Kim, name: Chien-Yu Lin, name: Liang Luo, name: Junjie Wang, name: Carole-Jean Wu
Organization: MetaSuperintelligenceLab

Recent progress in large language models demonstrates that hybrid
architectures--combining self-attention mechanisms with structured state space
models like Mamba--can achieve a compelling balance between modeling quality
and computational efficiency, particularly for long-context tasks. While these
hybrid models show promising performance, systematic comparisons of
hybridization strategies and analyses on the key factors behind their
effectiveness have not been clearly shared to the community. In this work, we
present a holistic evaluation of hybrid architectures based on inter-layer
(sequential) or intra-layer (parallel) fusion. We evaluate these designs from a
variety of perspectives: language modeling performance, long-context
capabilities, scaling analysis, and training and inference efficiency. By
investigating the core characteristics of their computational primitive, we
identify the most critical elements for each hybridization strategy and further
propose optimal design recipes for both hybrid models. Our comprehensive
analysis provides practical guidance and valuable insights for developing
hybrid language models, facilitating the optimization of architectural
configurations.