Falcon Mamba: The First Competitive Attention-free 7B Language Model
Authors: name: Jingwei Zuo; name: Maksim Velikanov; name: Dhia Eddine Rhaiem; name: Ilyas Chahed; name: Younes Belkada; name: Guillaume Kunsch; name: Hakim Hacid
Published: 2024-10-07 15:40:45

Abstract: In this technical report, we present Falcon Mamba 7B, a new base large
language model based on the novel Mamba architecture. Falcon Mamba 7B is
trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure
Mamba-based model, Falcon Mamba 7B surpasses leading open-weight models based
on Transformers, such as Mistral 7B, Llama3.1 8B, and Falcon2 11B. It is on par
with Gemma 7B and outperforms models with different architecture designs, such
as RecurrentGemma 9B and RWKV-v6 Finch 7B/14B. Currently, Falcon Mamba 7B is
the best-performing Mamba model in the literature at this scale, surpassing
both existing Mamba and hybrid Mamba-Transformer models, according to the Open
LLM Leaderboard. Due to its architecture, Falcon Mamba 7B is significantly
faster at inference and requires substantially less memory for long sequence
generation. Despite recent studies suggesting that hybrid Mamba-Transformer
models outperform pure architecture designs, we demonstrate that even the pure
Mamba design can achieve similar, or even superior results compared to the
Transformer and hybrid designs. We make the weights of our implementation of
Falcon Mamba 7B publicly available on
https://huggingface.co/tiiuae/falcon-mamba-7b, under a permissive license.