MambaByte: Token-free Selective State Space Model
Authors: name: Junxiong Wang; name: Tushaar Gangavarapu; name: Jing Nathan Yan; name: Alexander M Rush
Published: 2024-01-24 18:53:53

Abstract: Token-free language models learn directly from raw bytes and remove the bias
of subword tokenization. Operating on bytes, however, results in significantly
longer sequences, and standard autoregressive Transformers scale poorly in such
settings. We experiment with MambaByte, a token-free adaptation of the Mamba
state space model, trained autoregressively on byte sequences. Our experiments
indicate the computational efficiency of MambaByte compared to other byte-level
models. We also find MambaByte to be competitive with and even outperform
state-of-the-art subword Transformers. Furthermore, owing to linear scaling in
length, MambaByte benefits from fast inference compared to Transformers. Our
findings establish the viability of MambaByte in enabling token-free language
modeling.