Speech Slytherin: Examining the Performance and Efficiency of Mamba for
  Speech Separation, Recognition, and Synthesis
Authors: name: Xilin Jiang; name: Yinghao Aaron Li; name: Adrian Nicolas Florea; name: Cong Han; name: Nima Mesgarani
Published: 2024-07-13 00:35:21

Abstract: It is too early to conclude that Mamba is a better alternative to
transformers for speech before comparing Mamba with transformers in terms of
both performance and efficiency in multiple speech-related tasks. To reach this
conclusion, we propose and evaluate three models for three tasks: Mamba-TasNet
for speech separation, ConMamba for speech recognition, and VALL-M for speech
synthesis. We compare them with transformers of similar sizes in performance,
memory, and speed. Our Mamba or Mamba-transformer hybrid models show comparable
or higher performance than their transformer counterparts: Sepformer,
Conformer, and VALL-E. They are more efficient than transformers in memory and
speed for speech longer than a threshold duration, inversely related to the
resolution of a speech token. Mamba for separation is the most efficient, and
Mamba for recognition is the least. Further, we show that Mamba is not more
efficient than transformer for speech shorter than the threshold duration and
performs worse in models that require joint modeling of text and speech, such
as cross or masked attention of two inputs. Therefore, we argue that the
superiority of Mamba or transformer depends on particular problems and models.
Code available at https://github.com/xi-j/Mamba-TasNet and
https://github.com/xi-j/Mamba-ASR.