MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
Authors: name: Qian Chen, name: Yafeng Chen, name: Yanni Chen, name: Mengzhe Chen, name: Yingda Chen, name: Chong Deng, name: Zhihao Du, name: Ruize Gao, name: Changfeng Gao, name: Zhifu Gao, name: Yabin Li, name: Xiang Lv, name: Jiaqing Liu, name: Haoneng Luo, name: Bin Ma, name: Chongjia Ni, name: Xian Shi, name: Jialong Tang, name: Hui Wang, name: Hao Wang, name: Wen Wang, name: Yuxuan Wang, name: Yunlan Xu, name: Fan Yu, name: Zhijie Yan, name: Yexin Yang, name: Baosong Yang, name: Xian Yang, name: Guanrou Yang, name: Tianyu Zhao, name: Qinglin Zhang, name: Shiliang Zhang, name: Nan Zhao, name: Pei Zhang, name: Chong Zhang, name: Jinren Zhou

Recent advancements in large language models (LLMs) and multimodal
speech-text models have laid the groundwork for seamless voice interactions,
enabling real-time, natural, and human-like conversations. Previous models for
voice interactions are categorized as native and aligned. Native models
integrate speech and text processing in one framework but struggle with issues
like differing sequence lengths and insufficient pre-training. Aligned models
maintain text LLM capabilities but are often limited by small datasets and a
narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal
Large Language Model with approximately 8B parameters for seamless voice
interaction. We address the main limitations of prior aligned multimodal
models. We train MinMo through multiple stages of speech-to-text alignment,
text-to-speech alignment, speech-to-speech alignment, and duplex interaction
alignment, on 1.4 million hours of diverse speech data and a broad range of
speech tasks. After the multi-stage training, MinMo achieves state-of-the-art
performance across various benchmarks for voice comprehension and generation
while maintaining the capabilities of text LLMs, and also facilitates
full-duplex conversation, that is, simultaneous two-way communication between
the user and the system. Moreover, we propose a novel and simple voice decoder
that outperforms prior models in voice generation. The enhanced
instruction-following capabilities of MinMo supports controlling speech
generation based on user instructions, with various nuances including emotions,
dialects, and speaking rates, and mimicking specific voices. For MinMo, the
speech-to-text latency is approximately 100ms, full-duplex latency is
approximately 600ms in theory and 800ms in practice. The MinMo project web page
is https://funaudiollm.github.io/minmo, and the code and models will be
released soon.