VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Authors: name: Chaoyou Fu; name: Haojia Lin; name: Xiong Wang; name: Yi-Fan Zhang; name: Yunhang Shen; name: Xiaoyu Liu; name: Yangze Li; name: Zuwei Long; name: Heting Gao; name: Ke Li; name: Xiawu Zheng; name: Rongrong Ji; name: Xing Sun; name: Caifeng Shan; name: Ran He
Published: 2025-01-03 18:59:52

Abstract: Recent Multimodal Large Language Models (MLLMs) have typically focused on
integrating visual and textual modalities, with less emphasis placed on the
role of speech in enhancing interaction. However, speech plays a crucial role
in multimodal dialogue systems, and implementing high-performance in both
vision and speech tasks remains a significant challenge due to the fundamental
modality differences. In this paper, we propose a carefully designed
multi-stage training methodology that progressively trains LLM to understand
both visual and speech information, ultimately enabling fluent vision and
speech interaction. Our approach not only preserves strong vision-language
capacity, but also enables efficient speech-to-speech dialogue capabilities
without separate ASR and TTS modules, significantly accelerating multimodal
end-to-end response speed. By comparing our method against state-of-the-art
counterparts across benchmarks for image, video, and speech tasks, we
demonstrate that our model is equipped with both strong visual and speech
capabilities, making near real-time vision and speech interaction.