VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Authors: name: Chaoyou Fu, name: Haojia Lin, name: Xiong Wang, name: Yi-Fan Zhang, name: Yunhang Shen, name: Xiaoyu Liu, name: Yangze Li, name: Zuwei Long, name: Heting Gao, name: Ke Li, name: Xiawu Zheng, name: Rongrong Ji, name: Xing Sun, name: Caifeng Shan, name: Ran He

Recent Multimodal Large Language Models (MLLMs) have typically focused on
integrating visual and textual modalities, with less emphasis placed on the
role of speech in enhancing interaction. However, speech plays a crucial role
in multimodal dialogue systems, and implementing high-performance in both
vision and speech tasks remains a significant challenge due to the fundamental
modality differences. In this paper, we propose a carefully designed
multi-stage training methodology that progressively trains LLM to understand
both visual and speech information, ultimately enabling fluent vision and
speech interaction. Our approach not only preserves strong vision-language
capacity, but also enables efficient speech-to-speech dialogue capabilities
without separate ASR and TTS modules, significantly accelerating multimodal
end-to-end response speed. By comparing our method against state-of-the-art
counterparts across benchmarks for image, video, and speech tasks, we
demonstrate that our model is equipped with both strong visual and speech
capabilities, making near real-time vision and speech interaction.