OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment
  across Language with Real-time Self-Aware Emotional Speech Synthesis
Authors: name: Run Luo; name: Ting-En Lin; name: Haonan Zhang; name: Yuchuan Wu; name: Xiong Liu; name: Min Yang; name: Yongbin Li; name: Longze Chen; name: Jiaming Li; name: Lei Zhang; name: Yangyi Chen; name: Hamid Alinejad-Rokny; name: Fei Huang
Published: 2025-01-08 15:18:09

Abstract: Recent advancements in omnimodal learning have been achieved in understanding
and generation across images, text, and speech, though mainly within
proprietary models. Limited omnimodal datasets and the inherent challenges
associated with real-time emotional speech generation have hindered open-source
progress. To address these issues, we propose openomni, a two-stage training
method combining omnimodal alignment and speech generation to develop a
state-of-the-art omnimodal large language model. In the alignment phase, a
pre-trained speech model is further trained on text-image tasks to generalize
from vision to speech in a (near) zero-shot manner, outperforming models
trained on tri-modal datasets. In the speech generation phase, a lightweight
decoder facilitates real-time emotional speech through training on speech tasks
and preference learning. Experiments demonstrate that openomni consistently
improves across omnimodal, vision-language, and speech-language evaluations,
enabling natural, emotion-rich dialogues and real-time emotional speech
generation.