EMOVA: Empowering Language Models to See, Hear and Speak with Vivid
  Emotions
Authors: name: Kai Chen; name: Yunhao Gou; name: Runhui Huang; name: Zhili Liu; name: Daxin Tan; name: Jing Xu; name: Chunwei Wang; name: Yi Zhu; name: Yihan Zeng; name: Kuo Yang; name: Dingdong Wang; name: Kun Xiang; name: Haoyuan Li; name: Haoli Bai; name: Jianhua Han; name: Xiaohui Li; name: Weike Jin; name: Nian Xie; name: Yu Zhang; name: James T. Kwok; name: Hengshuang Zhao; name: Xiaodan Liang; name: Dit-Yan Yeung; name: Xiao Chen; name: Zhenguo Li; name: Wei Zhang; name: Qun Liu; name: Lanqing Hong; name: Lu Hou; name: Hang Xu
Published: 2024-09-26 16:44:02

Abstract: GPT-4o, an omni-modal model that enables vocal conversations with diverse
emotions and tones, marks a milestone for omni-modal foundation models.
However, empowering Large Language Models to perceive and generate images,
texts, and speeches end-to-end with publicly available data remains challenging
in the open-source community. Existing vision-language models rely on external
tools for the speech processing, while speech-language models still suffer from
limited or even without vision-understanding abilities. To address this gap, we
propose EMOVA (EMotionally Omni-present Voice Assistant), to enable Large
Language Models with end-to-end speech capabilities while maintaining the
leading vision-language performance. With a semantic-acoustic disentangled
speech tokenizer, we notice surprisingly that omni-modal alignment can further
enhance vision-language and speech abilities compared with the corresponding
bi-modal aligned counterparts. Moreover, a lightweight style module is proposed
for flexible speech style controls (e.g., emotions and pitches). For the first
time, EMOVA achieves state-of-the-art performance on both the vision-language
and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue
with vivid emotions.