EMOVA: Empowering Language Models to See, Hear and Speak with Vivid
  Emotions
Authors: name: Kai Chen, name: Yunhao Gou, name: Runhui Huang, name: Zhili Liu, name: Daxin Tan, name: Jing Xu, name: Chunwei Wang, name: Yi Zhu, name: Yihan Zeng, name: Kuo Yang, name: Dingdong Wang, name: Kun Xiang, name: Haoyuan Li, name: Haoli Bai, name: Jianhua Han, name: Xiaohui Li, name: Weike Jin, name: Nian Xie, name: Yu Zhang, name: James T. Kwok, name: Hengshuang Zhao, name: Xiaodan Liang, name: Dit-Yan Yeung, name: Xiao Chen, name: Zhenguo Li, name: Wei Zhang, name: Qun Liu, name: Lanqing Hong, name: Lu Hou, name: Hang Xu

GPT-4o, an omni-modal model that enables vocal conversations with diverse
emotions and tones, marks a milestone for omni-modal foundation models.
However, empowering Large Language Models to perceive and generate images,
texts, and speeches end-to-end with publicly available data remains challenging
in the open-source community. Existing vision-language models rely on external
tools for the speech processing, while speech-language models still suffer from
limited or even without vision-understanding abilities. To address this gap, we
propose EMOVA (EMotionally Omni-present Voice Assistant), to enable Large
Language Models with end-to-end speech capabilities while maintaining the
leading vision-language performance. With a semantic-acoustic disentangled
speech tokenizer, we notice surprisingly that omni-modal alignment can further
enhance vision-language and speech abilities compared with the corresponding
bi-modal aligned counterparts. Moreover, a lightweight style module is proposed
for flexible speech style controls (e.g., emotions and pitches). For the first
time, EMOVA achieves state-of-the-art performance on both the vision-language
and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue
with vivid emotions.