MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
Authors: name: Longtao Zheng; name: Yifan Zhang; name: Hanzhong Guo; name: Jiachun Pan; name: Zhenxiong Tan; name: Jiahao Lu; name: Chuanxin Tang; name: Bo An; name: Shuicheng Yan
Published: 2024-12-05 18:57:26

Abstract: Recent advances in video diffusion models have unlocked new potential for
realistic audio-driven talking video generation. However, achieving seamless
audio-lip synchronization, maintaining long-term identity consistency, and
producing natural, audio-aligned expressions in generated talking videos remain
significant challenges. To address these challenges, we propose Memory-guided
EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation
approach to generate identity-consistent and expressive talking videos. Our
approach is built around two key modules: (1) a memory-guided temporal module,
which enhances long-term identity consistency and motion smoothness by
developing memory states to store information from a longer past context to
guide temporal modeling via linear attention; and (2) an emotion-aware audio
module, which replaces traditional cross attention with multi-modal attention
to enhance audio-video interaction, while detecting emotions from audio to
refine facial expressions via emotion adaptive layer norm. Extensive
quantitative and qualitative results demonstrate that MEMO generates more
realistic talking videos across diverse image and audio types, outperforming
state-of-the-art methods in overall quality, audio-lip synchronization,
identity consistency, and expression-emotion alignment.