Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional
  Talking Portrait Generation
Authors: name: Weipeng Tan; name: Chuming Lin; name: Chengming Xu; name: FeiFan Xu; name: Xiaobin Hu; name: Xiaozhong Ji; name: Junwei Zhu; name: Chengjie Wang; name: Yanwei Fu
Published: 2025-04-25 05:28:21

Abstract: Recent advances in Talking Head Generation (THG) have achieved impressive lip
synchronization and visual quality through diffusion models; yet existing
methods struggle to generate emotionally expressive portraits while preserving
speaker identity. We identify three critical limitations in current emotional
talking head generation: insufficient utilization of audio's inherent emotional
cues, identity leakage in emotion representations, and isolated learning of
emotion correlations. To address these challenges, we propose a novel framework
dubbed as DICE-Talk, following the idea of disentangling identity with emotion,
and then cooperating emotions with similar characteristics. First, we develop a
disentangled emotion embedder that jointly models audio-visual emotional cues
through cross-modal attention, representing emotions as identity-agnostic
Gaussian distributions. Second, we introduce a correlation-enhanced emotion
conditioning module with learnable Emotion Banks that explicitly capture
inter-emotion relationships through vector quantization and attention-based
feature aggregation. Third, we design an emotion discrimination objective that
enforces affective consistency during the diffusion process through
latent-space classification. Extensive experiments on MEAD and HDTF datasets
demonstrate our method's superiority, outperforming state-of-the-art approaches
in emotion accuracy while maintaining competitive lip-sync performance.
Qualitative results and user studies further confirm our method's ability to
generate identity-preserving portraits with rich, correlated emotional
expressions that naturally adapt to unseen identities.