EMO2: End-Effector Guided Audio-Driven Avatar Video Generation
Authors: name: Linrui Tian; name: Siqi Hu; name: Qi Wang; name: Bang Zhang; name: Liefeng Bo
Published: 2025-01-18 07:51:29

Abstract: In this paper, we propose a novel audio-driven talking head method capable of
simultaneously generating highly expressive facial expressions and hand
gestures. Unlike existing methods that focus on generating full-body or
half-body poses, we investigate the challenges of co-speech gesture generation
and identify the weak correspondence between audio features and full-body
gestures as a key limitation. To address this, we redefine the task as a
two-stage process. In the first stage, we generate hand poses directly from
audio input, leveraging the strong correlation between audio signals and hand
movements. In the second stage, we employ a diffusion model to synthesize video
frames, incorporating the hand poses generated in the first stage to produce
realistic facial expressions and body movements. Our experimental results
demonstrate that the proposed method outperforms state-of-the-art approaches,
such as CyberHost and Vlogger, in terms of both visual quality and
synchronization accuracy. This work provides a new perspective on audio-driven
gesture generation and a robust framework for creating expressive and natural
talking head animations.