DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework
  for Talking Head Video Generation
Authors: name: Hanbo Cheng; name: Limin Lin; name: Chenyu Liu; name: Pengcheng Xia; name: Pengfei Hu; name: Jiefeng Ma; name: Jun Du; name: Jia Pan
Published: 2024-10-17 16:32:36

Abstract: Talking head generation intends to produce vivid and realistic talking head
videos from a single portrait and speech audio clip. Although significant
progress has been made in diffusion-based talking head generation, almost all
methods rely on autoregressive strategies, which suffer from limited context
utilization beyond the current generation step, error accumulation, and slower
generation speed. To address these challenges, we present DAWN (Dynamic frame
Avatar With Non-autoregressive diffusion), a framework that enables all-at-once
generation of dynamic-length video sequences. Specifically, it consists of two
main components: (1) audio-driven holistic facial dynamics generation in the
latent motion space, and (2) audio-driven head pose and blink generation.
Extensive experiments demonstrate that our method generates authentic and vivid
videos with precise lip motions, and natural pose/blink movements.
Additionally, with a high generation speed, DAWN possesses strong extrapolation
capabilities, ensuring the stable production of high-quality long videos. These
results highlight the considerable promise and potential impact of DAWN in the
field of talking head video generation. Furthermore, we hope that DAWN sparks
further exploration of non-autoregressive approaches in diffusion models. Our
code will be publicly at https://github.com/Hanbo-Cheng/DAWN-pytorch.