VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video
  Face Swapping
Authors: name: Hao Shao; name: Shulun Wang; name: Yang Zhou; name: Guanglu Song; name: Dailan He; name: Shuo Qin; name: Zhuofan Zong; name: Bingqi Ma; name: Yu Liu; name: Hongsheng Li
Published: 2024-12-15 18:58:32

Abstract: Video face swapping is becoming increasingly popular across various
applications, yet existing methods primarily focus on static images and
struggle with video face swapping because of temporal consistency and complex
scenarios. In this paper, we present the first diffusion-based framework
specifically designed for video face swapping. Our approach introduces a novel
image-video hybrid training framework that leverages both abundant static image
data and temporal video sequences, addressing the inherent limitations of
video-only training. The framework incorporates a specially designed diffusion
model coupled with a VidFaceVAE that effectively processes both types of data
to better maintain temporal coherence of the generated videos. To further
disentangle identity and pose features, we construct the Attribute-Identity
Disentanglement Triplet (AIDT) Dataset, where each triplet has three face
images, with two images sharing the same pose and two sharing the same
identity. Enhanced with a comprehensive occlusion augmentation, this dataset
also improves robustness against occlusions. Additionally, we integrate 3D
reconstruction techniques as input conditioning to our network for handling
large pose variations. Extensive experiments demonstrate that our framework
achieves superior performance in identity preservation, temporal consistency,
and visual quality compared to existing methods, while requiring fewer
inference steps. Our approach effectively mitigates key challenges in video
face swapping, including temporal flickering, identity preservation, and
robustness to occlusions and pose variations.