ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually
  Synced Facial Performer
Authors: name: Jiazhi Guan, name: Zhiliang Xu, name: Hang Zhou, name: Kaisiyuan Wang, name: Shengyi He, name: Zhanwang Zhang, name: Borong Liang, name: Haocheng Feng, name: Errui Ding, name: Jingtuo Liu, name: Jingdong Wang, name: Youjian Zhao, name: Ziwei Liu

Lip-syncing videos with given audio is the foundation for various
applications including the creation of virtual presenters or performers. While
recent studies explore high-fidelity lip-sync with different techniques, their
task-orientated models either require long-term videos for clip-specific
training or retain visible artifacts. In this paper, we propose a unified and
effective framework ReSyncer, that synchronizes generalized audio-visual facial
information. The key design is revisiting and rewiring the Style-based
generator to efficiently adopt 3D facial dynamics predicted by a principled
style-injected Transformer. By simply re-configuring the information insertion
mechanisms within the noise and style space, our framework fuses motion and
appearance with unified training. Extensive experiments demonstrate that
ReSyncer not only produces high-fidelity lip-synced videos according to audio,
but also supports multiple appealing properties that are suitable for creating
virtual presenters and performers, including fast personalized fine-tuning,
video-driven lip-syncing, the transfer of speaking styles, and even face
swapping. Resources can be found at
https://guanjz20.github.io/projects/ReSyncer.