UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified
  Model
Authors: name: Xiangyu Fan; name: Jiaqi Li; name: Zhiqian Lin; name: Weiye Xiao; name: Lei Yang
Published: 2024-08-01 17:59:27

Abstract: Audio-driven 3D facial animation aims to map input audio to realistic facial
motion. Despite significant progress, limitations arise from inconsistent 3D
annotations, restricting previous models to training on specific annotations
and thereby constraining the training scale. In this work, we present
UniTalker, a unified model featuring a multi-head architecture designed to
effectively leverage datasets with varied annotations. To enhance training
stability and ensure consistency among multi-head outputs, we employ three
training strategies, namely, PCA, model warm-up, and pivot identity embedding.
To expand the training scale and diversity, we assemble A2F-Bench, comprising
five publicly available datasets and three newly curated datasets. These
datasets contain a wide range of audio domains, covering multilingual speech
voices and songs, thereby scaling the training data from commonly employed
datasets, typically less than 1 hour, to 18.5 hours. With a single trained
UniTalker model, we achieve substantial lip vertex error reductions of 9.2% for
BIWI dataset and 13.7% for Vocaset. Additionally, the pre-trained UniTalker
exhibits promise as the foundation model for audio-driven facial animation
tasks. Fine-tuning the pre-trained UniTalker on seen datasets further enhances
performance on each dataset, with an average error reduction of 6.3% on
A2F-Bench. Moreover, fine-tuning UniTalker on an unseen dataset with only half
the data surpasses prior state-of-the-art models trained on the full dataset.
The code and dataset are available at the project page
https://github.com/X-niper/UniTalker.