Perceptually Accurate 3D Talking Head Generation: New Definitions,
  Speech-Mesh Representation, and Evaluation Metrics
Authors: name: Lee Chae-Yeon; name: Oh Hyun-Bin; name: Han EunGi; name: Kim Sung-Bin; name: Suekyeong Nam; name: Tae-Hyun Oh
Published: 2025-03-26 08:18:57

Abstract: Recent advancements in speech-driven 3D talking head generation have made
significant progress in lip synchronization. However, existing models still
struggle to capture the perceptual alignment between varying speech
characteristics and corresponding lip movements. In this work, we claim that
three criteria -- Temporal Synchronization, Lip Readability, and Expressiveness
-- are crucial for achieving perceptually accurate lip movements. Motivated by
our hypothesis that a desirable representation space exists to meet these three
criteria, we introduce a speech-mesh synchronized representation that captures
intricate correspondences between speech signals and 3D face meshes. We found
that our learned representation exhibits desirable characteristics, and we plug
it into existing models as a perceptual loss to better align lip movements to
the given speech. In addition, we utilize this representation as a perceptual
metric and introduce two other physically grounded lip synchronization metrics
to assess how well the generated 3D talking heads align with these three
criteria. Experiments show that training 3D talking head generation models with
our perceptual loss significantly improve all three aspects of perceptually
accurate lip synchronization. Codes and datasets are available at
https://perceptual-3d-talking-head.github.io/.