<figure><img src="image_1.png" /></figure>

HAL
open science

# A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Louis Airale, Dominique Vaufreydaz, Xavier Alameda-Pineda

<figure><img src="image_2.png" /></figure>

## To cite this version:

Louis Airale, Dominique Vaufreydaz, Xavier Alameda-Pineda. A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation. 2023. hal-04149083

HAL Id: hal-04149083

https://hal.science/hal-04149083

Preprint submitted on 3 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_1.png" /></figure>

HAL
open science

# A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Louis Airale, Dominique Vaufreydaz, Xavier Alameda-Pineda

<figure><img src="image_2.png" /></figure>

## To cite this version:

Louis Airale, Dominique Vaufreydaz, Xavier Alameda-Pineda. A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation. 2023. hal-04149083v2

HAL Id: hal-04149083

https://hal.science/hal-04149083v2

Preprint submitted on 4 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Louis Airale

Dominique Vaufreydaz

Univ. Grenoble Alpes, CNRS

Univ. Grenoble Alpes, CNRS

Grenoble INP, LIG

Grenoble INP, LIG

38000 Grenoble

38000 Grenoble

France

France

Xavier Alameda-Pineda

Univ. Grenoble Alpes, Inria, CNRS

Grenoble INP, LJK

38000 Grenoble

France

## Abstract

animating still face images with deep generative models using a speech input signal is an active research topic and has seen important recent progress. However, much of the effort has been put into lip syncing and rendering quality while the generation of natural head motion, let alone the audio-visual correlation between head motion and speech, has often been neglected. In this work, we propose a multi-scale audio-visual synchrony loss and a multi-scale autoregressive GAN to better handle short and long-term correlation between speech and the dynamics of the head and lips. In particular, we train a stack of syncer models on multimodal input pyramids and use these models as guidance in a multi-scale generator network to produce audio-aligned motion unfolding over diverse time scales. Our generator operates in the facial landmark domain, which is a standard low-dimensional head representation. The experiments show significant improvements over the state of the art in head motion dynamics quality and in multi-scale audio-visual synchrony both in the landmark domain and in the image domain. Our code, models and demo will be made available on the project's GitHub page.<fnref n="1" />

## 1. Introduction

Among the many computer vision tasks that have benefited from the breakthrough of deep learning, talking face generation, that aims to animate still images from a condi-

tioning audio signal, has received considerable attention in the previous years. The advent of potent reenactment systems, as [29] or [40], and powerful loss functions allowing for a finer correlation between the generated lip motion and the audio input [8] have paved the way for a new state of the art. In both tasks of talking head generation and face reenactment, where lip and head motion are given as a driving video sequence, it is customary to represent face dynamics in a low dimensional space [12, 6, 50, 46, 44, 14, 40, 47, 24]. For this reason recent breakthrough in face reenactment has also benefited the talking head synthesis task. The above approach assumes that image texture and face dynamics can be processed independently, and that all necessary cues to handle the dynamics fit on a low dimensional manifold. It is then a reliable strategy to treat audio-conditioned talking face synthesis as a two-step procedure, where the audio-correlated dynamics are first generated in the intermediate space of an off-the-shelf face reenactment model, which is later used to reconstruct photorealistic video samples [37, 38, 15]. This allows to focus on improving the audio-visual (AV) correlation between the input speech signal and the produced face and lips movements in a much sparser space than that of real-world images.

Nevertheless, synthesising natural-looking head and lip motion sequences adequately correlated with an input audio signal remains a challenging task. In particular, although it has long been known that speech and head motion are tightly associated [41], only recently has this relation attracted the attention of the computer vision community. A likely reason for the difficulty of producing realistic head motion is the lack of an adequate loss function. So far, the

<footnote n="1">¹https://github.com/LouisBearing/HMo-audio.</footnote>
# A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Louis Airale

Univ. Grenoble Alpes, CNRS

Grenoble INP, LIG

38000 Grenoble, France

Dominique Vaufreydaz

Univ. Grenoble Alpes, CNRS

Grenoble INP, LIG

38000 Grenoble, France

Xavier Alameda-Pineda

Univ. Grenoble Alpes, Inria, CNRS

Grenoble INP, LJK

38000 Grenoble, France

## Abstract

animating still face images with deep generative models using a speech input signal is an active research topic and has seen important recent progress. However, much of the effort has been put into lip syncing and rendering quality while the generation of natural head motion, let alone the audio-visual correlation between head motion and speech, has often been neglected. In this work, we propose a multi-scale audio-visual synchrony loss and a multi-scale autoregressive GAN to better handle short and long-term correlation between speech and the dynamics of the head and lips. In particular, we train a stack of syncer models on multimodal input pyramids and use these models as guidance in a multi-scale generator network to produce audio-aligned motion unfolding over diverse time scales. Both the pyramid of audio-visual syncers and the generative models are trained in a low-dimensional space that fully preserves dynamics cues. The experiments show significant improvements over the state-of-the-art in head motion dynamics quality and especially in multi-scale audio-visual synchrony on a collection of benchmark datasets<fnref n="1" />

## 1. Introduction

The task of talking face generation, which aims to animate still images from a conditioning audio signal, has received considerable attention in the previous years. The advent of potent reenactment systems, as in *Siarohin et al.* [40] or *Wang et al.* [56], and powerful loss functions allowing for

a finer correlation between the generated lip motion and the audio input [7] have paved the way for a new state of the art. In both tasks of talking head generation and face reenactment, where lip and head motion are given as a driving video sequence, it is customary to represent face dynamics in a low dimensional space [14, 5, 73, 68, 64, 16, 56, 69, 32]. For this reason recent breakthroughs in face reenactment have also benefited the talking head synthesis task: the above approach assumes that image texture and face dynamics can be processed independently, and that all necessary cues to handle the dynamics fit on a low dimensional manifold. It is then a reliable strategy to treat audio-conditioned talking face synthesis as a two-step procedure, where the audio-correlated dynamics are first generated in the intermediate space of an off-the-shelf face reenactment model, which is later used to reconstruct photorealistic video samples [53, 54, 20]. This allows to focus on improving the audio-visual (AV) correlation between the input speech signal and the produced face and lips movements in a much sparser space than that of real-world images.

Nevertheless, synthesizing natural-looking head and lip motion sequences adequately correlated with an input audio signal remains a challenging task. In particular, although it has long been known that speech and head motion are tightly associated [60] (see also Section 4.3), only recently has this relation attracted the attention of the computer vision community. A likely reason for the difficulty of producing realistic head motion is the lack of an established adequate loss function. So far, the most successful strategy to produce synchronized lip movements has relied on the maximization of the cross-modal correlation between short audio and output motion clips, measured by a pre-trained

<footnote n="1">¹Code and demo available on github page: https://github.com/LouisBearing/HMo-audio.</footnote>
most successful strategy to produce synchronized lip movements has relied on the maximization of the cross-modal correlation between short audio and output motion clips, measured by a pre-trained model [8, 51, 27, 26, 43]. This fails, however, to account for lower frequency motion as that of the head which remains quasi-static over the short duration considered, typically of the order of a few hundreds of milliseconds. Surprisingly, there was no attempt to generalize this approach beyond lip synchronization. Neither has possible multi-scale audio-visual correlation been explored in the talking face generation literature. Head motion is often produced through the use of a separate sub-network trained to match the dynamics of a ground truth sequence, which in practice decouples the animation of head and lips.

We argue that to account for motion that unfolds over longer duration such as the head rhythm, a dedicated loss enforcing the synchrony of AV segments of various lengths is needed. We propose to implement this loss using a *pyramid of syncers*, replacing the lip-sync expert of [27] with a stack of *syncer models* evaluating the correlation between the audio input and the dynamics of the whole face over different time scales. One advantage of this syncer-pyramid loss function is that it allows to produce head and lip movements together; here one may train a single network end-to-end on the dynamics of both head and lips, resulting in overall lighter architecture and training procedure. A natural way to exploit the gradients from the multi-scale AV correlation loss is then to construct a similar hierarchical structure in the generative model itself. The proposed method, hereafter labelled MS-Sync, is implemented in the landmark domain [3]: for the reasons previously mentioned it is sufficient to parameterize the speech-correlated facial dynamics, which is the focus of the present work. Our generative model, loss functions and most of the metrics used to measure the quality and synchrony of the produced motion therefore apply in this domain. Although it is out of the scope of this study, several landmark-based real-world face reconstruction methods exist [45, 14, 47, 24]. Last, in contrast with the current trend in talking face synthesis, we rely on an autoregressive generative network for its inherent ability to model sequential dependencies, and its flexibility to handle sequences of arbitrary length. To do so, we build on the autoregressive Generative Adversarial Network (GAN) baseline of [2], and show that the conditioning speech signal has a stabilizing effect that hinders error accumulation on a much longer term than in the unconditional setting. In particular, we demonstrate experimentally that the error drift can be mitigated on test sequences more than five times the length of the training sequences. More importantly, we show that the proposed model, coupled with the multi-scale discriminator of [2], largely outperforms the state of the art in terms of *multi-scale audio-visual correlation* and head dynamics quality.

The contributions of the present work are:

* A multi-scale audio-visual correlation loss based on a pyramid of syncer networks,

* A multi-scale autoregressive GAN for the generation of speech-synchronized head and lip motion in the 2D-landmarks domain with minimal error accumulation,

* Extensive experiments on three datasets that show that our architecture outperforms previous works on all metrics related to both quality of head dynamics and AV correlation.

# 2. Related Work

## 2.1. Talking head generation

The task of animating a human face with a neural network can be either guided, when the head motion comes from a driving sequence, or unguided, in which case the head and lip motion must be inferred by the generative model from other modalities. Compelling results have been achieved over the years to improve the photorealistic rendering of guided methods [45, 29, 13, 28]. Among these, several works rely on low dimensional representations, e.g. facial landmarks [47, 24, 44], learned keypoints [29, 40], or morphable models [46] to handle the dynamics, which are later used to warp or normalize the style of the source identity image. On the other hand, the primary focus of audio-driven talking head synthesis has been on syncing output lip movements and input speech signal, either leaving visual reenactment as a separate task, or limiting it to static pose scenarios [34, 16, 32, 12, 31, 51, 48, 35, 11]. For this reason, there has been comparatively few endeavors to generate realistic head motion [4, 50]. As a noticeable improvement over previous research, recent works showed very promising results producing rich head motion in a low-dimensional keypoint space in combination with proficient visual reenactment systems [37, 38]. However there remains a margin of improvement in particular in the diversity of output head motion, and in the time alignment between speech, lips and head motion over different time scales, which has never been addressed before.

## 2.2. Learning to align speech and head dynamics

Two trends coexist regarding the syncing of audio and face dynamics. Originally, learning audio-correlated lip movements was only done with a mean squared error loss to the ground truth sequence [6, 9, 50, 42, 15, 30]. In parallel, following SyncNET [8], the use of contrastive loss variants turned out to be a strong alternative for its effectiveness on cross-modal training tasks [49, 36]. In particular, in [27] the authors proposed to train a *lip-sync expert* network to
<figure><img src="image_4.png" /><figcaption>Figure 1. Given a source image and a driving audio signal, our model generates a talking head video sequence correlated with the input speech over multiple time scales, ensuring accurate lips and head dynamics.</figcaption></figure>

model [7, 74, 36, 35, 62]. This fails, however, to account for lower frequency motion as that of the head which remains quasi-static over the short duration considered, typically of the order of a few hundreds of milliseconds. Surprisingly, there was no attempt to generalize this approach beyond lip synchronization. Neither has possible multi-scale audio-visual correlation been explored in the talking face generation literature. Head motion is often produced through the use of a separate sub-network trained to match the dynamics of a ground truth sequence, which in practice decouples the animation of head and lips.

We argue that to account for motion that unfolds over longer durations such as the head rhythm, a dedicated loss enforcing the synchrony of AV segments of various lengths is needed. We propose to implement this loss using a *pyramid of syncers*, replacing the lip-sync expert of Prajwal et al. [36] with a stack of syncer models evaluating the correlation between the audio input and the dynamics of the whole face over different time scales. How to achieve this however is not trivial, as simply increasing the length of the video segments does not guarantee that the expert model will focus on lower frequency motion. Applying a low-pass filtering on the video input, on the other end, would produce blurry and unusable results. Conversely, one may readily compute a multi-scale representation of motion by successively filtering a sequence of facial points coordinates. We take advantage of this observation in two different ways: 1) by using smoothed facial coordinate sequences as input to

the pyramid of syncers, 2) by producing facial coordinates as output from the generative model.

Concretely, we propose to construct Gaussian pyramids of head dynamics and audio on two sets of inputs: first, on paired samples from an audio-visual dataset for the training of the syncers, then on the generated dynamics and their corresponding driving audio signal during generative model training. The resulting *multi-scale audio-visual synchrony loss* hence represents a powerful means to enforce the correlation of both rigid, low-frequency and non-rigid, high-frequency generated motion with the input speech. Here, another advantage of operating on a low dimensional motion manifold is that the loss can be applied directly on the model's output without propagating through the visual rendering network, significant accelerating training. In addition, the multi-scale AV synchrony loss allows to produce head and lip movements with a single network, resulting in overall lighter architecture and training procedure compared to previous works that use an *ad hoc* network to model head movement.

To exploit the gradients from the multi-scale syncers, we build a hierarchical generative model, using a Feature Pyramid Network (FPN) [28] backbone. We use an autoregressive model for its flexibility to handle sequences of arbitrary length, and complement the AV synchrony loss with a window-based multi-scale discriminator architecture that proved to perform well on the generation of facial landmarks [2]. The resulting method, hereafter labeled MS-
regress the cross-modal alignment between short audio and video segments. The expert would later be frozen during the generative model training phase, and used as a loss function to enhance output audio-visual alignment. This strategy was later employed successfully in several works [43, 38, 26]. We argue however that the commonly used segment length of 200 ms is insufficient to properly align lower-frequency movements like that of the head, for which several such syncer networks operating on various segment lengths are required.

## 2.3. Multi-scale data processing

Learning on representations of the input data over multi-scale scales has become the standard in computer vision tasks such as object detection or semantic segmentation where objects of the same class can have different sizes [20, 33]. In the generative models literature, multi-scale approaches may either be implemented in the discriminator network of GANs as a way to improve multi-scale faithfulness of generated data [39, 21, 18] but also in the generative model itself [10, 17]. Although this was not explored so far in talking head generation, multi-scale feature hierarchies can be readily computed to align speech and dynamics of various motion frequencies.

## 3. Method

Given a set of initial landmark coordinates $x_0 \in \mathbb{R}^{2L}$ (the 2D coordinates of the $L = 68$ landmarks) and a conditioning audio signal $a_{0:T} = (a_0, \dots, a_T) \in \mathbb{R}^{d \times T}$ (here $d = 26$) over $T$ time steps, we aim to produce a sequence of landmark positions $x_{1:T}$ such that the joint distributions over generated and data samples match:

$$p_g(x_{0:T}, a_{0:T}) = p_{\text{data}}(x_{0:T}, a_{0:T}), \quad \forall x_{0:T}, a_{0:T}. \quad (1)$$

In this section we describe our procedure to tackle this problem as follows. We start by introducing in 3.1 the multi-scale AV synchrony loss which is the major contribution of the present work. Then in 3.2 we propose a multi-scale generator architecture able to exploit appropriately the devised multi-scale AV loss. Finally section 3.3 details our overall training procedure.

### 3.1. Multi-scale audio-visual synchrony loss

The most prominent procedure to align dynamics with speech input relies on the optimization of a correlation score computed on short audio-visual segments of the generated sequence using a pre-trained AV syncer network [27]. Several contrastive loss formulations are possible to train the syncer network, that suppose the maximization of the agreement between in-sync AV segments or positive pairs $(a_t, x_t)$ versus that of out-of-sync or negative pairs. One

particularly interesting formulation is the Info Noise Contrastive Estimation loss, that maximizes the mutual information between its two input modalities [25]. Given a set $X = (a_t, x_t, x_1^{\text{neg}}, \dots, x_N^{\text{neg}})$ containing a positive pair and $N$ negative position segments, this loss writes:

$$\mathcal{L}_{\text{InfoNCE}} = -\mathbb{E}_X \frac{e^{S(a_t, x_t)}}{e^{S(a_t, x_t)} + \sum_{n=1}^{N} e^{S(a_t, x_n^{\text{neg}})}}, \quad (2)$$

with $S$ the syncer model score function, which is hereafter implemented as the cosine similarity of the outputs from the audio and position embeddings $e_a$ and $e_x$ of the syncer network:

$$S(a_t, x_t) = \frac{e_a(a_t)^\top e_x(x_t)}{\|e_a(a_t)\| \cdot \|e_x(x_t)\|}. \quad (3)$$

Following the usual practice, we take $a_t$ and $x_t$ respectively as the MFCC spectrogram and position segment of a 200 ms window centered on time step $t$. Negative pairs can be indifferently misaligned audio and position segments from the same sequence sample, or segments from different samples, and $N$ is hereafter fixed to 48.

Once trained, the weights of $e_a$ and $e_x$ are frozen and the following term is added to the loss function of the generative model:

$$\mathcal{L}_{\text{AV}} = -\mathbb{E}_{a_t, x_t} S(a_t, x_t), \quad (4)$$

where $a_t$ is now part of the conditioning signal and $x_t$ is output by the model.

The above procedure is insufficient when one needs to discover AV correlations over different time scales. A straightforward extension consists in building multi-scale representations of the audio-visual inputs and training one syncer network $S^i$ for each level $i$ in the resulting pyramid. The training process of the pyramid of syncers is represented in Figure 1 (left). Specifically, the audio and landmark position hierarchies $\{a_{0:T/2^{i-1}}^i\}_i$ and $\{x_{0:T/2^{i-1}}^i\}_i$ are constructed by successive passes through an average pooling operator that blurs and downscales its input by a factor 2, e.g. for positions:

$$x_t^i = \frac{1}{2k+1} \sum_{\tau=-k}^{k} x_{2t+\tau}^{i-1} \quad (5)$$

where we choose $k = 3$. The objective is to progressively blur out the highest frequency motion when moving upward in the pyramid, forcing the top level syncers to exploit better the rhythm of the head motion. A total of four syncer networks are trained on the input pyramid following (2), input segment duration ranging from the standard 200 ms on the bottom level to 1600 ms at the coarsest scale.

After the training of the pyramid of syncers, all networks $S^1$ to $S^4$ are frozen and used to compute the multi-scale audio-visual synchrony loss. The principle of this loss is
Sync (for Multi-Scale Synchrony), produces head dynamics in a low dimensional motion space, namely that of 2d un-supervised keypoints. Videos are then reconstructed using an off-the-shelf, frozen reenactment model [40]. We would however like to point out that the multi-scale AV synchrony loss is a standalone contribution that can fuse in diverse generative model architecture, making it a versatile tool for audio-driven talking head generation.

The main contributions of the present work are summarized below.

* We train audio-visual syncer networks on rigid head motion, measuring for the first time, to the best of our knowledge, the correlation between head motion and speech,

* A multi-scale audio-visual synchrony loss to enable the generation of diverse audio-correlated facial dynamics,

* A multi-scale autoregressive GAN [13] framework, labelled MS-Sync, effective at producing speech-synchronized head and lips motion as shown in extensive experiments on four benchmark datasets.

# 2. Related Work

## 2.1. Talking Head Generation

The task of talking head generation consists of animating a source image or an initial face using a driving audio signal, and is especially difficult as it supposes to produce audio-synced head and lip motion while preserving high visual quality results. Hence trade-offs usually need to be done on one aspect or the other, or alternatively on the inference duration or the generalizability to unseen identities, and talking head generation methods therefore come in many different flavors.

An important part of the literature consists of identity-dependent methods, that typically require fine-tuning on a target identity at inference [61, 4, 66]. Those comprise pioneering research works [45, 21], but also many recent approaches based on the NeRF framework [33] that although providing outputs of compelling realism, require fine-tuning on video clips of the source subject [15, 38, 30, 59, 24]. Another class of high visual quality methods is based on diffusion models [42, 18, 10], that typically trade the generation of head pose for that of visual sharpness. Indeed head pose is either provided as a driving signal [39], or only weakly controlled with a mean-square-error (MSE) loss in the visual domain, giving less accurate audio correlation [44, 63].

A second, overlapping part of the literature focuses on the lip-syncing accuracy: head pose is either extracted from a driving video sequence, possibly with the lip region

masked out, or it is simply omitted [11, 43, 74, 5, 71, 49, 9, 20, 41, 12, 26, 57, 70, 47, 52, 19, 58]. These methods provide strong audio-correlation baselines, and a typical strategy is to use the output of a frozen lip-sync model to improve the synchrony between output lips and input speech signal [72, 25, 50, 62, 35, 51].

Finally, close to the proposed approach are one-shot methods that reenact single source images of unseen identities in real time, with an explicit treatment of head motion [73, 53, 68, 31]. Although successful attempts have been made to leverage pre-trained syncer models for precise lip-syncing [54, 67], head pose is still usually learned through the minimization of a MSE loss that fails to explicitly account for the correlation between speech and rigid head motion. Besides, we argue that the duration of the audio segments commonly used for the synchronization of the lips is insufficient to properly align lower-frequency movements like that of the head with the driving audio signal, advocating for novel approaches.

## 2.2. Video-Driven Face Reenactment

The task of animating a source human face with a neural network can also be fully guided by a driving sequence of a target identity that provides the supervision for head and lip motion [65, 16, 69, 32]. Compelling results have been achieved over the years to bridge the identity gap between source and target images and hallucinate unseen poses of the source identity [40, 15, 37]. These works rely on low dimensional representations, e.g. facial landmarks [69, 32, 64] or learned keypoints [40, 56] to measure the deformation from a given target image to the source image, which is later used to warp or normalize the style of the source identity image. Following previous works [53, 54], we train a generative model to predict the speech-conditioned driving motion sequence, and use the reenactment model of Siarohin et al. [40] to reconstruct the output videos.

## 2.3. Multi-scale Data Processing

Learning on representations of the input data over multiple time or spatial scales has become the standard in computer vision tasks such as object detection or semantic segmentation where objects of the same class can have different sizes [28, 46]. In the generative models literature, multi-scale approaches may either be implemented in the discriminator network of GANs as a way to improve multi-scale faithfulness of generated data [55, 29, 23] but also in the generative model itself [8, 22]. Although this was not explored so far in talking head generation, multi-scale feature hierarchies can be readily computed to align speech and dynamics of various motion frequencies.
<figure><img src="image_4.png" /><figcaption>Figure 1. Left. A stack of syncer networks $S^i$ are trained on multi-scale positive and negative multimodal pairs using contrastive losses. Right. The syncer models are frozen and used to compute the multi-scale audio visual synchrony loss of the generative model.</figcaption></figure>

presented in Figure 1 (right). Similar to the input pyramids used to train the syncer networks, we construct a multi-scale representation of the input speech $a_{0:T}$ and the generated landmark positions $x_{0:T}$. Then for each hierarchy level $i$ one loss term $\mathcal{L}_{AV}^i$ is computed according to (4) using pre-trained syncer $S^i$. Those terms are then averaged to give the overall multi-scale AV synchrony loss $\mathcal{L}_{AV}^{MS}$. To better exploit the effects of this loss, we propose a multi-scale autoregressive generator network that we describe in the following section.

## 3.2. Multi-scale autoregressive generator

Through the multi-scale synchrony loss, the generator receives gradients that push it to produce audio-synced landmark positions over multiple time scales. In this section, we describe the architecture of our generator network, which is itself implemented with a multi-scale structure to better exploit the loss gradients. The overall architecture is described in Figure 2.

The proposed generative model is inspired from SUHMo [2] which implements an autoregressive model to generate facial landmark velocities. This however requires substantial adaptations to deal with the present multimodal data. Very generally, given landmark positions $x_{0:t}$ until time step $t$ and next frame audio input $a_{t+1}$, the generator $G$ produces instantaneous velocities $v_{t+1}$:

$$v_{t+1} = G(x_{0:t}, a_{t+1}), \quad (6)$$

$$x_{t+1} = x_t + v_{t+1}. \quad (7)$$

As depicted in Figure 2, $G$ is constituted of a temporal module operating on a sequence of landmark positions, and of a multi-scale module that takes the output of the temporal module $h_t$, the positions $x_t$ and audio $a_{t+1}$ as input to produce $v_{t+1}$. We implement the multi-scale module as the bottom-up path of a Feature Pyramid Network [20]. Namely, the input spectrogram is processed by

several downsampling convolutional layers, producing feature maps $a_{0:T}^1$ to $a_{0:T/2^3}^4$ of the same resolution as those used to compute the AV loss pyramid. Feature maps 2 to 4 are later interpolated back to the length $T$ of the finest map, such that one vector $a_{t+1}^i$ can be extracted from each pyramid level $i$ to produce the next step velocity. Concretely, each vector $a_{t+1}^i$ is concatenated with $x_t$ and $h_t$ and is processed by an independent fully connected branch, the ratio-nale being that processing each input resolution separately would allow the model to produce different motion frequencies.

The outputs of the four branches of the multi-scale generator are merged using a learnable soft spatial mask. Each branch $i$ outputs a velocity vector $v^i \in \mathbb{R}^{2L}$ (note that time index is omitted for the sake of clarity) and a mask vector $w^i \in \mathbb{R}^{2L}$ such that $w_{2k-1}^i = w_{2k}^i, \forall k \le L$, responsible for enhancing or weakening the contribution of each landmark on the given branch. This is because we expect facial regions to play different roles depending on the scale: the finest resolution branch might emphasize lip landmarks, while at the coarsest scale, more weight may be put on head contour. The output of the multi-scale module finally writes:

$$v_{t+1} = \sum_{i=1}^{4} \left( \frac{e^{w^i}}{\sum_{j} e^{w^j}} \right) v^i \quad (8)$$

## 3.3. Overall architecture and training

In addition to the AV synchrony loss $\mathcal{L}_{AV}^{MS}$, we make use of the discriminator networks proposed in [2], that proved effective to train an autoregressive generator on landmark sequences. These consist in one frame discriminator $D_f$ which computes the realism of static landmarks, and two window-based multi-scale networks $D_s$ and $D_s^j$ on sequences. The difference between those is that $D_s^j$ processes pairs of samples to help reducing mode collapse [22]. Although in our audio-conditioned setting mode collapse is at most a minor issue, we found that using this additional
<figure><img src="image_5.png" /><figcaption>Figure 2. Left. A stack of syncer networks $S^i$ are trained on multi-scale positive and negative multimodal pairs using contrastive losses. Right. The syncer models are frozen and used to compute the multi-scale audio-visual synchrony loss of the generative model.</figcaption></figure>

## 3. Method

This section introduces the problem formulation and associated notations. We follow the conventions from Siarohin et al. [40] and represent the dynamics using a set of 10 2d keypoints coordinates, together with two-by-two Jacobian matrices that provide finer details of the spatial deformation around the keypoints. In the following the overall inputs are simply referred to as keypoints, of total dimension 60. Given a set of initial keypoints coordinates $x_0 \in \mathbb{R}^{60}$ and a conditioning audio signal $a_{0:T} = (a_0, \dots, a_T) \in \mathbb{R}^{d \times T}$ (here $d = 26$) over $T$ time steps, we aim to produce a sequence of keypoint positions $x_{1:T}$ such that the joint distributions over generated and data samples match:

$$p_g(x_{0:T}, a_{0:T}) = p_{\text{data}}(x_{0:T}, a_{0:T}), \quad \forall x_{0:T}, a_{0:T}. \quad (1)$$

The procedure to tackle this problem is as follows. The multi-scale AV synchrony loss, which is the major contribution of this work, is first introduced in section 3.1. Then a multi-scale generator architecture able to exploit appropriately the devised multi-scale AV loss is developed in section 3.2. Finally the overall training procedure is detailed in section 3.3.

### 3.1. Multi-scale Audio-Visual Synchrony Loss

The most prominent procedure to align dynamics with speech input relies on the optimization of a correlation score computed on short audio-visual segments of the generated sequence using a pre-trained AV syncer network [36]. Several contrastive loss formulations are possible to train

the syncer network, that suppose the maximization of the agreement between in-sync AV segments or positive pairs $(a_t, x_t)$ versus that of out-of-sync or negative pairs. One particularly interesting formulation is the Info Noise Contrastive Estimation loss, which maximizes the mutual information between its two input modalities [34]. Given a set $X = (a_t, x_t, x_1^{\text{neg}}, \dots, x_N^{\text{neg}})$ containing a positive pair and $N$ negative position segments, this loss writes:

$$\mathcal{L}_{\text{InfoNCE}} = -\mathbb{E}_X \frac{e^{S(a_t, x_t)}}{e^{S(a_t, x_t)} + \sum_{n=1}^{N} e^{S(a_t, x_n^{\text{neg}})}}, \quad (2)$$

with $S$ the syncer model score function, which is classically implemented hereafter as the cosine similarity of the outputs from an audio and a position embeddings $e_a$ and $e_x$:

$$S(a_t, x_t) = \frac{e_a(a_t)^\top e_x(x_t)}{\|e_a(a_t)\| \cdot \|e_x(x_t)\|}. \quad (3)$$

Following the usual practice, $a_t$ and $x_t$ are respectively taken as the MFCC spectrogram and position segment of a 200 ms window centered on time step $t$. Negative pairs can be misaligned audio and position segments from the same audio-visual sequence and therefore constitute hard negatives, or can alternatively be segments from different samples, e.g. when an insufficient number $N$ of hard negatives can be sampled.

Once trained, the weights of $e_a$ and $e_x$ are frozen and the following term is added to the loss function of the generative model:

$$\mathcal{L}_{\text{AV}} = -\mathbb{E}_t S(a_t, x_t), \quad (4)$$
<figure><img src="image_5.png" /><figcaption>Figure 2. Left. Our network is composed of a temporal module and a multi-scale module. Right. Details of the multi-scale module.</figcaption></figure>

<table><caption>Table 1. Training dataset and head motion preprocessing and generation for the considered methods. The preservation of head translation typically implies re-creating the datasets from the original sources following the strategy in [29].</caption><thead><tr><th>Method</th><th>Train Dataset</th><th>Head transl. in prepro.</th><th>Predicted head motion</th></tr></thead><tbody><tr><td>Wav2Lip [27]</td><td>LRS2</td><td>×</td><td>×</td></tr><tr><td>PC-AVS [49]</td><td>VoxCeleb2 (I)</td><td>×</td><td>×</td></tr><tr><td>MakeItTalk [50]</td><td>VoxCeleb2 (I)</td><td>×</td><td>✓</td></tr><tr><td>Audio2Head [37]</td><td>VoxCeleb2 (II)</td><td>✓</td><td>✓</td></tr><tr><td>OSTF [38]</td><td>Obama Weekly Address</td><td>✓</td><td>✓</td></tr><tr><td>MS-Sync (ours)</td><td>VoxCeleb2 (II)</td><td>✓</td><td>✓</td></tr></tbody></table>

loss slightly improves the dynamics quality. Adversarial losses are implemented with the geometric GAN formulation of [19]. Namely, given the generated and ground truth landmark position distributions $p_g$ and $p_{\text{data}}$, the generator losses write:

$$ \mathcal{L}_{G_f} = -\mathbb{E}_{x_{0:T} \sim p_g} \left[ \frac{1}{T} \sum_{t \ge 1} D_f(x_t) \right], \quad (9) $$

$$ \mathcal{L}_{G_s} = -\mathbb{E}_{x_{0:T} \sim p_g} [D_s(x_{0:T})], \quad (10) $$

$$ \mathcal{L}_{G_s^j} = -\mathbb{E}_{x_{0:T} \sim p_g, x'_{0:T} \sim p_g} [D_s^j(x_{0:T}, x'_{0:T})], \quad (11) $$

as for the generic discriminator loss:

$$ \mathcal{L}_{D_*} = \mathbb{E}_{x \sim p_g} [\max(0, 1 + D_*(x))] \\ + \mathbb{E}_{x \sim p_{\text{data}}} [\max(0, 1 - D_*(x))], \quad (12) $$

where $D_*$ is replaced respectively by $D_f$, $D_s$ and $D_s^j$ and sequences are sampled according to equations 9 to 11. Additionally, we trained with the weak $L_2$ reconstruction loss in [2] but found no significant improvement. Overall, training consists in minimizing alternatively the two following

terms:

$$ \mathcal{L}_D = \mathcal{L}_{D_f} + \mathcal{L}_{D_s} + \mathcal{L}_{D_s^j}, \quad (13) $$

$$ \mathcal{L} = \lambda \mathcal{L}_{\text{AV}}^{\text{MS}} + \mathcal{L}_{G_f} + \mathcal{L}_{G_s} + \mathcal{L}_{G_s^j}, \quad (14) $$

with $\lambda = 8$ in all experiments.

## 4. Experiments

We conducted three benchmark evaluations to measure the proficiency of our model, assessing respectively head dynamics quality, multi-scale AV synchrony in the landmark domain, and AV synchrony in the image domain.

### 4.1. Experimental protocol

**Datasets.** Experiments are conducted on two versions of the VoxCeleb2 dataset [7] with different preprocessing. The first version, labelled VoxCeleb2 (I), follows the standard preprocessing that centers the face in every frame. Second, we use the preprocessing strategy in [29] to re-generate subsets of respectively ~18k and 500 short video clips from the original VoxCeleb2 train and test sets. The interest of this preprocessing method is that it keeps the reference frames fixed, thus preserving head motion. We refer to this second version as VoxCeleb2 (II). HDTF dataset [46] contains ~400 long duration frontal-view videos from political addresses, which despite limited dynamics diversity makes it suitable for AV correlation measurements. Last, we use LRS2 [1], which is preprocessed similarly to VoxCeleb2 (I), to measure the AV synchrony in the image space.

**Benchmark models.** We compare our method, MS-Sync, with the following prominent speech-driven talking head generation models. Wav2Lip [27] uses a pre-trained lip syncer to learn the AV synchrony, and achieved state-of-the-art performances on the visual dubbing task. However,
where $a_t$ is now part of the conditioning signal and $x_t$ is output by the model.

The above procedure is insufficient when one needs to discover AV correlations over different time scales. One solution consists in building multi-scale representations of the audio-visual inputs and training one syncer network $S^i$ for each level $i$ in the resulting pyramid. Here the use of keypoints instead of the raw video is crucial for two reasons: they can be easily downscaled by successive low-pass filtering operations, and it avoids propagating through the visual reenactment model, substantially saving computation. The training process of the pyramid of syncers is represented in fig. 2 (left). Specifically, audio and keypoint coordinates pyramids $\{a_{0:T/2^{i-1}}^i\}_i$ and $\{x_{0:T/2^{i-1}}^i\}_i$ are constructed by successive passes through an average pooling operator that blurs and downscales its input by a factor 2, e.g. for positions:

$$x_t^i = \frac{1}{2k+1} \sum_{\tau=-k}^{k} x_{2t+\tau}^{i-1} \quad (5)$$

where we choose $k=3$. The objective is to progressively blur out the highest frequency motion when moving upward in the pyramid, forcing the top level syncers to exploit better the rhythm of the head motion. A total of four syncer networks are trained on the input pyramid following (2), input segment duration ranging from the standard 200 ms on the bottom level to 1600 ms at the coarsest scale.

After the training of the pyramid of syncers, all networks $S^1$ to $S^4$ are frozen and used to compute the multi-scale audio-visual synchrony loss. The principle of this loss is presented in fig. 2. Similar to the input pyramids used to train the syncer networks, we construct a multi-scale representation of the input speech $a_{0:T}$ and the generated keypoints positions $x_{0:T}$. Then for each hierarchy level $i$ one loss term $\mathcal{L}_{AV}^i$ is computed according to (4) using pretrained syncer $S^i$. Those terms are then summed to give the overall multi-scale AV synchrony loss $\mathcal{L}_{AV}^{MS}$:

$$\mathcal{L}_{AV}^{MS} = \sum_{i} \mathcal{L}_{AV}^{i}(x_{0:T/2^{i-1}}^{i}, a_{0:T/2^{i-1}}^{i}), \quad (6)$$

$$\text{with } \mathcal{L}_{AV}^{i}(x_{0:T/2^{i-1}}^{i}, a_{0:T/2^{i-1}}^{i}) = -\mathbb{E}_{t}[S^{i}(a_{t}^{i}, x_{t}^{i})] \quad (7)$$

To better exploit the effects of this loss, we propose a multi-scale autoregressive generator network which is described in the following section.

## 3.2. Multi-scale Autoregressive Generator

Through the multi-scale synchrony loss, the generator receives gradients that push it to produce audio-synced keypoint positions over multiple time scales. In this section, we describe the architecture of the generator network, which is itself implemented with a multi-scale structure to allow distinct loss terms to act preferentially on different layers of the network. The overall architecture is depicted in fig. 3.

A residual autoregressive formulation is employed for the generative model, such that given keypoints positions $x_{0:t}$ up to time step $t$ and the audio input $a_{t+1}$ for the next time step, the generator $G$ produces instantaneous velocities $v_{t+1}$:

$$v_{t+1} = G(x_{0:t}, a_{t+1}), \quad (8)$$

$$x_{t+1} = x_t + v_{t+1}. \quad (9)$$

As depicted in fig. 3, $G$ contains a temporal module that operates on a sequence of keypoint positions, and a multi-scale module that takes the output of the temporal module $h_t$, the positions $x_t$ and audio $a_{t+1}$ as input to produce $v_{t+1}$. A Transformer encoder [48] is used for the temporal module, and the multi-scale module is implemented as the bottom-up path of a Feature Pyramid Network [28]. Namely, the input spectrogram is processed by several downsampling convolutional layers, producing feature maps $a_{0:T}^i$ to $a_{0:T/2^3}^i$ of the same resolution as those used to compute the pyramids for the audio-visual synchrony loss. Feature maps 2 to 4 are later interpolated back to the length $T$ of the finest map, such that one vector $a_{t+1}^i$ can be extracted from each pyramid level $i$ to produce the next step velocity. Concretely, each vector $a_{t+1}^i$ is concatenated with $x_t$ and $h_t$ and is processed by an independent fully connected branch, the rationale being that processing each input resolution separately would allow the model to produce different motion frequencies.

The outputs of the four branches of the multi-scale generator are merged using a learnable soft spatial mask. Each branch $i$ outputs a velocity vector $v^i \in \mathbb{R}^{10 \times 6}$ (recall that there are both 2d coordinates and 4d Jacobian matrices, and time index is omitted for the sake of clarity) and a mask vector $w^i \in \mathbb{R}^{10 \times 6}$ such that $w^i_{j,k} = w^i_{j,l}$ for all $j, k$ and $l$ (or one mask value for all 6 coordinates of a given keypoint), responsible for enhancing or weakening the contribution of each keypoint on the given branch. This is because facial regions are expected to play different roles depending on the scale: the finest resolution branch might emphasize lip keypoints, while at the coarsest scale, more weight may be put on rigid head motion. The output of the multi-scale module finally writes:

$$v_{t+1} = \sum_{i=1}^{4} \left( \frac{e^{w^i}}{\sum_{j} e^{w^j}} \right) v^i \quad (10)$$

## 3.3. Overall Architecture and Training

The audio-visual synchrony loss $\mathcal{L}_{AV}^{MS}$ is complemented with two discriminator networks to improve the static and dynamic quality of the generated keypoints. One frame discriminator $D_f$ computes the realism of static keypoints, and a window-based multi-scale network $D_s$ computes that of keypoint sequences [2]. Adversarial losses are implemented
<table><caption>Table 2. Dynamics quality of rasterized landmarks on VoxCeleb2 (II) test set. All metrics need to be minimized. Bold indicates best score, underline second best. We additionally report the static face Wav2Lip results as reference scores for the reader.</caption><thead><tr><th rowspan="2">Duration (frames)</th><th colspan="3">40</th><th colspan="5">80</th><th colspan="3">200</th></tr><tr><th>FVD<sub>40</sub></th><th>FID</th><th>t-FID<sub>40</sub></th><th>FVD<sub>40</sub></th><th>FID</th><th>t-FID<sub>40</sub></th><th>FVD<sub>80</sub></th><th>t-FID<sub>80</sub></th><th>FVD<sub>40</sub></th><th>FID</th><th>t-FID<sub>40</sub></th></tr></thead><tbody><tr><td colspan="12"><em>Methods that predict head motion</em></td></tr><tr><td>MakeItTalk [50]</td><td>236</td><td>4.0</td><td>107</td><td>234</td><td><u>3.2</u></td><td>101</td><td>476</td><td>133</td><td><u>224</u></td><td><u>4.2</u></td><td>114</td></tr><tr><td>Audio2Head [37]</td><td>406</td><td>66.4</td><td>109</td><td>593</td><td>82.5</td><td>133</td><td>682</td><td>149</td><td>649</td><td>92.5</td><td>141</td></tr><tr><td>OSTF [38]</td><td><u>113</u></td><td>12.4</td><td><b>36</b></td><td>164</td><td>25.4</td><td>50</td><td><u>249</u></td><td><u>33</u></td><td>225</td><td>30.5</td><td><u>65</u></td></tr><tr><td>MS-Sync-short</td><td><b>105</b></td><td><u>3.3</u></td><td>52</td><td><u>126</u></td><td>5.5</td><td><u>47</u></td><td><b>239</b></td><td>41</td><td>279</td><td>25.2</td><td>68</td></tr><tr><td>MS-Sync-long</td><td>134</td><td>6.6</td><td><u>42</u></td><td><b>104</b></td><td>6.8</td><td><b>39</b></td><td>257</td><td><b>28</b></td><td><b>144</b></td><td>8.3</td><td><b>47</b></td></tr><tr><td colspan="12"><em>Methods with fixed head pose</em></td></tr><tr><td>Wav2Lip [27]</td><td>263</td><td><b>0.9</b></td><td>167</td><td>261</td><td><b>1.0</b></td><td>167</td><td>557</td><td>188</td><td>265</td><td><b>3.2</b></td><td>182</td></tr></tbody></table>

it only reenacts the lip region and therefore does not produce any head motion. Similarly, PC-AVS [49] produces speech-synchronized talking head videos using a driving head motion sequence, mapping the results directly in image space without any explicit intermediate representation. MakeItTalk [50] was one of the first successful attempts to produce speech-correlated head motion. Its dynamics are learned in the landmark domain on VoxCeleb2 (I), i.e. no head translation was seen at training. Audio2Head [37] and its follow-up model OSTF [38] propose methods to generate vivid dynamics, learning head motion and AV synchrony in a sparse keypoint space using a two-step training procedure. Audio2Head dynamics module is trained on VoxCeleb2 (II), while OSTF is trained on a single identity, namely using Obama Weekly Address dataset. As a noticeable improvement over Audio2Head, in OSTF AV synchrony is controlled with the contrastive loss of [27]. An overview of the different models can be found in Table 1.

**Training details.** The temporal module introduced in Section 3 is implemented as a 1-layer LSTM with hidden size 256. All convolutions and fully connected layers are implemented as 1D ConvNeXt blocks (with kernel size 1 for dense layers) [23]. We trained two versions of our model, varying only the training sequence length from 40 to 80 frames, resulting in MS-Sync-short and MS-Sync-long: the aim is to see how this affects the quality of the produced sequences on various output lengths. Models were trained on VoxCeleb2 (II) for 70k iterations (about 500 epochs) using Adam optimizers with $\beta_1 = 0$ and $\beta_2 = 0.999$ and learning rates $2 \times 10^{-5}$ and $1 \times 10^{-5}$ respectively for the generator and the discriminator, after which a decay factor of 0.1 was applied on the learning rates for 5k additional iterations. All audio inputs are sampled at 16 kHz, and to generate the 26-dimensional MFCC spectrogram we used a window size of 400 and hop size of 160.

<figure><img src="image_6.png" /><figcaption>Figure 3. Distributions of confidence ($\text{AV-Conf}_1$) and offset ($\text{AV-Off}_1$, without absolute value) scores measured on VoxCeleb2 (II) test dataset by our metric syncer (equivalent to landmark-domain SyncNET [8]) at the finest time scale.</figcaption></figure>

## 4.2. Dynamics quality

**Protocol.** The quality of the produced dynamics is evaluated on the 500 videos of VoxCeleb2 (II) test set, which preserve head motion. The Fréchet Inception Distance (FID) is used to measure static face realism, while the Fréchet Video Distance (FVD) and temporal FID [2] metrics measure the distance between the distributions of data and generated motion. The two latter metrics require a fixed sequence length that we set to either 40 or 80 frames (equivalent to 1.6 s and 3.2 s at 25 fps), and we refer to the resulting metrics as FVD$_{40}$ (t-FID$_{40}$) and FVD$_{80}$ (t-FID$_{80}$), respectively. When generating longer sequences, we measure the FVD$_{40}$, t-FID$_{40}$ and FID on the last 40 frames. Image-rasterized landmarks are used to compute the metrics (see Figure 4 and [2]).

**Results.** The results of the dynamics quality evaluations are reported in Table 2. MS-Sync-short shows similar FVD and t-FID scores to OSTF but significantly better FID, especially on 40 and 80 frame sequences. Since the faces produced by OSTF are also very sharp, we interpret this result
<figure><img src="image_6.png" /><figcaption>**Figure 3.** **Left.** Our network is composed of an autoregressive Transformer temporal module and a convolutional multi-scale module. **Right.** Details of the multi-scale module.</figcaption></figure>

with the geometric GAN formulation of Lim & Ye [27]. Namely, given the generated and ground truth keypoint position distributions $p_g$ and $p_{\text{data}}$, the generator losses write:

$$ \mathcal{L}_{G_f} = -\mathbb{E}_{x_{0:T} \sim p_g} \left[ \frac{1}{T} \sum_{t \ge 1} D_f(x_t) \right], \quad (11) $$

$$ \mathcal{L}_{G_s} = -\mathbb{E}_{x_{0:T} \sim p_g} [D_s(x_{0:T})], \quad (12) $$

as for the generic discriminator loss:

$$ \mathcal{L}_{D_*} = \mathbb{E}_{x \sim p_g} [\max(0, 1 + D_*(x))] + \mathbb{E}_{x \sim p_{\text{data}}} [\max(0, 1 - D_*(x))], \quad (13) $$

where $D_*$ is replaced respectively by $D_f$ and $D_s$. A $L_2$ reconstruction loss to the ground truth keypoints, $\mathcal{L}_{\text{rec}}$, is finally added to the loss function:

$$ \mathcal{L}_{\text{rec}} = \mathbb{E}_{(x_{0:T}, a_{0:T}) \sim p_{\text{data}}} \|x_{0:T} - G(x_0, a_{0:T})\|^2 \quad (14) $$

The balance between loss terms is achieved by the use of weighting factors $\lambda_{\text{av}}$, $\lambda_{\text{adv}}$ and $\lambda_{\text{rec}}$ (with $\lambda_{\text{av}} = 8$, $\lambda_{\text{adv}} = 0.1$ and $\lambda_{\text{rec}} = 1$ providing the best results), such that the overall training consists in minimizing alternatively the two following terms:

$$ \mathcal{L}_D = \mathcal{L}_{D_f} + \mathcal{L}_{D_s} \quad (15) $$

$$ \mathcal{L} = \lambda_{\text{av}} \mathcal{L}_{\text{AV}}^{\text{MS}} + \lambda_{\text{adv}} (\mathcal{L}_{G_f} + \mathcal{L}_{G_s}) + \lambda_{\text{rec}} \mathcal{L}_{\text{rec}}. \quad (16) $$

## 4. Experiments

We conduct benchmark evaluations to measure the proficiency of our method on visual audio-visual synchrony, landmark-domain multi-scale audio-visual synchrony and output visual quality. We also carry an ablation study to investigate the contribution of the different terms of the loss function, including the multi-scale AV loss.

### 4.1. Experimental protocol

**Datasets.** Experiments are conducted on the VoxCeleb2 dataset [6] with two different preprocessings. First, we use the standard test set of VoxCeleb2, hereafter VoxCeleb2 (I), containing ~2000 short audio-visual clips centered on subject faces. Second, following the preprocessing strategy of Siarohin et al. [40], subsets of respectively ~18k and 500 short video clips from the original VoxCeleb2 train and test sets are generated, the former for the training of our model, and the latter for the evaluation. The interest of this preprocessing is that it keeps the reference frames fixed, thus preserving head motion. In the following section, this second dataset is referred to as VoxCeleb2 (II). Evaluations are also conducted on the HDTF dataset [68], which contains ~400 long duration frontal-view videos from political addresses, where head motion is also preserved but span a rather narrow distribution. Last, we use LRS2 [1], which is preprocessed similarly to VoxCeleb2 (I), to measure the audio-visual synchrony in the visual domain.

**Benchmark Models.** We compare our method, MS-Sync, with other one-shot talking head generation models. Wav2Lip [36] uses a pre-trained lip syncer to learn the AV synchrony, and achieved state-of-the-art performances on the visual dubbing task. However, it only reenacts the lip region and therefore does not produce any head motion. IP_LAP [70] does not produce head motion either, contrary to PC-AVS [72] and EAMM [20], albeit being rather limited. On the other hand, MakeItTalk [73], Audio2Head [53] and its follow-up work [54], and SadTalker [67] output head poses. Noticeably, the two methods presented in Wang et al. [53] and Wang et al. [54] rely on the same facial key-
<table><caption>Table 3. Landmark domain multi-scale AV synchrony on VoxCeleb2 (II) test set. A separate syncer model inspired from SyncNet is trained in the landmark domain for each corresponding time scale and used to compute the correlation scores.</caption><thead><tr><th rowspan="2">Time scale<br>Method</th><th colspan="2">200 ms (1)</th><th colspan="2">400 ms (2)</th><th colspan="2">800 ms (3)</th><th colspan="2">1600 ms (4)</th></tr><tr><th>|AV-Off<sub>1</sub>| ↓</th><th>AV-Conf<sub>1</sub> ↑</th><th>|AV-Off<sub>2</sub>| ↓</th><th>AV-Conf<sub>2</sub> ↑</th><th>|AV-Off<sub>3</sub>| ↓</th><th>AV-Conf<sub>3</sub> ↑</th><th>|AV-Off<sub>4</sub>| ↓</th><th>AV-Conf<sub>4</sub> ↑</th></tr></thead><tbody><tr><td>Static</td><td>9.45±4.49</td><td>0.63±0.23</td><td>10.22±4.17</td><td>1.07±0.34</td><td>9.01±4.21</td><td>1.06±0.35</td><td>5.72±4.26</td><td>0.99±1.11</td></tr><tr><td>Wav2Lip [27]</td><td>0.47±0.50</td><td>2.42±0.43</td><td>0.03±0.51</td><td>2.75±0.62</td><td>1.47±3.82</td><td>1.76±0.71</td><td>3.67±4.22</td><td>1.48±1.61</td></tr><tr><td>MakeItTalk [50]</td><td>1.55±2.75</td><td>1.36±0.59</td><td>2.48±4.74</td><td>1.71±0.70</td><td>4.63±5.81</td><td>1.47±0.68</td><td>4.02±4.26</td><td>1.27±1.42</td></tr><tr><td>Audio2Head [37]</td><td>2.58±0.51</td><td>1.67±0.48</td><td>2.41±3.34</td><td>1.88±0.64</td><td>3.74±4.89</td><td>1.45±0.63</td><td>3.43±4.10</td><td>1.39±1.43</td></tr><tr><td>OSTF [38]</td><td>2.13±2.26</td><td>1.38±0.50</td><td>3.15±4.53</td><td>1.62±0.64</td><td>5.47±5.42</td><td>1.24±0.50</td><td>3.62±4.07</td><td>1.40±1.45</td></tr><tr><td>MS-Sync-short</td><td><b>0.0±0.0</b></td><td><b>3.37±0.41</b></td><td><b>0.01±0.08</b></td><td><b>3.62±0.52</b></td><td><b>0.07±0.67</b></td><td><b>2.48±0.74</b></td><td><b>2.60±3.92</b></td><td><b>2.18±1.85</b></td></tr><tr><td>MS-Sync-long</td><td><u>0.01 ± 0.08</u></td><td><u>3.26 ± 0.41</u></td><td><u>0.01±0.08</u></td><td><u>3.56 ± 0.51</u></td><td><u>0.10 ± 0.82</u></td><td><u>2.49±0.71</u></td><td><u>1.84 ± 3.38</u></td><td><u>2.20±1.74</u></td></tr><tr><td>Ground truth</td><td>0.87±0.93</td><td>2.41±0.46</td><td>0.52±1.40</td><td>2.50±0.62</td><td>2.39±4.31</td><td>1.57±0.67</td><td>2.97±3.83</td><td>1.69±1.64</td></tr></tbody></table>

<table><caption>Table 4. Landmark domain multi-scale AV synchrony on HDTF test set. As above, separate syncer models inspired from SyncNet are trained on each time scale on HDTF dataset and used to compute the correlation scores.</caption><thead><tr><th rowspan="2">Time scale<br>Method</th><th colspan="2">200 ms (1)</th><th colspan="2">400 ms (2)</th><th colspan="2">800 ms (3)</th><th colspan="2">1600 ms (4)</th></tr><tr><th>|AV-Off<sub>1</sub>| ↓</th><th>AV-Conf<sub>1</sub> ↑</th><th>|AV-Off<sub>2</sub>| ↓</th><th>AV-Conf<sub>2</sub> ↑</th><th>|AV-Off<sub>3</sub>| ↓</th><th>AV-Conf<sub>3</sub> ↑</th><th>|AV-Off<sub>4</sub>| ↓</th><th>AV-Conf<sub>4</sub> ↑</th></tr></thead><tbody><tr><td>Static</td><td>9.77±4.41</td><td>0.63±0.24</td><td>9.92±4.22</td><td>1.10±0.36</td><td>9.29±4.14</td><td>1.00±0.32</td><td>3.75±2.92</td><td>0.58±0.48</td></tr><tr><td>Wav2Lip [27]</td><td><b>0.29±0.57</b></td><td>1.84±0.50</td><td>0.98±2.62</td><td>2.00±0.62</td><td>2.32±4.59</td><td>1.37±0.52</td><td>1.26±2.45</td><td>1.07±0.67</td></tr><tr><td>MakeItTalk [50]</td><td>1.42±2.51</td><td>1.15±0.43</td><td>1.53±3.20</td><td>1.86±0.59</td><td>2.22±4.03</td><td>1.57±0.59</td><td>1.46±2.28</td><td>1.21±0.80</td></tr><tr><td>Audio2Head [37]</td><td>1.34±0.80</td><td>1.90±0.53</td><td><b>0.45±2.09</b></td><td>2.24±0.68</td><td>1.82±3.61</td><td>1.59±0.60</td><td>1.19±2.05</td><td>1.22±0.74</td></tr><tr><td>OSTF [38]</td><td>0.78±1.89</td><td>1.54±0.58</td><td>1.71±3.58</td><td>1.99±0.73</td><td>3.77±5.15</td><td>1.45±0.61</td><td>1.78±2.44</td><td>1.14±0.82</td></tr><tr><td>MS-Sync-short</td><td>0.76±0.62</td><td><b>2.68±0.48</b></td><td>0.93±0.27</td><td><b>2.80±0.58</b></td><td><b>0.48±2.12</b></td><td><b>2.03±0.67</b></td><td><b>0.86±1.93</b></td><td><b>1.42±0.85</b></td></tr><tr><td>MS-Sync-long</td><td><u>0.72 ± 0.62</u></td><td><u>2.58 ± 0.48</u></td><td><u>0.87 ± 0.34</u></td><td><u>2.63 ± 0.61</u></td><td><u>0.48±2.07</u></td><td><u>2.01 ± 0.67</u></td><td><u>0.96 ± 2.07</u></td><td><u>1.38 ± 0.85</u></td></tr><tr><td>Ground truth</td><td>1.03±0.98</td><td>1.95±0.52</td><td>1.14±2.21</td><td>2.25±0.70</td><td>1.90±3.84</td><td>1.58±0.64</td><td>1.22±2.06</td><td>1.17±0.76</td></tr></tbody></table>

as a hint that this model lacks diversity. Audio2Head suffers from the same limitation to an even greater extent: although visually compelling, the movements it produces are stereotypical and therefore penalized by their too small variance in the Fréchet distance calculations. On the other hand, MiT performs well in FID but its dynamics are of noticeable lower quality. Finally, the MS-Sync-long results show that a mere change in training strategy allows to greatly reduce error accumulation over 200 time steps, although it is here at the cost of a slightly lower quality on shorter sequences.

### 4.3. Landmark-domain multi-scale AV synchrony

**Protocol.** Ideal multi-scale AV synchrony scores should convey how much a model succeeds in exploiting the audio signal to produce motion over diverse time scales. To that end, we resort to audio-visual datasets which preserve motion dynamics, namely VoxCeleb2 (II) and HDTF [46], and carry our evaluations in the landmark domain. We split HDTF into 291 and 51 train and test identities, and further split the test videos into 1058 80-frame clips. Likewise, measurements on VoxCeleb2 (II) are made on sequences of

80 frames. Pyramids of AV syncers dedicated to metrics calculation, equivalent to landmark-domain SyncNETs [8] on full faces, are trained beforehand on both datasets. Note that contrary to Section 3, we use the triplet loss of [8] to train the syncers. The AV synchrony is evaluated using the absolute value of the audio-visual offset (|AV-Off|) and the confidence score (AV-Conf) introduced in [8], measured at four different scales via the syncer pyramids on successively downsampled audio-visual chunks of duration 200 ms, 400 ms, 800 ms and 1600 ms. Hence an offset of 1 at the finest resolution, sampled at 25 fps, amounts to a misalignment of 40 ms between modalities, whereas at the coarsest scale this rises to 320 ms. We report in Figure 3 the performances of the first level syncer on VoxCeleb2 (II) test set. The distributions of ground truth synchrony scores are closer to Gaussian than perfect Dirac. This is partly because the same audio signal may correspond to more than one facial configuration, and the syncers may not fully grasp this diversity. The synchrony scores reported here should therefore be viewed in light of this assumption: rather than an actual finer AV alignment, results that appear "better" than
points and reenactment model as in the present work, al-
though here the reenactment model is not fine-tuned.

**Training Details.** The temporal module introduced in section 3 is implemented using 4 self-attention layers with 8 heads each. Both audio and keypoints are encoded as 512-dimensional vectors, and both syncers and the generative models were trained on VoxCeleb2 (II). The frame discriminator $D_f$ is a multi-layer perceptron, and the window-based multi-scale $D_s$ follows the LSTM implementation of Airale et al. [2]. Syncers training lasts from around one day for the finest time scale to a few hours for the coarsest model. We rely on hard negative mining for the first three time scales with $N = 12$, and use negative pairs from different audio and dynamics samples at the coarsest scale where the shortened sequences reduce the number of available hard negative pairs, and set $N = 48$ for this last model. Syncers are then frozen and used for the training of the generative model, which is trained to predict sequences of 40 frames for 70k iterations (about 500 epochs) using Adam optimizers with $\beta_1 = 0$ and $\beta_2 = 0.999$ and learning rates of $2 \times 10^{-5}$ and $1 \times 10^{-5}$ respectively for the generator and the discriminator, after which a decay factor of 0.1 is applied on the learning rates for 5k additional iterations. All audio inputs are sampled at 16 kHz, and we use a window size of 400 and hop size of 160 to generate the 26-dimensional MFCC spectrograms.

## 4.2. Image-Domain AV Synchrony

**Protocol.** The first benchmark follows the customary evaluation of the audio-visual synchrony in the visual domain using several standard metrics, on both VoxCeleb2 (I) and LRS2. To cope with the imbalanced duration of VoxCeleb2 videos and keep computation time manageable, we work with the first 40 frames of each test clip, while we use the whole LRS2 test set, which contains shorter videos (41 frames in average, ranging from 15 to 145 frames). We use the absolute offset [*AV-Off*] and the confidence score *AV-Conf* output by SyncNET [7], and also extract the facial landmarks from the output videos [3] to compute the frontalized landmark distance (LMD) from the predicted mouth region to the ground truth landmarks, by rotating each frame to a canonical pose. This ensures that all methods are placed on an equal footing, whether they produce head motion or not. Results are presented in Table 1.

**Results.** Audio-visual correlation scores provided by SyncNet show strong results for MS-Sync, significantly ahead of all other methods regarding either the absolute offset or the confidence. One exception is the confidence score of Wav2Lip, which is biased by the use of the SyncNet score as a loss function. The LMD gives a complementary picture of the audio-visual alignment, with less variance between

models partly due to a lower influence of outputs’ visual
attributes. MS-Sync performs very well regarding this met-
ric, being only surpassed by IP_LAP on VoxCeleb2 and by
SadTalker on LRS2. Although the novelty of the proposed
approach does not lie in the improvement of the fine-scale
AV correlation *per se*, the results from Table 1 show that the
use of a pyramid of syncers in the multi-scale loss function
has a positive effect on the AV synchrony at the finest time
scale.

4.3. Multi-scale Correlation between Speech and Rigid Head Motion

**Protocol.** In this section we explore a research direction generally overlooked in the talking head generation literature. Although several previous works include a mechanism to learn meaningful head motion, none of them attempt to measure the relevance of the produced rigid dynamics. To tackle this issue, it is once again useful to consider a low-dimensional motion representation: audio-visual input pyramids can be efficiently built in a similar fashion to that of Section 3, on a set of points restricted to rigid facial parts. Although no such subset was used to train our model, it is crucial for a fair comparison with other methods that the multi-scale syncers should not be trained on the same facial keypoints as the ones used to compute the loss. To this end, we rely instead on the 31 facial landmarks of the eyes and nose extracted from the Face Alignment method [3], discarding lips and jaws altogether, and further use a triplet loss for the training instead of the cross-entropy loss described in Section 3. Three syncer networks operating on three different time scales were thus trained to quantify the correlation between speech and rigid head motion on both VoxCeleb2 (II) and HDTF, two datasets that preserve motion dynamics. To prevent unwanted correlations from interfering, e.g. between voice pitch and facial structure, contrastive loss pairs are mined from the same sequences. In addition, all test identities are unseen during training. Results, reported in Table 2, are measured on sequences of 80 frames. For HDTF, a split of 1058 80-frame test clips were extracted from 51 long duration samples chosen randomly for testing. The AV synchrony is evaluated using the absolute value of the audio-visual offset provided by the newly trained syncer pyramid at three different scales, corresponding to audio-visual chunks of 200 ms, 400 ms and 800 ms. For 25 fps, this means that an offset of 1 at the finest resolution corresponds to a misalignment of 40 ms between modalities, while this rises to 160 ms at the coarsest time scale.

**Results.** The first important finding is that it is possible to train neural networks to measure the temporal syncing between speech and rigid motion on three different time scales

<footnote n="3"><sup>3</sup>In all tables, bold indicates best result, underline second best.</footnote>
<table><caption>Table 5. Image domain AV synchrony. † We rescaled the extracted landmarks from PC-AVS that are cropped differently by a factor 0.75 to make it comparable with the original data scale.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="4">VoxCeleb2 (I)</th><th colspan="4">LRS2</th></tr><tr><th>|AV-Off|↓</th><th>AV-Conf ↑</th><th>LMD↓</th><th>LMD<sub>front</sub>↓</th><th>|AV-Off|↓</th><th>AV-Conf ↑</th><th>LMD↓</th><th>LMD<sub>front</sub>↓</th></tr></thead><tbody><tr><td>Ground truth</td><td>1.89±1.92</td><td>6.29±1.66</td><td>0.0</td><td>0.0</td><td>0.08±0.4</td><td>8.36±1.62</td><td>0.0</td><td>0.0</td></tr><tr><td>GT landmarks + MiT</td><td>3.52±4.50</td><td>3.55±1.52</td><td>1.60±0.30</td><td>1.53±0.29</td><td>1.72±3.91</td><td>4.61±1.70</td><td>1.60±0.31</td><td>1.54±0.30</td></tr><tr><td colspan="9"><em>Methods with fixed head pose</em></td></tr><tr><td>Wav2Lip [27]</td><td>2.86±0.34</td><td><strong>8.07±1.33</strong></td><td><strong>2.90±1.09</strong></td><td><strong>2.66±0.95</strong></td><td>2.76±0.55</td><td><strong>8.53±1.37</strong></td><td><strong>2.99±1.05</strong></td><td>2.80±0.89</td></tr><tr><td>PC-AVS<sup>†</sup> [49]</td><td>5.18±3.31</td><td>3.85±1.55</td><td>3.19±1.60</td><td>3.00±1.44</td><td>5.48±3.65</td><td>4.42±1.65</td><td>3.16±1.28</td><td>2.96±1.03</td></tr><tr><td colspan="9"><em>Methods that predict head motion</em></td></tr><tr><td>MakeItTalk [50]</td><td>5.23±4.29</td><td>3.50±1.49</td><td>3.33±1.41</td><td>3.07±1.33</td><td>8.43±6.16</td><td>2.56±0.96</td><td>3.31±1.45</td><td>3.08±1.28</td></tr><tr><td>Audio2Head [37]</td><td>6.83±6.66</td><td>2.66±1.38</td><td>3.90±1.33</td><td>3.61±1.23</td><td>6.78±6.72</td><td>3.18±1.43</td><td>3.80±1.30</td><td>3.60±1.18</td></tr><tr><td>OSTF [38]</td><td>2.59±4.29</td><td>4.12±1.72</td><td>3.44±1.45</td><td>3.18±1.27</td><td>2.59±4.23</td><td>4.56±1.67</td><td>3.45±1.46</td><td>3.25±1.37</td></tr><tr><td>MS-Sync-short + MiT</td><td><strong>2.00±2.56</strong></td><td><u>4.53 ± 1.51</u></td><td>3.15±1.20</td><td>2.85±1.11</td><td><strong>1.60±2.39</strong></td><td><u>5.09 ± 1.47</u></td><td>3.07±1.11</td><td>2.83±0.83</td></tr><tr><td>MS-Sync-long + MiT</td><td><u>2.20 ± 2.61</u></td><td>4.35±1.49</td><td>3.06 ± 1.17</td><td><u>2.80 ± 1.10</u></td><td><u>2.05 ± 3.09</u></td><td>4.83±1.51</td><td><u>3.00 ± 1.18</u></td><td><strong>2.76±1.02</strong></td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Figure 4. Qualitative comparison of the results produced by different methods on three VoxCeleb2 (II) test sequences.</figcaption></figure>

the ground truth instead correspond to a closer match to the modes of the AV distribution discovered by the syncers.

**Results.** The AV synchrony scores are reported in Table 3 and 4 for VoxCeleb2 (II) and HDTF, respectively. We did not include PC-AVS in this section because of distinct cropping strategies producing inconsistent results. Although the

loss functions are different, the syncer pyramid used to train the model and the one which serves to compute the metrics were both trained on VoxCeleb2 (II): our model almost perfectly learned to optimize this loss, hence the very strong AV correlation scores. The HDTF results expose the generalization abilities of the different methods. Although Wav2Lip presents the best $|AV-Off_1|$ at the first scale and
<table><caption>Table 1. Image domain AV synchrony. LMD is the frontalized landmark distance, with the face rotated back to a canonical pose.† We rescale PC-AVS by a factor 0.75 to account for cropping. *Wav2Lip is trained to optimize the SyncNet scores, hence the very strong confidence.³</caption><thead><tr><th rowspan="2">Dataset<br>Method</th><th colspan="3">VoxCeleb2 (I)</th><th colspan="3">LRS2</th></tr><tr><th>|AV-Off| ↓</th><th>AV-Conf ↑</th><th>LMD ↓</th><th>|AV-Off| ↓</th><th>AV-Conf ↑</th><th>LMD ↓</th></tr></thead><tbody><tr><td>Ground truth</td><td>1.89±1.92</td><td>6.29±1.66</td><td>0.0</td><td>0.08±0.4</td><td>8.36±1.62</td><td>0.0</td></tr><tr><td>MakeItTalk [73]</td><td>5.23±4.29</td><td>3.50±1.49</td><td>2.80±1.10</td><td>8.43±6.16</td><td>2.56±0.96</td><td>2.80±1.07</td></tr><tr><td>Wav2Lip* [36]</td><td>2.86±0.34</td><td><b>8.07±1.33</b></td><td>2.41±0.77</td><td>2.79±0.54</td><td><b>8.72±1.25</b></td><td>2.54±0.70</td></tr><tr><td>PC-AVS<sup>†</sup> [72]</td><td>5.18±3.31</td><td>3.85±1.55</td><td>2.65±1.23</td><td>5.48±3.65</td><td>4.42±1.65</td><td>2.61±0.85</td></tr><tr><td>Audio2Head [53]</td><td>6.83±6.66</td><td>2.66±1.38</td><td>3.23±1.19</td><td>6.78±6.72</td><td>3.18±1.43</td><td>3.20±1.09</td></tr><tr><td>EAMM [20]</td><td>9.52±5.76</td><td>1.94±0.75</td><td>3.30±1.24</td><td>9.07±5.94</td><td>2.41±0.87</td><td>3.50±1.18</td></tr><tr><td>Wang et al. [54]</td><td>2.59±4.29</td><td>4.12±1.72</td><td>2.89±1.26</td><td><u>2.59±4.23</u></td><td>4.56±1.67</td><td>2.89±1.26</td></tr><tr><td>IP_LAP [70]</td><td>4.78±5.01</td><td>3.15±1.40</td><td><b>2.34±0.79</b></td><td>4.73±5.03</td><td>3.80±1.57</td><td>2.41±0.71</td></tr><tr><td>SadTalker [67]</td><td><u>2.19±3.92</u></td><td>4.63±1.91</td><td>2.38±0.75</td><td>2.72±4.60</td><td>5.01±1.97</td><td><b>2.28±0.86</b></td></tr><tr><td>MS-Sync (Ours)</td><td><b>1.00 ± 1.64</b></td><td><u>5.81±1.66</u></td><td><u>2.36 ± 0.73</u></td><td><b>1.06 ± 2.14</b></td><td><u>5.75±1.69</u></td><td><u>2.34 ± 0.81</u></td></tr></tbody></table>

<table><caption>Table 2. Landmark domain rigid multi-scale AV synchrony on VoxCeleb2 (II), where test syncers are trained on rigid facial parts: eyes and nose are considered, but not lips and jaws landmarks.</caption><thead><tr><th rowspan="2">Dataset<br>Time scale<br>Method</th><th colspan="3">VoxCeleb2 (II)</th><th colspan="3">HDTF</th></tr><tr><th>200 ms (1)<br>|AV-Off<sub>1</sub>| ↓</th><th>400 ms (2)<br>|AV-Off<sub>2</sub>| ↓</th><th>800 ms (3)<br>|AV-Off<sub>3</sub>| ↓</th><th>200 ms (1)<br>|AV-Off<sub>1</sub>| ↓</th><th>400 ms (2)<br>|AV-Off<sub>2</sub>| ↓</th><th>800 ms (3)<br>|AV-Off<sub>3</sub>| ↓</th></tr></thead><tbody><tr><td>Random</td><td>10.59±4.18</td><td>10.50±4.09</td><td>10.11±4.03</td><td>12.14±4.01</td><td>10.32±4.56</td><td>9.48±4.63</td></tr><tr><td>Ground truth</td><td>4.57±5.32</td><td>4.64±5.60</td><td>7.48±5.47</td><td>9.43±5.53</td><td>7.63±6.00</td><td>7.89±5.74</td></tr><tr><td>MakeItTalk [73]</td><td>6.91±5.95</td><td>6.77±6.09</td><td>8.64 ± 5.42</td><td>14.40±1.90</td><td>13.64 ± 2.62</td><td>14.18 ± 2.05</td></tr><tr><td>Wav2Lip [36]</td><td>5.38±5.90</td><td>6.72 ± 6.28</td><td>9.22 ± 5.52</td><td>15.00±0.05</td><td>14.88 ± 0.61</td><td>14.87 ± 0.72</td></tr><tr><td>Audio2Head [53]</td><td>9.43±4.79</td><td>8.95±5.09</td><td>9.61±4.91</td><td><u>9.55±5.40</u></td><td>8.60±5.55</td><td>7.97±5.30</td></tr><tr><td>EAMM [20]</td><td><u>4.98±5.71</u></td><td><u>4.71±5.87</u></td><td><u>7.30±5.80</u></td><td>14.09±2.39</td><td>13.78±2.49</td><td>14.30±1.76</td></tr><tr><td>Wang et al. [54]</td><td>8.26±5.61</td><td>8.58±5.58</td><td>9.14±5.07</td><td>10.68±4.42</td><td><u>8.55±4.55</u></td><td><u>7.21±4.78</u></td></tr><tr><td>IP_LAP [70]</td><td>9.66±5.47</td><td>9.89±5.24</td><td>10.90±4.65</td><td>15.00±0.00</td><td>14.88±0.65</td><td>14.92±0.58</td></tr><tr><td>SadTalker [67]</td><td>6.11±5.74</td><td>7.59±5.84</td><td>9.52±4.99</td><td>12.68±3.64</td><td>11.26±4.24</td><td>10.01±4.62</td></tr><tr><td>MS-Sync (Ours)</td><td><b>4.55 ± 4.91</b></td><td><b>4.40 ± 5.30</b></td><td><b>7.04 ± 5.70</b></td><td><b>9.01 ± 5.45</b></td><td><b>7.86 ± 5.47</b></td><td><b>6.53 ± 5.20</b></td></tr></tbody></table>

on both datasets. The evidence of that is the significant difference (p-value < 0.05 in all cases) in AV offsets calculated by the syncers between ground truth and randomly sampled audio-visual pairs. This supports the existence of a correlation between speech and head motion, albeit fainter with HDTF which features videos of codified political addresses. The second finding is that MS-Sync consistently performs the best in terms of offset over all other methods, often with a large margin, for all scales of both VoxCeleb2

and HDTF. This suggests that the proposed method succeeds in producing head motion that are time-aligned with the audio input over various time scales and datasets, which, in average, cannot be said for any other method.

## 4.4. Visual Quality

**Quantitative Results.** The Fréchet Inception Distance (FID [17]) of the output sequences is measured and presented in table 3. We found SSIM and PSNR, which are
Audio2Head the best $|AV-Off_2|$ at the second scale, MS-Sync possesses the second best scores and largely outperforms all models in terms of AV confidence. What is more, the gap in favor of our model increases at the two coarsest scales, highlighting the effectiveness of the proposed approach to correlate input speech and generated motion on multiple time scales. Last it is noteworthy that although it produces no head motion, Wav2Lip results remain way above the static, uncorrelated boundary even at the coarsest resolution. This means that the mouth region is still partly informative at the top pyramid level, possibly pleading for a stronger blurring strategy.

## 4.4. Image-domain AV synchrony

**Protocol.** In a third batch of experiments the synchrony is calculated in the image domain, similar to the classical evaluation protocol. To do so, we first map the landmarks output by our model in the image space using the reenactment system from MakeItTalk (hereafter MiT [50]). Although this leads to blurry results when the pose changes from the initial orientation, we found it sufficient for the sake of AV synchrony measurements. We use two datasets for these experiments: a subset of 2141 videos from the original test set of VoxCeleb2 (I), and LRS2. To cope with the imbalanced duration of VoxCeleb2 videos and keep computation time manageable, we work with the first 40 frames in each clip, while we use the whole LRS2 test set, which contains shorter videos. In addition to the absolute AV offset and confidence score given by SyncNET, we compute the Landmark Distance (LMD [5]), together with a frontalized version LMD$_{front}$ that better accounts for face rotation. For a fair comparison, we do not directly use the landmarks produced by MS-Sync to measure the LMD but extract it back from the reenacted video clips; the same procedure is applied to the ground truth landmarks to help assess the effects of each of the previous steps on the metrics.

**Results.** Although not the primary scope of our study, the landmarks produced by MS-Sync and reenacted with MiT behave surprisingly well in the image domain (Table 5). MS-Sync outperforms all other models in terms of AV offset on both VoxCeleb2 (I) and LRS2, simply falling short of Wav2Lip in terms of AV confidence on the two datasets, and of LMD on VoxCeleb2. In particular, MS-Sync performs better than all other models with head motion (and notably MakeItTalk) on all considered metrics. Notice also how the fact that Wav2Lip leaves the whole input face beyond the lips intact seems to bias the calculation of AV confidence in its favor, especially when considering the MiT-reenacted ground truth landmarks. This suggests that SyncNET is sensitive to the image sharpness: the shortcomings of the image reenactment systems probably set limits on the achievable offset and confidence values.

<figure><img src="image_8.png" /><figcaption>Figure 5. Evolution of multi-scale audio-visual confidence over training, measured on VoxCeleb2 (II) validation set. Top is the finest scale, bottom the coarsest.</figcaption></figure>

## 4.5. Qualitative results

In Figure 4 we present several sequences output by different models and the corresponding ground truth sequences over 120 frames. An examination of these examples shows that mouth closing and opening produced by MS-Sync look correctly aligned with the original, but interestingly this also seems to be the case for head motion although the loss only enforces convergence of distributions. Although the motion produced by OSTF is qualitatively good, it is slightly less diverse and tends to frontalize the face disregarding the original orientation. Wav2Lip, on the other hand, only synchronizes the lips.

## 4.6. Ablation study

In this section we explore the roles of the multi-scale AV synchrony loss and of the multi-scale generator on the output results, and in particular in AV confidence at different resolutions. As can be seen in Figure 5 for the evolution of the validation AV confidence along training, almost no difference is visible at the finest resolution between the full MS-Sync model and its single-scale loss, single-scale generator equivalent. However as expected the confidence of the latter model falls significantly below as one moves upward in the feature pyramid as the loss does not explicitly enforce multi-scale synchrony. It is possible to circumvent this effect by enabling the multi-scale AV synchrony loss, however it is clear that if the generator is not itself a multi-scale network, it lacks capacity to fully exploit the loss, resulting in average performances on every scale.
<figure><img src="image_7.png" /><figcaption>Figure 4. Comparison between different one-shot talking head generation methods with explicit head motion treatment on hard samples: left with a widely open initial mouth, right with closed eyes. Consecutive frames are 600ms apart. MS-Sync can deal with these samples while producing rigid head motion synchronized with the speech.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 5. Comparison between different one-shot talking head generation methods with explicit head motion treatment. Consecutive frames are 600ms apart.</figcaption></figure>

usually used as complementary metrics of proximity to target images, to correlate poorly with visual sharpness in this free head motion generation setting and therefore do not include them here. SadTalker, which produces very sharp outputs based on the 3DMM model, obtains the best results. However, although using the same reenactment model as that of Audio2Head [53] and Wang et al. [54], MS-Sync

obtains better FID scores, which likely expresses the higher diversity of head pose it produces, better matching the original data distribution. Interestingly, these results computed on output sequences of 120 frames also highlight the ability of the proposed autoregressive generator to produce accurate keypoints dynamics over extended duration, as training was done on shorter sequences of 40 frames.
## 5. Conclusion

The approach proposed in this work is the first attempt to learn and model audio-visual correlations at multiple scales for talking head generation. This is enabled thanks to a pyramid of syncer models that are trained on hierarchical representations of input audio and landmark position sequences, and then used to compute the loss for the training of the generative model. Importantly, we showed that this model should also be built on a multi-scale backbone, implemented here as a feature pyramid network together with individual branches for each pyramid level that are merged using a soft learnable mask. The very encouraging results of MS-Sync let us foresee numerous applications of similar approaches on other audio-visual generation tasks. One research direction could thus consist in replacing the facial landmarks with other quantities, be it low dimensional key-points or body joints, or real-world images. Another orthogonal direction may lead to extend the focus to additional cross modal relationships, such as audio-visual emotions.

## References

[1] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman. Deep audio-visual speech recognition. In *arXiv:1809.02108*, 2018.

[2] Louis Airale, Xavier Alameda-Pineda, Stéphane Lathuilière, and Dominique Vaufreydaz. Autoregressive gan for semantic unconditional head motion generation. *arXiv preprint arXiv:2211.00987*, 2022.

[3] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In *International Conference on Computer Vision*, 2017.

[4] Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. Talking-head generation with rhythmic head motion. In *European Conference on Computer Vision*, pages 35–51. Springer, 2020.

[5] Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements generation at a glance. In *Proceedings of the European conference on computer vision (ECCV)*, pages 520–535, 2018.

[6] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 7832–7841, 2019.

[7] J. S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. In *INTERSPEECH*, 2018.

[8] Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In *Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13*, pages 251–263. Springer, 2017.

[9] Dipanjan Das, Sandika Biswas, Sanjana Sinha, and Brojeshwar Bhowmick. Speech-driven facial animation using cascaded gans for learning of motion and texture. In *European

conference on computer vision, pages 408–424. Springer, 2020.

[10] Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. *Advances in neural information processing systems*, 28, 2015.

[11] Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech-driven 3d facial animation with transformers. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 18770–18780, 2022.

[12] David Greenwood, Iain Matthews, and Stephen Laycock. Joint learning of facial expression and head pose from speech. In *Interspeech*, 2018.

[13] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 5784–5794, 2021.

[14] Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim. Marionette: Few-shot face reenactment preserving identity of unseen targets. In *Proceedings of the AAAI conference on artificial intelligence*, volume 34, pages 10893–10900, 2020.

[15] Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. Eamm: One-shot emotional talking face via audio-based emotion-aware motion model. *arXiv preprint arXiv:2205.15278*, 2022.

[16] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial animation by joint end-to-end learning of pose and emotion. *ACM Transactions on Graphics (TOG)*, 36(4):1–12, 2017.

[17] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 4401–4410, 2019.

[18] Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. *Advances in neural information processing systems*, 32, 2019.

[19] Jae Hyun Lim and Jong Chul Ye. Geometric gan. *arXiv preprint arXiv:1705.02894*, 2017.

[20] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2117–2125, 2017.

[21] Xiao Lin and Mohamed R Amer. Human motion modeling using dvgans. *arXiv preprint arXiv:1804.10652*, 2018.

[22] Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. *Advances in neural information processing systems*, 31, 2018.

[23] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the
<table><caption>Table 3. Visual quality comparison on VoxCeleb (II) on sequences of 120 frames</caption><thead><tr><th>Method</th><th>FID ↓</th></tr></thead><tbody><tr><td>MakeItTalk [73]</td><td>23.6</td></tr><tr><td>Wav2Lip [36]</td><td>21.9</td></tr><tr><td>Audio2Head [53]</td><td>28.6</td></tr><tr><td>Wang et al. [54]</td><td>19.5</td></tr><tr><td>SadTalker [67]</td><td><strong>14.1</strong></td></tr><tr><td>MS-Sync (ours)</td><td><u>18.7</u></td></tr></tbody></table>

<table><caption>Table 4. Effects of adversarial and reconstruction loss functions on AV correlation and visual quality.</caption><thead><tr><th>(λ<sub>rec</sub>, λ<sub>adv</sub>)</th><th>AV-Off ↓</th><th>AV-Conf ↑</th><th>FID ↓</th></tr></thead><tbody><tr><td>(1, 0)</td><td><u>0.84</u></td><td>5.09</td><td>29.6</td></tr><tr><td>(1, 0.01)</td><td><strong>0.75</strong></td><td><strong>6.40</strong></td><td>19.7</td></tr><tr><td>(1, 1)</td><td>1.26</td><td>5.61</td><td><u>19.0</u></td></tr><tr><td>(0.01, 1)</td><td>1.41</td><td>5.52</td><td>20.1</td></tr><tr><td>Full Model (1, 0.1)</td><td>1.06</td><td><u>5.75</u></td><td><strong>18.7</strong></td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Figure 6. Evolution of multi-scale audio-visual confidence over training, measured on VoxCeleb2 (II) validation set. Bottom is the finest scale, top the coarsest.</figcaption></figure>

**Qualitative Comparison.** Finally qualitative results on VoxCeleb2 are represented in fig. 4 and fig. 5, along with other one-shot reenactment methods that explicitly handle head motion. These comprise difficult samples, such as an open mouth or closed eyes in the initial frame. Contrary to other methods, MS-Sync deals successfully with all these samples. MakeItTalk [73] produces overall good quality results but with almost no head motion. Audio2head [53] struggles to preserve the input identity, a limitation that is overcome in Wang *et al.* [54] although their strategy of learning on a single identity results in a lack of naturalness and audio correlation. Finally, SadTalker [67] produces very sharp video outputs with high quality lip-syncing but fails to correlate output head motion with speech.

## 4.5. Ablation Study

**Weighting Factors of the Loss Function.** In this first ablation we explore the effects of the adversarial and the reconstruction terms on the visual quality and the audio-visual synchrony through their respective scaling factors $\lambda_{adv}$ and $\lambda_{rec}$ (see table 4). Results show that a small, but non-zero, $\lambda_{adv}$ provides results that are better synced with the input audio, but at the cost of a degraded visual quality. This is also the case when the scale of the reconstruction is reduced with respect to the adversarial term. The chosen hyperparameters overall provide the best trade-off, yielding in particular the lowest FID score.

**Effects of the Multi-scale AV Loss.** In fig. 6 we represent the training evolution of the validation AV confidence over

three pyramid layers, and evaluate the effects of removing the multi-scale audio-visual loss, thereby enforcing the synchrony at the finest time scale only (bottom panel in the figure). One can see that doing so indeed gives improvements in the correlation at the finest time scale, but at the same time the confidence degrades for all coarser time scales. Finally, using the multi-scale AV loss provides significantly stronger long-range audio-visual synchrony results, achieving the objectives for which it was designed.

## 5. Discussion and Limitations

Experiments show that our framework is an efficient way to explicitly handle the correlation between speech and facial dynamics, including both lips and head motion. But it also has the advantage of being readily generalizable: the multi-scale AV synchrony loss can fuse in any talking head architecture that relies on an implicit or explicit low-dimensional representation of motion, as long as the use of Gaussian smoothing on this representation makes sense.

The MS-Sync model also comes with a number of limitations. As all continuous autoregressive models, it is prone to error accumulation which limits the output sequence length, although we found the results to remain visually sharp past 120 frames (or ~5s) of video duration, which is three times the training sequence length. Fig. 7 provides examples of such failure cases, although in this case a tighter cropping can partially alleviate the visual defaults. We also found that jitter could be a problem especially on internet images, which, together with the previous observation, could be
2020s. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 11976–11986, 2022.

[24] Moustafa Meshry, Saksham Suri, Larry S Davis, and Abhinav Shrivastava. Learned spatial representations for few-shot talking-head synthesis. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 13829–13838, 2021.

[25] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. *arXiv preprint arXiv:1807.03748*, 2018.

[26] Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, and Yong Man Ro. Synctalkface: Talking face generation with precise lip-syncing via audio-lip memory. In *36th AAAI Conference on Artificial Intelligence (AAAI 22)*. Association for the Advancement of Artificial Intelligence, 2022.

[27] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiriri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In *Proceedings of the 28th ACM International Conference on Multimedia*, pages 484–492, 2020.

[28] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 13759–13768, 2021.

[29] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. *Advances in Neural Information Processing Systems*, 32, 2019.

[30] Sanjana Sinha, Sandika Biswas, Ravindra Yadav, and Brojeshwar Bhowmick. Emotion-controllable generalized talking face generation. *arXiv preprint arXiv:2205.01155*, 2022.

[31] Yang Song, Jingwen Zhu, Dawei Li, Xiaolong Wang, and Hairong Qi. Talking face generation by conditional recurrent adversarial network. *arXiv preprint arXiv:1804.04786*, 2018.

[32] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learning lip sync from audio. *ACM Transactions on Graphics (ToG)*, 36(4):1–13, 2017.

[33] Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficienttdet: Scalable and efficient object detection. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10781–10790, 2020.

[34] Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and Iain Matthews. A deep learning approach for generalized speech animation. *ACM Transactions on Graphics (TOG)*, 36(4):1–11, 2017.

[35] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial animation with gans. *International Journal of Computer Vision*, 128(5):1398–1413, 2020.

[36] Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, and Baoyuan Wang. Progressive disentangled representation learning for fine-grained controllable talking head synthesis. *arXiv preprint arXiv:2211.14506*, 2022.

[37] Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. In *IJCAI*, 2021.

[38] Suzhen Wang, Lincheng Li, Yu Ding, and Xin Yu. One-shot talking face generation from single-speaker audio-visual correlation learning. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 36, pages 2531–2539, 2022.

[39] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 8798–8807, 2018.

[40] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10039–10049, 2021.

[41] Hani C Yehia, Takaaki Kuratate, and Eric Vatikiotis-Bateson. Linking facial animation, head motion and speech acoustics. *Journal of phonetics*, 30(3):555–568, 2002.

[42] Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu. Audio-driven talking face video generation with learning-based personalized head pose. *arXiv preprint arXiv:2002.10137*, 2020.

[43] Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In *Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVII*, pages 85–101. Springer, 2022.

[44] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of one-shot realistic head avatars. In *European Conference on Computer Vision*, pages 524–540. Springer, 2020.

[45] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 9459–9468, 2019.

[46] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 3661–3670, 2021.

[47] Ruiqi Zhao, Tianyi Wu, and Guodong Guo. Sparse to dense motion transfer for face image animation. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 1991–2000, 2021.

[48] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially disentangled audio-visual representation. In *Proceedings of the AAAI conference on artificial intelligence*, volume 33, pages 9299–9306, 2019.

[49] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In *Proceedings of the IEEE/CVF conference on*
<figure><img src="image_10.png" /><figcaption>Figure 7. Examples of improvements obtained from cropping the source image on two failure cases from HDTF test set.</figcaption></figure>

hinting for a possible sensitivity to out-of-distribution samples. We hypothesize that propagating visual loss gradients through the reenactment model could alleviate this issue, at the cost of additional computation.

## 6. Conclusion

The approach proposed in this research work is the first attempt to learn and model audio-visual correlations at multiple scales for talking head generation. For this we use a pyramid of syncer models, trained on hierarchical representations of input audio and head dynamics, which are later used in a multi-scale AV synchrony loss for the training of the generative model. Experiments showed that the devised multi-scale generative network succeeds in producing realistic head and lip motion output over various time scales. The very encouraging results of MS-Sync let us foresee numerous applications of similar approaches on other audio-visual generation tasks, for instance to enforce the consistency of an apparent emotional state which will typically evolve on much longer time scales than the ones considered here.

## References

[1] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 44(12):8717-8727, 2018.

[2] Louis Airale, Xavier Alameda-Pineda, Stéphane Lathuilière, and Dominique Vaufreydaz. Autoregressive

GAN for Semantic Unconditional Head Motion Generation. ACM Transactions on Multimedia Computing, Communications and Applications, pages 1-11, 2024.

[3] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In International Conference on Computer Vision, 2017.

[4] Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. Talking-head generation with rhythmic head motion. In European Conference on Computer Vision, pages 35-51. Springer, 2020.

[5] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7832-7841, 2019.

[6] J. S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. In INTERSPEECH, 2018.

[7] Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251-263. Springer, 2017.

[8] Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. Advances in neural information processing systems, 28, 2015.

[9] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. In Proceedings of the IEEE/CVF International conference on Computer Vision, pages 14398-14407, 2021.

[10] Chenpeng Du, Qi Chen, Tianyu He, Xu Tan, Xie Chen, Kai Yu, Sheng Zhao, and Jiang Bian. Dae-talker: High fidelity speech-driven talking face generation with diffusion autoencoder. In Proceedings of the 31st ACM International Conference on Multimedia, pages 4281-4289, 2023.

[11] Sefik Emre Eskimez, Ross K Maddox, Chenliang Xu, and Zhiyao Duan. Generating talking face landmarks from speech. In Latent Variable Analysis and Signal Separation: 14th International Conference, LVA/ICA 2018, Guildford, UK, July 2-5, 2018, Proceedings 14, pages 372-381. Springer, 2018.

[12] Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech-driven 3d facial animation with transformers. In Proceedings
*computer vision and pattern recognition*, pages 4176–4186, 2021.

[50] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. *ACM Transactions on Graphics (TOG)*, 39(6):1–15, 2020.

[51] Hao Zhu, Huaibo Huang, Yi Li, Aihua Zheng, and Ran He. Arbitrary talking face generation via attentional audio-visual coherence learning. *arXiv preprint arXiv:1812.06589*, 2018.
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770–18780, 2022.

[13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), pages 2672–2680, 2014.

[14] David Greenwood, Iain Matthews, and Stephen Laycock. Joint learning of facial expression and head pose from speech. In Interspeech, 2018.

[15] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5784–5794, 2021.

[16] Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim. Marionette: Few-shot face reenactment preserving identity of unseen targets. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 10893–10900, 2020.

[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 30:6626–6637, 2017.

[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.

[19] Ricong Huang, Peiwen Lai, Yipeng Qin, and Guanbin Li. Parametric implicit face representation for audio-driven facial reenactment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12759–12768, 2023.

[20] Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model. In ACM SIGGRAPH 2022 Conference Proceedings, SIGGRAPH '22, 2022.

[21] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics (TOG), 36(4):1–12, 2017.

[22] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.

[23] Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.

[24] Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7568–7578, 2023.

[25] Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. Expressive talking head generation with granular audio-visual control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3387–3396, 2022.

[26] Sen Liang, Zhize Zhou, Rong Li, Juyong Zhang, and Hujun Bao. Talkingflow: Talking facial landmark generation with multi-scale normalizing flow network. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4628–4632. IEEE, 2022.

[27] Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv:1705.02894, 2017.

[28] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.

[29] Xiao Lin and Mohamed R Amer. Human motion modeling using dvgans. arXiv preprint arXiv:1804.10652, 2018.

[30] Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. Semantic-aware implicit neural audio-driven video portrait generation. In European Conference on Computer Vision, pages 106–125. Springer, 2022.

[31] Yunfei Liu, Lijian Lin, Fei Yu, Changyin Zhou, and Yu Li. Moda: Mapping-once audio-driven portrait animation with dual attentions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23020–23029, 2023.

[32] Moustafa Meshry, Saksham Suri, Larry S Davis, and Abhinav Shrivastava. Learned spatial representations for few-shot talking-head synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13829–13838, 2021.

[33] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view synthesis. *Communications of the ACM*, 65(1):99-106, 2021.

[34] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. *arXiv preprint arXiv:1807.03748*, 2018.

[35] Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, and Yong Man Ro. Synctalkface: Talking face generation with precise lip-syncing via audio-lip memory. In *36th AAAI Conference on Artificial Intelligence (AAAI 22)*. Association for the Advancement of Artificial Intelligence, 2022.

[36] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In *Proceedings of the 28th ACM International Conference on Multimedia*, pages 484-492, 2020.

[37] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 13759-13768, 2021.

[38] Shuai Shen, Wanhua Li, Zheng Zhu, Yueqi Duan, Jie Zhou, and Jiwen Lu. Learning dynamic facial radiance fields for few-shot talking head synthesis. In *European Conference on Computer Vision*, pages 666-682. Springer, 2022.

[39] Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 1982-1991, 2023.

[40] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. *Advances in Neural Information Processing Systems*, 32, 2019.

[41] Sanjana Sinha, Sandika Biswas, Ravindra Yadav, and Brojeshwar Bhowmick. Emotion-controllable generalized talking face generation. In Lud De Raedt, editor, *Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22*, pages 1320-1327, Vienna (Austria), 7 2022. International Joint Conferences on Artificial Intelligence Organization.

[42] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In *International conference on machine learning*, pages 2256-2265. PMLR, 2015.

[43] Yang Song, Jingwen Zhu, Dawei Li, Andy Wang, and Hairong Qi. Talking face generation by conditional recurrent adversarial network. In *Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19*, pages 919-925, Macao, China, Jul 2019. International Joint Conferences on Artificial Intelligence Organization.

[44] Michał Stypułkowski, Konstantinos Vougioukas, Sen He, Maciej Zięba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face generation. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pages 5091-5100, 2024.

[45] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learning lip sync from audio. *ACM Transactions on Graphics (ToG)*, 36(4):1-13, 2017.

[46] Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10781-10790, 2020.

[47] Shuai Tan, Bin Ji, and Ye Pan. Emmn: Emotional motion memory network for audio-driven emotional talking face generation. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 22146-22156, 2023.

[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. *Advances in neural information processing systems*, 30, 2017.

[49] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial animation with gans. *International Journal of Computer Vision*, 128(5):1398-1413, 2020.

[50] Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, and Baoyuan Wang. Progressive disentangled representation learning for fine-grained controllable talking head synthesis. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 17979-17989, 2023.

[51] Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. Seeing what you said: Talking face generation guided by a lip reading expert. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 14653-14662, 2023.

[52] Jiayu Wang, Kang Zhao, Shiwei Zhang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Lipformer: High-fidelity and generalizable talking face
generation with a pre-learned facial codebook. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13844–13853, 2023.

[53] Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. In *IJCAI*, 2021.

[54] Suzhen Wang, Lincheng Li, Yu Ding, and Xin Yu. One-shot talking face generation from single-speaker audio-visual correlation learning. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 36, pages 2531–2539, 2022.

[55] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 8798–8807, 2018.

[56] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10039–10049, 2021.

[57] Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 12780–12790, 2023.

[58] Chao Xu, Junwei Zhu, Jiangning Zhang, Yue Han, Wenqing Chu, Ying Tai, Chengjie Wang, Zhifeng Xie, and Yong Liu. High-fidelity generalized emotional talking face generation with multi-modal emotion space learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 6609–6619, 2023.

[59] Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. In *The Eleventh International Conference on Learning Representations*, Kigali (Rwanda), May 2023.

[60] Hani C Yehia, Takaaki Kuratate, and Eric Vatikiotis-Bateson. Linking facial animation, head motion and speech acoustics. *Journal of phonetics*, 30(3):555–568, 2002.

[61] Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu. Audio-driven talking face video generation with learning-based personalized head pose. *arXiv preprint arXiv:2002.10137*, 2020.

[62] Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pretrained stylegan. In *Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII*, pages 85-101. Springer, 2022.

[63] Zhentao Yu, Zixin Yin, Deyu Zhou, Duomin Wang, Finn Wong, and Baoyuan Wang. Talking head generation with probabilistic audio-to-visual diffusion priors. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 7645-7655, 2023.

[64] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of one-shot realistic head avatars. In *European Conference on Computer Vision*, pages 524-540. Springer, 2020.

[65] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 9459-9468, 2019.

[66] Chenxu Zhang, Yifan Zhao, Yifei Huang, Ming Zeng, Saifeng Ni, Madhukar Budagavi, and Xiaohu Guo. Facial: Synthesizing dynamic talking face with implicit attribute learning. In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 3867-3876, 2021.

[67] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 8652-8661, 2023.

[68] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 3661-3670, 2021.

[69] Ruiqi Zhao, Tianyi Wu, and Guodong Guo. Sparse to dense motion transfer for face image animation. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 1991-2000, 2021.

[70] Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, and Guanbin Li. Identity-preserving talking face generation with landmark and appearance priors. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 9729-9738, 2023.
[71] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiao-gang Wang. Talking face generation by adversarially disentangled audio-visual representation. In *Proceed-ings of the AAAI conference on artificial intelligence*, volume 33, pages 9299–9306, 2019.

[72] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In *Proceed-ings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 4176–4186, 2021.

[73] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. *ACM Transactions on Graphics (TOG)*, 39(6):1–15, 2020.

[74] Hao Zhu, Huaibo Huang, Yi Li, Aihua Zheng, and Ran He. Arbitrary talking face generation via attentional audio-visual coherence learning. In *Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence*, IJCAI’20, 2021.