DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation

## Abstract

Abstract Talking head generation intends to produce vivid and realistic talking head videos from a single portrait and speech audio clip. Although significant progress has been made in diffusion-based talking head generation, almost all methods rely on autoregressive strategies, which suffer from limited context utilization beyond the current generation step, error accumulation, and slower generation speed. To address these challenges, we present DAWN ( D ynamic frame A vatar W ith N on-autoregressive diffusion), a framework that enables all-at-once generation of dynamic-length video sequences. Specifically, it consists of two main components: (1) audio-driven holistic facial dynamics generation in the latent motion space, and (2) audio-driven head pose and blink generation. Extensive experiments demonstrate that our method generates authentic and vivid videos with precise lip motions, and natural pose/blink movements. Additionally, with a high generation speed, DAWN possesses strong extrapolation capabilities, ensuring the stable production of high-quality long videos. These results highlight the considerable promise and potential impact of DAWN in the field of talking head video generation. Furthermore, we hope that DAWN sparks further exploration of non-autoregressive approaches in diffusion models. Our code will be publicly available at https://github.com/Hanbo-Cheng/DAWN-pytorch .

## 1 Introduction

Talking head generation aims at synthesizing a realistic and expressive talking head from a given portrait and audio clip, which is garnering growing interest due to its potential applications in virtual meetings, gaming, and film production. For talking head generation, it is essential that the lip motions in the generated video precisely match the accompanying speech, while maintaining high overall visual fidelity (Guo et al., ). Furthermore, natural coordination between head pose, eye blinking, and the rhythm of the audio is also crucial for a convincing output (Liu et al., ).

Recently, Diffusion Models (DM) (Ho et al., ) have achieved significant success in video and image generation tasks (Rombach et al., ; Ho et al., ; ; Peebles & Xie, ; Ni et al., ). However, their application in talking head generation (Shen et al., ; Bigioi et al., ) still faces several challenges.
Although many methods yield high-quality results, most of them rely on autoregressive (AR) (Tian et al., ; Ma et al., ) or semi-autoregressive (SAR) (Xu et al., ; He et al., ) strategies. In each iteration, AR generates one frame, while SAR generates a fixed-length video segment. The two strategies significantly slow down the inference speed and fail to adequately utilize contextual information from future frames, which leads to constrained performance and potential error accumulation, especially in long video sequences (Stypułkowski et al., ).

To address these challenges, we present DAWN (Dynamic frame Avatar With Non-autoregressive diffusion), a novel approach that significantly improves both the quality and efficiency of talking head generation.
Our approach leverages the DM to generate motion representation sequences from a given audio and portrait. These motion representations are subsequently used to reconstruct the video. Unlike other methods, our approach produces videos of arbitrary length in a non-autoregressive (NAR) manner.
However, employing the NAR strategy to generate long videos often results in either over-smoothing or significant content inconsistencies due to limited extrapolation (Qiu et al., ). In the context of talking head generation, we suggest that the model’s temporal modeling capability is significantly hindered by the strong coupling relationship among multiple motions. Typically, the motions in talking head include (1) lip motions and (2) head pose and blink movements. The temporal dependency of head and blink movements extends over several seconds, far longer than that of lip motions (Zhou et al., ). Training models to capture these long-term dependencies requires extensive sequences, thus increasing the difficulty and cost of training. Fortunately, head and blink movements can be represented as low-dimensional vectors (Zhang et al., ), enabling the design of a lightweight model that learns these long-term dependencies by training on extended sequences. Thus, to further enhance the temporal modeling and extrapolation capabilities of DAWN, we disentangle the motion components involved in talking head videos. Specifically, we use the Audio-to-Video Flow Diffusion Model (A2V-FDM) to learn the implicit mapping between the lips and audio, while generating the head pose and blinks via explicit control signals. Additionally, we propose a lightweight Pose and Blink generation Network (PBNet) trained on long sequences, to generate natural pose/blink movements during inference in a NAR manner. In this way, we simplify the training of A2V-FDM as well as achieve the long-term dependency modeling of the pose/blink movement. To further strengthen the convergence and extrapolation capabilities of A2V-FDM, we propose a two-stage training strategy based on curriculum learning to guide the model in generating accurate lip motion and precise pose/blink movement control.

The main contributions of this work are as follows:
1) We present DAWN (Dynamic frame Avatar With Non-autoregressive diffusion) for generating dynamic-length talking head videos from portrait images and audio clips in a non-autoregressive (NAR) manner, achieving faster inference speeds and high-quality results. To the best of our knowledge, this is the first NAR solution based on diffusion models designed for general talking head generation.
2) To compensate for the limitations of extrapolation in NAR strategies and enhance the temporal modeling capabilities for long videos, we decouple the motions of the lips, head, and blink, achieving precise control over these movements.
3) We propose the Pose and Blink generation Network (PBNet) to generate natural head pose and blink sequences exclusively from audio in a NAR manner.
4) We introduce the Two-stage Curriculum Learning (TCL) strategy to guide the model in mastering lip motion generation and precise pose/blink control, ensuring strong convergence and extrapolation ability.

## 2 Related works

Audio-driven talking head generation. Initial approaches for talking head generation employed deterministic models to map audio to video streams (Fan et al., ), with later methods introducing generative models such as GANs (Isola et al., ), VAEs (Kingma & Welling, ), and diffusion models (DMs) (Ho et al., ). GAN-related methods (Vougioukas et al., ; Pumarola et al., ; Hong et al., ) improved visual realism but faced convergence and mode collapse issues (Xia et al., ). Following this, VAEs (Kingma & Welling, ) generated 3D priors like 3D Morphable Models (3DMM) (Blanz & Vetter, ) followed by high-fidelity rendering (Ren et al., ), which limited the realism and vividness. In contrast, DMs have been introduced into talking head generation due to their good convergence, excellent generation performance, and diversity. Stypułkowski et al. () presented a DM-based talking head generation solution using an AR strategy to generate videos frame-by-frame iteratively. Subsequently, Tian et al. () improved this AR strategy by incorporating motion conditions extracted by a VAE-based network as priors for each iteration, effectively mitigating degradation issues. Concurrently, Xu et al. (); He et al. () advocated for motion modeling instead of image modeling. They utilized DMs to iteratively generate latent motion representations over a fixed number of frames in a SAR manner, subsequently converting these motion representations into video frames. While most diffusion-based methods produce promising results, their AR or SAR strategies incur slow generation speeds and collapse in long-video generation. Although methods like Tian et al. (); Xu et al. () alleviate issues such as inconsistencies in content across iterations and long video generation collapse, the risk of error accumulation remains unresolved. While methods like Du et al. () use identity-specific NAR strategies, to the best of our knowledge, none have addressed NAR talking head generation for arbitrary identities. Consequently, we propose a novel NAR dynamic frame generation framework based on DM, which aims to achieve the low-cost and high-quality rapid generation of realistic talking head video through a clip of audio and arbitrary portrait.

Audio-driven pose and blink generation.
Head pose and blink movements significantly impact the naturalness of talking head videos. However, the mapping from audio to pose and blink movement is a one-to-many problem, which presents a significant challenge (Xu et al., ; Chen et al., ). Early works primarily focused on controlling poses directly using facial landmarks or video references (Zhou et al., ; Guo et al., ). However, these approaches require additional guidance information, which impairs the diversity of the results. Later studies considered generating both pose and blink within the context of talking head generation (Zhou et al., ). However, simultaneously generating all facial movements can cause interference and ambiguity (Zhang et al., ). Therefore, some works attempted to decouple the speaker’s actions into components like lip, head pose, and blink, using discriminative models to predict these conditions separately (Wang et al., ; He et al., ). Later, researchers recognized that probabilistic modeling is better suited for the one-to-many mapping relationship, leading to the proposal of a VAE-based pose generation framework (Liu et al., ).
However, most existing pose generation strategies also depend on AR or SAR approaches, negatively impacting efficiency, smoothness, and naturalness. To address these issues, we design a VAE-based NAR pose generation method to produce vivid and smooth pose and blink movements while maintaining the NAR generation of the entire framework.

## 3 Method

As shown in Figure 1, DAWN is divided into three main parts: (1) the Latent Flow Generator (LFG); (2) the conditional Audio-to-Video Flow Diffusion Model (A2V-FDM); and (3) the Pose and Blink generation Network (PBNet). First, we train the LFG to estimate the motion representation between different video frames in the latent space.
Subsequently, the A2V-FDM is trained to generate temporally coherent motion representation from audio.
Finally, PBNet is used to generate poses and blinks from audio to control the content in the A2V-FDM. To enhance the model’s extrapolation ability while ensuring better convergence, we propose a novel Two-stage Curriculum Learning (TCL) training strategy. We will first discuss preliminaries, then present the specific details of DAWN’s three main components, namely LFG, A2V-FDM, and PBNet in Sections 3.2, 3.3, and 3.4, respectively. Finally, we will introduce the TCL strategy in Section 3.5.

### 3.1 Preliminaries

Task definition. The task of talking head generation involves creating a natural and vivid talking head video from two inputs: a static single-person portrait, ${\bm{x}}_{\text{src}}$, and a speech sequence, $\displaystyle{\bm{y}}_{1:N}=\{{\bm{y}}_{0},{\bm{y}}_{1},\ldots,{\bm{y}}_{N}\}$. The static image ${\bm{x}}_{\text{src}}$ and the speech sequence $\displaystyle{\bm{y}}$ can originate from any individual, and the output is $\hat{{\bm{x}}}_{1:N}=\{\hat{{\bm{x}}}_{0},\hat{{\bm{x}}}_{1},\ldots,\hat{{\bm{
x}}}_{N}\}$, where $N$ represents the total number of frames.

Diffusion models. Diffusion models generate samples conforming to a given data distribution by progressively denoising Gaussian noise (Ho et al., ). Let ${\bm{x}}_{0}$ represent data sampled from a given distribution $q({\bm{x}}_{0})$. In the forward diffusion process, Gaussian noise is progressively added to ${\bm{x}}_{0}$ after $T$ steps, resulting in noisy data ${\bm{x}}_{T}$ (Nichol & Dhariwal, ; Song et al., ), and the conditional transition distribution at each step is defined as:

$$ $\displaystyle q({\bm{x}}_{t}|{\bm{x}}_{0})=\mathcal{N}({\bm{x}}_{t};\sqrt{\bar {\alpha}_{t}}\,{\bm{x}}_{0},(1-\bar{\alpha}_{t}){\bm{I}})$ (1) $$

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$ . The reverse diffusion process gradually recovers the original data from the Gaussian noise ${\bm{x}}_{T}\sim\mathcal{N}(0,I)$, utilizing a neural network to predict $p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t})$, where $\theta$ represents the parameters of the neural network. The model is trained using the following loss function:

$$ $\displaystyle L_{\text{simple}}=\mathbb{E}_{t,{\bm{x}}_{0},\epsilon}[\| \epsilon-\epsilon_{\theta}({\bm{x}}_{t},t)\|^{2}]$ (2) $$

where $\epsilon$ is the Gaussian noise added to ${\bm{x}}_{0}$ in the forward diffusion process to obtain ${\bm{x}}_{t}$, and $\epsilon_{\theta}({\bm{x}}_{t},t)$ is the noise predicted by the model. In video-related tasks, the denoising model can be implemented via a 3D U-Net (Ho et al., ; Çiçek et al., ).

Figure: Figure 1: The pipeline of DAWN. First, we train the Latent Flow Generator (LFG) in (a) to extract the motion representation from the video. Then the Pose and Blink generation Network (PBNet) in (b) is utilized to generate the head pose and blink sequences of the avatar. Subsequently, the Audio-to-Video Flow Diffusion Model (A2V-FDM) in (c) generates the talking head video from the source image conditioned by the audio and pose/blink sequences provided by the PBNet.
Refer to caption: x1.png

### 3.2 Latent Flow Generator

The Latent Flow Generator (LFG) is a self-supervised training framework designed to model motion information between the source image ${\bm{x}}_{\text{src}}$ and the driving image ${\bm{x}}_{\text{dri}}$. As illustrated in Figure 1 (a), LFG consists of three trainable modules: the image encoder $\mathcal{E}$, the flow predictor $\mathcal{P}$, and the image decoder $\mathcal{D}$. During training, ${\bm{x}}_{\text{src}},{\bm{x}}_{\text{dri}}\in\mathbb{R}^{H\times W\times 3}$ are images randomly selected from the same video. The image encoder $\mathcal{E}$ encodes the source image ${\bm{x}}_{\text{src}}$ into a latent code ${\bm{z}}_{\text{src}}\in\mathbb{R}^{H_{z}\times W_{z}\times C_{z}}$. The flow predictor estimates a dense flow map ${\bm{f}}$ and a blocking map ${\bm{m}}$ (Siarohin et al., ; ), corresponding to ${\bm{x}}_{\text{src}}$ and ${\bm{x}}_{\text{dri}}$ :

$$ $\displaystyle{\bm{f}},{\bm{m}}=\mathcal{P}({\bm{x}}_{\text{src}},{\bm{x}}_{ \text{dri}})$ (3) $$

The flow map ${\bm{f}}\in\mathbb{R}^{H_{z}\times W_{z}\times 2}$ describes the feature-level movement of ${\bm{x}}_{\text{dri}}$ relative to ${\bm{x}}_{\text{src}}$ in horizontal and vertical directions. The blocking map ${\bm{m}}\in\mathbb{R}^{H_{z}\times W_{z}\times 1}$ ranging from 0 to 1, indicates the degree of area blocking in the transformation from ${\bm{x}}_{\text{src}}$ to ${\bm{x}}_{\text{dri}}$. The flow map ${\bm{f}}$ is used to perform the affine transformation $\mathcal{A}$, serving as a coarse-grained warping of ${\bm{z}}_{\text{src}}$. Subsequently, the blocking map ${\bm{m}}$ guides the model in repairing the occlusion area, thereby serving as fine-grained repair. Finally, the image decoder $\mathcal{D}$ converts the warped latent code into the target image $\hat{{\bm{x}}}_{\text{gen}}$, where the $\otimes$ is the element-wise product:

$$ $\displaystyle\hat{{\bm{x}}}_{\text{gen}}=\mathcal{D}(\mathcal{A}({\bm{z}}_{ \text{src}},{\bm{f}})\otimes{\bm{m}})$ (4) $$

The LFG is trained in an unsupervised manner and optimized using the following reconstruction loss:

$$ $\displaystyle L_{\text{LFG}}=L_{\text{rec}}(\hat{{\bm{x}}}_{\text{gen}},{\bm{x }}_{\text{dri}})$ (5) $$

where $L_{\text{rec}}$ is a multi-resolution reconstruction loss derived from a pre-trained VGG-19 network, used to evaluate perceptual differences between $\hat{{\bm{x}}}_{\text{gen}}$ and ${\bm{x}}_{\text{dri}}$ (Johnson et al., ). We consider the concatenation of ${\bm{m}}$ and ${\bm{f}}$ as ${\bm{z}}^{m}=[{\bm{f}},{\bm{m}}]$ to represent the motion of ${\bm{x}}_{\text{dri}}$ relative to ${\bm{x}}_{\text{src}}$. In this way, we achieve two objectives: 1) finding an effective explicit motion representation ${\bm{z}}^{m}$, which is identity-agnostic and well-supported by physical meaning, and 2) reconstructing ${\bm{x}}_{\text{dri}}$ from ${\bm{x}}_{\text{src}}$ and ${\bm{z}}^{m}$ without the need for a full pixel generation.

### 3.3 Conditional Audio2Video Flow Diffusion Model

Through the LFG in Section 3.2, we can specify a ${\bm{x}}_{\text{src}}$, and extract the identity-agnostic motion representations ${\bm{z}}^{m}_{1:N}$ of the talking head video clip ${\bm{x}}_{1:N}$ as well as the latent code ${\bm{z}}_{\text{src}}$. Therefore, we design the A2V-FDM to generate the motion representations $\hat{{\bm{z}}}^{m}_{1:N}=[\hat{{\bm{f}}}_{1:N},\hat{{\bm{m}}}_{1:N}]$ of each frame relative to ${\bm{x}}_{\text{src}}$:

$$ $\displaystyle\hat{{\bm{z}}}^{m}_{1:N}=\text{DM}({\bm{x}}_{\text{src}},{\bm{y}} _{1:N},\bm{\rho}_{1:N})$ (6) $$

where the DM refers the diffusion model. After generating $\hat{{\bm{z}}}^{m}_{1:N}$, we use the decoder $\mathcal{D}$ in LFG to reconstruct the $\hat{{\bm{z}}}^{m}_{1:N}$ to the target video frames, via the Equation 4. The structure of A2V-FDM is illustrated in Figure 1 (c). The A2V-FDM model includes a 3D U-Net denoising backbone (Ho et al., ). The residue block in the 3D U-Net contains temporal attention and spatial attention modules, which handle frame-level and pixel-level dependencies respectively. The parameters of the temporal attention module are independent of the input length, so we believe that 3D U-Net can theoretically process video sequences of any length. However, to ease training difficulty, we train on short sequences and aim for long-sequence inference. To enhance the 3D U-Net’s extrapolation ability when handling long sequences, we used Rotary Positional Encoding (RoPE) (Su et al., ) instead of traditional absolute position embeddings in the temporal attention module.

Conditioning. We incorporate the following conditions to control its generative behavior: audio embedding ${\bm{a}}_{1:N}$, pose/blink signal $\bm{\rho}_{1:N}$, and source image latent code ${\bm{z}}_{\text{src}}$. The audio embedding ${\bm{a}}_{1:N}$, extracted from the audio ${\mathbf{y}}_{1:N}$ using Hubert (Hsu et al., ), implicitly controls the lip motion. Due to the strict alignment with video frames, we apply the audio embedding to its corresponding image. Additionally, the avatar’s pose and blink are controlled via explicit signals. The pose is described by a 6D vector Ji et al. (). During training, the pose is extracted from video using an open-source tool (Guo et al., ). For the blink signal, we adopt the aspect ratio of the left and right eyes, following (Zhang et al., ). To account for the arbitrary pose and eye-opening degree of the source image ${\bm{x}}_{\text{src}}$, we use the difference between the current frame ${\bm{x}}_{i}$ and the source frame ${\bm{x}}_{\text{src}}$: $\Delta\bm{\rho}_{i}=\bm{\rho}_{i}-\bm{\rho}_{\text{src}}$, which models the transition of state rather than the state itself. The model is provided with features ${\bm{z}}_{\text{src}}$ to supply facial visual details.
Each frame’s latent code performs cross attention with the audio embedding ${\bm{a}}_{i}$ and pose/blink information $\Delta\bm{\rho}_{i}$, respectively. This process injects these conditions into the latent code with different spatial weights, controlling specific regions of the generated content.
The image feature ${\bm{z}}_{\text{src}}$ is regarded as a global condition and is concatenated directly with the noisy data as the initial input to the 3D U-Net. We also utilize landmarks to create a face region mask for ${\bm{x}}_{\text{src}}$, embedded with a lightweight convolutional network, similar to the approach by Tian et al. (). This mask adds to the denoising process in the same manner as ${\bm{z}}_{\text{src}}$.

Loss function. We employed the DM regular denoising loss, $L_{\text{simple}}$, in Equation 2 to train our model. The synchronization of lip motions with audio is crucial for the talking head task, while the lips often constitute only a small portion of the frame. Consequently, during training, we employed landmarks to isolate the lip region by generating a lip mask, ${\bm{m}}_{\text{lip}}$. We then applied an additional weight, $w_{\text{lip}}$, to the denoising process of this region, similar to Stypułkowski et al. (). Ultimately, our loss function is defined as:

$$ $\displaystyle L=L_{\text{simple}}+w_{\text{lip}}\cdot(L_{\text{simple}}\otimes {\bm{m}}_{\text{lip}})$ (7) $$

where the symbol $\otimes$ denotes element-wise product.

### 3.4 Audio-Driven Pose and Blink Generation

To prevent the generated results from exhibiting overly monotonous and minimal movements, we design a separate module, namely the Pose and Blink generation Network (PBNet). As shown in Figure 1 (b), PBNet learns the mapping between audio and pose/blink movements. To maintain the non-autoregressive (NAR) generation capabilities of A2V-FDM, PBNet employs a transformer-based Variational Autoencoder (VAE) (Petrovich et al., ) to generate variable-length pose and blink sequences.
The inputs to PBNet include the initial pose/blink state $\bm{\rho}_{\text{src}}$, the residual pose/blink $\Delta\bm{\rho}_{1:N}$, and the audio embedding ${\bm{a}}_{1:N}$. The transformer encoder $\mathcal{E}_{\text{t}}$ embeds these inputs into a Gaussian-distributed and obtain latent code ${\bm{h}}\in\mathbb{R}^{N\times C_{h}}$ through resampling :

$$ $\displaystyle\bm{\mu},\text{log}\bm{\sigma}=\mathcal{E}_{\text{t}}(\bm{\rho}_{ \text{src}},\Delta\bm{\rho}_{1:N},{\bm{a}}_{1:N}),\;s.t.\;{\bm{h}}\sim\mathcal {N}(\bm{\mu},\bm{\sigma})$ (8) $$

We design ${\bm{h}}$ to have the same length as $\Delta\bm{\rho}$ to ensure it can carry sufficient information for variable-length inputs. The transformer decoder $\mathcal{D}_{\text{t}}$ generates the final pose/blink sequence $\Delta\hat{\bm{\rho}}_{1:N}$, conditioned on ${\bm{a}}_{1:N}$ and $\bm{\rho}_{\text{src}}$ :

$$ $\displaystyle\Delta\hat{\bm{\rho}}_{1:N}=\mathcal{D}_{\text{t}}({\bm{h}},{\bm{ a}}_{1:N},\bm{\rho}_{\text{src}})$ (9) $$

To enhance the model’s extrapolation capability, we use RoPE as the positional encoding in the decoder, consistent with A2V-FDM. During training, we apply an MSE-based reconstruction loss $L_{\text{rec}}$ and an adversarial loss $L_{\text{GAN}}$ to guide the model in completing basic reconstruction tasks (Isola et al., ; Ginosar et al., ). Additionally, we employ a KL divergence loss $L_{\text{KL}}$ to ensure that the latent code ${\bm{h}}$ closely approximates a standard Gaussian distribution.

### 3.5 two-stage curriculum learning for talking head generation

Empirical evidence indicates that training our A2V-FDM model solely with fixed-length short video clips leads to inaccurate control of poses and blinks, as well as poor generalization to longer videos. We argue that a one-step training approach impedes the model’s convergence to an optimal solution and fails to achieve satisfactory results in the complex task of talking head generation. To address these issues, we propose an innovative Two-Stage Curriculum Learning (TCL) strategy inspired by the theory of curriculum learning (Bengio et al., ).

Overall, the goal of the A2V-FDM during the training process can be expressed as:

$$ $\displaystyle\hat{{\bm{x}}}_{1:N}=\mathcal{D}(\text{DM}({\bm{x}}_{\text{src}}, {\bm{y}}_{1:N},\bm{\rho}_{1:N}))$ (10) $$

In the first stage, we set ${\bm{x}}_{\text{src}}={\bm{x}}_{1}$, and the sequence length $N=K^{\prime}$ is a fixed, relatively small constant. This stage primarily focuses on enabling the model to generate basic lip motions. However, utilizing ${\bm{x}}_{1}$ as the source image often exhibits limited variations in poses and blinks, and using short clips can result in a scarcity of training samples with significant pose or blink movements. Therefore, in the second stage, we set ${\bm{x}}_{\text{src}}\in{\mathbb{X}}$ randomly, where ${\mathbb{X}}$ is the set of frames in the entire video, to learn control capabilities of large pose transformation. Additionally, differing from stage one, we randomly set $N\in[K_{\min},K_{\max}]$, $K_{\min}>K^{\prime}$. This approach aims to enhance control over poses and blinks while maintaining precise lip motions, as longer clips contain more diverse pose and blink movements. Training with random-length sequences also helps the model avoid a bias towards fixed-length sequences, further enhancing the model’s extrapolation.

### 3.6 Inference

Our inference process consists of four steps: 1) Extract the audio embedding ${\bm{a}}_{1:N}$ from the input audio. 2) Provide the initial pose/blink state $\bm{\rho}_{\text{src}}$, extracted from the source image ${\bm{x}}_{\text{src}}$, to the PBNet. Along with ${\bm{a}}_{1:N}$ and a latent space vector ${\bm{h}}_{1:N}$ sampled from a standard Gaussian distribution, PBNet generates the pose/blink sequences $\hat{\bm{\rho}}_{1:N}$. 3) Input ${\bm{x}}_{\text{src}}$, ${\bm{a}}_{1:N}$, and $\hat{\bm{\rho}}_{1:N}$ into the A2V-FDM, which generates the motion representation sequences $\hat{{\bm{z}}}^{m}_{1:N}$. 4) Finally, decode the video sequence $\hat{{\bm{x}}}_{1:N}$ from ${\bm{x}}_{\text{src}}$ and $\hat{{\bm{z}}}^{m}_{1:N}$.

Our method leverages non-autoregressive (NAR) generation during the inference process. To enhance extrapolation during inference, we utilize local attention (Luong, ) in the temporal attention module for both the PBNet decoder and the 3D U-Net in A2V-FDM, which restricts the attention scores to a local region. This approach effectively models local dependencies in talking head videos. To accommodate the different temporal dependencies of lip motions and pose/blink movements, we use a larger window size in the local attention mechanism of PBNet compared to A2V-FDM.

## 4 Experiment

### 4.1 Setup

Dataset. Our method is evaluated on two datasets: CREMA (Cao et al., ) and HDTF (Zhang et al., ). The CREMA dataset was collected in a controlled laboratory environment and contains 7,442 videos from 91 identities, with durations ranging from 1 to 5 seconds. The HDTF dataset consists of 410 videos, with an average duration exceeding 100 seconds. These videos are gathered from wild scenarios and feature over 10,000 unique sentences for speech content, along with diverse head pose movements. We partitioned the CREMA dataset into training and testing sets following Stypułkowski et al. (). As for the HDTF dataset, we conducted a random split with a 9:1 ratio between the training and testing sets. We resized the videos at a resolution of $128\times 128$ and a frame rate of 25 frames per second (fps) without any additional preprocessing.

Implementation details.
The architecture of the encoder and decoder in our model aligns with the design proposed by Johnson et al. (), while the flow predictor is implemented based on MRAA (Siarohin et al., ).
The PBNet model is trained using pose and blink movement sequences of 200 frames. During the inference phase of the PBNet model, a local attention mechanism with a window size of 400 is employed. The motion representation space of the A2V-FDM model is $32\times 32\times 3$. The training of the A2V-FDM model is conducted using the TCL strategy, which involves two stages: 1) the model is trained using video clips of 20 frames, with the source image set as the first frame in the clip; 2) training is performed on sequences ranging from 30 to 40 frames. For the inference phase of the A2V-FDM model, local attention with a window size of 80 is applied. In our evaluation, the length of one-time inference for CREMA is dynamic and depends on the ground truth, while for HDTF, it is fixed at 200 frames for better comparison.

Evaluation metrics. We evaluate the performance of our method using various metrics. Specifically, we employ the Fréchet Inception Distance (FID) (Heusel et al., ) to assess the image quality. We utilize the $\text{FVD}_{16}$ and $\text{FVD}_{32}$ scores, which calculate the Fréchet Video Distance (FVD) based on window sizes of 16 and 32 frames, respectively, to evaluate video quality across different temporal scales. Furthermore, we assess the perception loss of lip shape using the confidence score ($\text{LSE}_{\text{C}}$) and distance score ($\text{LSE}_{\text{D}}$) (Chung & Zisserman, ). To evaluate the preservation of speaker identity during the generation, we use ArcFace (Deng et al., ) to extract features from both the ground truth image and the generated image, and use the cosine similarity (CSIM) between the two as the evaluation metric. Moreover, we employ the Beat Align Score (BAS) (Siyao et al., ) to evaluate the synchronization of head motion and audio, and calculate the number of blinks per second (blink/s) to assess the liveliness of the eyes.

### 4.2 Overall Comparison

We compared our method with several state-of-the-art methods: Wav2Lip (Prajwal et al., ), MakeItTalk (Zhou et al., ), Audio2Head (Wang et al., ), SadTalker (Zhang et al., ), and Diffused Heads (Stypułkowski et al., ). To ensure a fair comparison, we provided the same conditions for each baseline and generated the same number of frames in a single generation. The quantitative experimental results are presented in Table 1.

**Table 1: Quantitative comparison with several state-of-the-art methods methods on HDTF (Zhang et al., ) and CREMA (Cao et al., ) datasets. * Wav2Lip generated videos that only contain lip motions, while the rest remain still images. “$\uparrow$” indicates better performance with higher values, while “$\downarrow$” indicates better performance with lower values. For both BAS and Blink/s, we consider performance to be better when they are closer to the ground truth.**
|  | Method | FID$\downarrow$ | $\text{FVD}_{16}$$\downarrow$ | $\text{FVD}_{32}$$\downarrow$ | $\text{LSE}_{\text{C}}$$\uparrow$ | $\text{LSE}_{\text{D}}$$\downarrow$ | CSIM$\uparrow$ | BAS | Blink/s |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | GT | - | - | - | 5.88 | 7.87 | 1 | 0.192 | 0.24 |
|  | Audio2Head | 29.58 | 188.54 | 208.44 | 5.13 | 7.92 | 0.660 | 0.274 | 0.01 |
|  | MakeItalk | 19.87 | 159.38 | 320.77 | 3.78 | 9.15 | 0.788 | 0.261 | 0.05 |
|  | SadTalker | 16.05 | 101.43 | 158.85 | 5.57 | 7.36 | 0.808 | 0.244 | 0.33 |
|  | Diffused Heads | 13.01 | 64.27 | 116.18 | 4.56 | 9.26 | 0.673 | 0.185 | 0.26 |
|  | Wav2Lip* | 10.23 | 130.23 | 242.19 | 6.08 | 7.74 | 0.801 | - | - |
| CREMA | DAWN (ours) | 5.77 | 56.33 | 75.82 | 5.77 | 8.14 | 0.845 | 0.231 | 0.29 |
|  | GT | - | - | - | 7.95 | 7.33 | 1 | 0.267 | 0.75 |
|  | Audio2Head | 30.10 | 122.26 | 205.42 | 6.88 | 7.58 | 0.705 | 0.290 | 0.09 |
|  | MakeItalk | 23.65 | 120.42 | 221.14 | 4.41 | 9.69 | 0.744 | 0.295 | 0.09 |
|  | SadTalker | 26.11 | 97.43 | 187.43 | 6.27 | 8.03 | 0.767 | 0.297 | 0.47 |
|  | Wav2Lip* | 23.85 | 166.15 | 281.73 | 7.42 | 7.44 | 0.701 | - | - |
| HDTF | DAWN (ours) | 9.60 | 60.34 | 95.64 | 6.71 | 7.94 | 0.790 | 0.281 | 0.86 |

Figure: Figure 2: Qualitative comparison with several state-of-the-art methods methods on HDTF (Zhang et al., ) and CREMA (Cao et al., ) datasets. Our method produces higher-quality results in video quality, lip-sync consistency, identity preservation, and head motions.
Refer to caption: x2.png

**Table 2: Comparison with other generation strategies. The semi-autoregressive (SAR) generation strategy is similar to He et al. (). The two temporal resolution (TTR) generation method is mentioned in Harvey et al. ().**
| Method | Time(s)$\downarrow$ | FID$\downarrow$ | $\text{FVD}_{16}$$\downarrow$ | $\text{FVD}_{32}$$\downarrow$ | $\text{LSE}_{\text{C}}$$\uparrow$ | $\text{LSE}_{\text{D}}$$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| SAR | 11.42 | 13.00 | 120.33 | 210.52 | 4.34 | 8.29 |
| TTR | 19.25 | 9.77 | 95.42 | 137.14 | 4.87 | 8.68 |
| Ours | 7.32 | 9.60 | 60.34 | 95.64 | 6.71 | 7.94 |

**Table 3: The experiment of extrapolation evaluation. “Inference length" refers to the number of frames generated in a single inference process.**
| Inference length | FID$\downarrow$ | $\text{FVD}_{16}$$\downarrow$ | $\text{FVD}_{32}$$\downarrow$ | $\text{LSE}_{\text{C}}$$\uparrow$ | $\text{LSE}_{\text{D}}$$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| 40 | 9.35 | 59.58 | 94.09 | 5.76 | 7.89 |
| 100 | 9.83 | 61.72 | 98.80 | 6.41 | 7.96 |
| 200 | 9.60 | 60.34 | 95.64 | 6.71 | 7.94 |
| 400 | 10.36 | 61.57 | 97.84 | 6.63 | 8.12 |
| 600 | 10.30 | 60.44 | 96.62 | 6.76 | 8.02 |

**Table 4: Ablation study on TCL and PBNet. The “GT PB" refers to whether to use ground truth pose/blink signal.**
| Method | GT PB | FID$\downarrow$ | $\text{FVD}_{16}$$\downarrow$ | $\text{FVD}_{32}$$\downarrow$ | $\text{LSE}_{\text{C}}$$\uparrow$ | $\text{LSE}_{\text{D}}$$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| only stage 1 | ✓ | 7.95 | 81.84 | 126.52 | 4.38 | 10.04 |
| only stage 2 | ✓ | 13.71 | 125.75 | 166.83 | 6.14 | 8.43 |
| DAWN | ✓ | 9.68 | 52.05 | 87.11 | 6.71 | 7.99 |
| w/o PBNet | × | 15.20 | 100.94 | 162.35 | 5.79 | 8.36 |
| DAWN | × | 9.60 | 60.34 | 95.64 | 6.71 | 7.94 |

According to the results, our method achieves the best performance in FID, $\text{FVD}_{32}$, $\text{FVD}_{16}$, CSIM, BAS, and Blink/s metrics for both the CREMA and HDTF datasets. Additionally, our method achieves nearly best scores for $\text{LSE}_{\text{C}}$ and $\text{LSE}_{\text{D}}$, closely matching those of the Audio2Head and Wav2Lip baselines. These results demonstrate that our method produces talking head videos with the highest visual quality, rhythmic head poses, and natural blink movements. Additionally, it preserves the identity features of the source image to the greatest extent and achieves highly synchronized lip motions with the audio. Additionally, we test the generation speed of the aforementioned methods in Appendix A.2.2, suggesting our method achieves the fastest or near-fastest generation speed and requires significantly less time compared to the previous diffusion-based method, Diffused Heads.

For the qualitative experiment, as shown in Figure 2, we visualize the generation results for each baseline on different datasets. Our method evidently achieves the visual quality most similar to the ground truth, showcasing the most realistic and vivid visual effects. Compared to our method, SadTalker relies on the 3DMM prior, which limits its animation capability to the facial region only, resulting in significant artifacts when merging the face with the static torso below the neck. Additionally, SadTalker exhibits unnatural head pose movements and gaze direction, partially due to limited temporal modeling ability. Wav2Lip only drives the mouth region and cannot generate head poses or blinks. The Audio2Head fails to preserve the speaker’s identity during generation. For the HDTF dataset, the Diffused Heads method collapsed due to the error accumulation.

### 4.3 comparison with other generation strategies

We compared our non-autoregressive generation strategy with two regular video generation strategies: 1) semi-autoregressive (SAR) generation similar to He et al. (), and 2) two-temporal resolution (TTR), which trains two models with different temporal resolutions (Harvey et al., ). The time cost represents the time required to generate an 8-second talking head video. The models were evaluated on the CREMA dataset, and the results are shown in Table 2. According to the results, our non-autoregressive method produces videos with the highest quality and fastest speed. The generation speed of the SAR method is faster than TTR, while TTR produces higher video quality than SAR. This is because autoregressive methods suffer from degradation issues during iterative generation. Although TTR somewhat alleviates the degradation issue, it compromises generation speed. In summary, our non-autoregressive method addresses the degradation problem while preserving fast generation speed.

### 4.4 Extrapolation validation

To evaluate the extrapolation ability of our method, we conducted experiments to assess the impact of inference length on model performance using the HDTF dataset. We set the inference length to range from 40 to 600 frames in a single generation process. The results are illustrated in Table 3, suggesting that performance remains relatively stable as the inference length increases. Specifically, the image and video quality metrics, FID and FVD, are not significantly affected by the inference length. Besides, we found that the $\text{LSE}_{\text{C}}$ and $\text{LSE}_{\text{D}}$ scores tend to improve with increasing inference length while stabilizing when the length is sufficiently large. It suggests that audio with sufficient length helps the model produce more precise lip movement.

### 4.5 Ablation study

Ablation study on TCL strategy. Since we only use the TCL strategy on A2V-FDM, we excluded the PBNet in the evaluation process by using ground truth pose/blink to drive the video. The results are illustrated in Table 4. We separately trained A2V-FDM using either stage 1 or stage 2 of TCL strategy. According to the results, using only stage 1 causes an overall performance decrease across all metrics except FID. This is because, in the first stage, the model is trained on shorter clips. It tends to perform minor warping on the source image, which results in a lower FID score. Although minor warping can improve FID scores, it significantly diminishes the vividness of longer videos, leading to poorer FVD scores. Furthermore, training with a fixed length introduces biases into the non-autoregressive generation process, impacting the video’s smoothness and the accuracy of lip motions. This, in turn, results in worse FVD and penalties in $\text{LSE}_{\text{C}}$ and $\text{LSE}_{\text{D}}$ scores. Using only stage 2 also negatively affects the model’s performance. Compared to using only stage 1, the model achieves better $\text{LSE}_{\text{C}}$ and $\text{LSE}_{\text{D}}$ scores. This improvement is because training with longer sequences is beneficial for enhancing the precision of lip motions (Tian et al., ). However, the worse FID and FVD scores result from the model struggling to learn both pose/blink control and lip movement simultaneously. Using either stage 1 or stage 2 alone clearly falls behind our proposed TCL strategy, demonstrating its significant contribution to performance.

Ablation study on PBNet. We evaluate the effectiveness of the PBNet in Table 4. The term “w/o PBNet" indicates that the PBNet module was removed from the architecture, requiring the A2V-FDM to simultaneously generate pose, blink, and lip motions from the audio by itself. The results suggest an overall enhancement of all evaluation metrics with the inclusion of PBNet. This is because modeling the long-term dependency of pose and blink movements through PBNet simplifies training for the A2V-FDM. We also visualized the effectiveness of PBNet in Appendix A.2.4.

### 4.6 Pose/blink controllable generation

In addition to generating lifelike avatars, our method also enables the controllable generation of pose and blink actions. Users can either use pose and blink information generated by our PBNet or provide these sequences directly, such as by extracting them from a given video. The results, as shown in Figure 3, demonstrate that our method not only provides high-precision control over the pose/blink movements of the generated avatars, but also effectively transfers rich facial dynamics, including expressions, blinks, and lip motions.

Figure: Figure 3: Visualization of cross-identity reenactment. We extract the audio, head pose, and blink signals from the video in the first row, and use them to drive the source image, generating the talking head video in the second row.
Refer to caption: x3.png

## 5 conclusion

We introduce DAWN, an innovative architecture that non-autoregressively generates dynamic frames of talking head videos from given portraits and audio. We utilized the LFG to extract motion representations from speech videos. To produce vivid talking head videos, we propose PBNet and A2V-LDM. The PBNet generates natural pose/blink movements from speech, while A2V-LDM produces motion representations conditioned on audio and pose/blink movements. Finally, these generated motion representations are decoded into videos using LFG. We demonstrate on two datasets that our model can generate extended talking head videos with high-quality dynamic frames in a single pass, achieving realistic visual effects, accurate lip synchronization, and strong extrapolation capabilities.

## Appendix A Appendix

### A.1 Social Impact consideration

DAWN focuses on creating realistic talking head videos with the aim of generating positive social impact. We firmly oppose the malicious misuse of our method, including fraud, creating fake news, and violating portrait rights. Thus, we assert that our open-source code and model should be used exclusively for research purposes. We hope that our technique can provide social benefits in the future, such as promoting education, enabling face-to-face communication for separated people, and treating certain psychological disorders.

Figure: Figure 4: The qualitative study on higher resolution ($256\times 256$) and different portrait styles.
Refer to caption: x4.png

### A.2 Additional experiments

#### A.2.1 Experiment on higher resolution and different portrait styles

We further investigate the generalization ability of our method on higher-resolution images and different portrait styles. Training DAWN on the HDTF dataset with a resolution of $256\times 256$, we tested it on multiple out-of-dataset source images featuring diverse styles, as showcased in Figure 4. The results indicate that our method yields promising outcomes in high-resolution generation and demonstrates considerable generalization ability across various image styles, including photos, paintings, anime, and sketches.

#### A.2.2 Comparison experiment on generation time cost

We experimented to test the time cost of video generation. Using the aforementioned methods, we generated 8-second talking head videos with the same source image and audio, then recorded the time consumption for each method. To ensure a fair comparison, we excluded the audio encoding step for all methods. The testing was performed on a single V100 16G GPU. As shown in Figure 5, our method achieves the fastest or near-fastest generation speed and requires significantly less time compared to the previous diffusion-based method, Diffused Heads.

Figure: Figure 5: The comparison experiment on generation time cost. The “*" refers to diffusion-based methods.
Refer to caption: extracted/5936384/time_cost_2.png

#### A.2.3 Ablation study on the local attention mechanism

In our work, we utilized a local attention mechanism to enhance the extrapolation capability of our model. We conducted experiments to evaluate the effect of varying the window size of the local attention mechanism in A2V-FDM, ranging from 20 to 200, and also assessed the model’s performance without the local attention mechanism. To eliminate the influence of the generated pose/blink, we used the ground-truth pose/blink signals to drive the model. The results, presented in Table 5, indicate that the model’s performance peaks around a window size of 80. This is attributed to the maximum length of the video clips during training, which is 40 frames. Consequently, the model learns to process temporal dependencies within a 40-frame distance. Thus, using the window size of twice the max training clip length takes full advantage of the model’s capability. Reducing or increasing the window size degrades performance; a smaller window size leads to a loss of contextual information, significantly impairing the model’s performance, while a larger window size or the absence of the local attention mechanism reduces extrapolation ability, also resulting in lower performance. Since extrapolation ability is also supported by our TCL strategy and the intrinsic structure of A2V-FDM, removing the local attention mechanism causes relatively minor damage compared to the loss of contextual information.

**Table 5: Ablation study on the local attention mechanism. The “window" means the window size in the local attention operation. The “None" means we use the original attention mechanism instead.**
| Window | FID$\downarrow$ | $\text{FVD}_{16}$$\downarrow$ | $\text{FVD}_{32}$$\downarrow$ | $\text{LSE}_{\text{C}}$$\uparrow$ | $\text{LSE}_{\text{D}}$$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| 20 | 14.47 | 159.19 | 217.54 | 5.69 | 8.97 |
| 40 | 10.93 | 72.93 | 114.52 | 6.35 | 8.33 |
| 80 | 9.68 | 52.05 | 87.11 | 6.71 | 7.99 |
| 200 | 9.44 | 53.48 | 88.84 | 6.60 | 7.94 |
| None | 9.70 | 63.95 | 103.83 | 6.37 | 8.15 |

#### A.2.4 Ablation study on the PBNet

We further provide the visualization of the ablation study on PBNet in Figure 6, while the quantitative results are illustrated in Table 4.
It suggests that using the PBNet to generate the pose exclusively will provide the pose with more vividness and diversity. However, generating lip, head pose, and blink movement from audio simultaneously will cause a relatively static head pose, which severely impacts the vividness and naturalness.

Figure: Figure 6: The visualization of the ablation study on PBNet demonstrates different methodologies. The term “w/o PBNet" indicates “without PBNet", whereby the A2V-FDM is utilized to infer pose and blink movements. Conversely, “w PBNet" signifies the “with PBNet", which directly generates explicit pose and blink signals to control the generation of A2V-FDM.
Refer to caption: x5.png

### A.3 Limitation and future works

Our work still has certain limitations. For instance, the model cannot fully comprehend the physical common sense during the generation, particularly when individuals in the portraits wearing items such as hats, helmets or headpieces. Sometimes, these items will not move along with the head, thus causing artifacts in the results. We leave the injection of these physical dependency into the model without the loss of vividness as our future work.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see above.
soundness: 3
strengths: The paper is well written and easy to follow

First non-autoregressive diffusion-based approach for talking head generation
summary: This work presents a non-autoreregrssive diffusion based approach for talking head generation. This allows the generation of videos with non-fixed length. To enhance the performance the authors also decouple the motion of lips, head, and blinks and also introduce a two stage curriculum learning strategy. The proposed model is evaluated on the CREMA and HTDF datasets achieving state-of-the-art results.
weaknesses: Why is the proposed approach compared against wav2lip? Wav2lip only generates the mouth region, therefore the comparison is not really fair. The authors are encouraged to explain in the paper this decision and clearly acknowledge this limitation.

The authors use the syncnet model for measuring the quality of the generated lip movements. Why not using a lipreading model? It would be a more accurate measure of the lip movement quality. An explanation why syncnet is preferred over lipreading models would be very useful. Alternatively, the authors could use some public lipreading models to evaluate the performance (e.g., AutoAVSR, Raven, AV-Hubert).

The authors use several quantitative metrics to evaluate the model's performance and this is good. However, a user study is missing. There are no quantitative metrics which correlate highly with human perception, therefore evaluating the performance of generative models via user studies is highly desirable. The paper would be much stronger if a user study is included. Otherwise, the authors should explain why it's not included.

Table 1, why are some results underlined? This should be explained in the captions.

On which dataset is the LFG trained? Is it trained on each dataset separately? Also, what's the impact of pre-training? Has the option of training the LFG component jointly with the rest of the model been considered (and if yes can some results be presented)? The paper would be stronger if the author provide additional details regarding the above questions.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. I have a question regarding the ablation study on the TCL strategy: when the model was trained only with stage 1 and only with stage 2, did you maintain the same number of training epochs and steps as in the proposed method? I am curious to know whether the effectiveness of the two-stage curriculum learning is due to the training time or the strategy itself.

New after rebuttal: I raised the soundness score from 1 to 2 and the rating from 3 to 5.
soundness: 2
strengths: 1. The paper is well written and easy to read. 
2. The proposed method can performed nearly in real time on a single V100 16G GPU.
summary: This paper proposes a diffusion-based non-autoregressive talking head synthesis method that operates with input from only a source portrait and an audio sequence. The method is performed in a two-stage manner: a Latent Flow Generator (LFG) serves as the video reenactment model, and a diffusion model together with PBNet serve as the motion generation models. To achieve non-autoregressive synthesis (NAR), PBNet generates the entire blink and pose sequence at once. The authors also propose a two-stage curriculum learning strategy to enhance the efficiency and performance of the diffusion model's learning process.
weaknesses: 1. I believe the proposed method lacks novelty. The overall architecture and the two-stage approach, which involves training a reenactment model first and then generating motion latents, have already been proposed in many existing works, such as TH-PAD[1], GAIA[2], VASA-1[3], AniTalker[4], and so on.. Although the authors attempted to innovate from a non-autoregressive perspective, I believe that their discussion and argumentation regarding the advantages of NAR are not sufficiently robust, and I will state my reasons below.
2. I don't think the proposed NAR method and experiments conducted demonstrate the authors' claim:  
a) Generating only blinks and poses through PBNet is not sufficient; the expression-related motion is still generated from the diffusion branch with audio input as a condition.  
b) The authors claim that there is error accumulation in long video sequences of existing AR and SAR models, but no comparative experiments have been conducted between the proposed NAR method and existing methods to prove this claim.  
c) Given that the authors consider NAR as the main contribution of this paper, the paper does not adequately discuss the advantages of NAR over AR and SAR, nor does it provide reasonable comparative metrics to substantiate these claims. Merely comparing general metrics such as image quality, lip sync, identity, and motion matching is not sufficient to demonstrate that the performance improvements are due to the adoption of NAR.  
3. Methods like Anitalker[4], TH-PAD[1], and VASA-1[3], etc., have similar overall architectures with the proposed one, but the authors did not illustrate the differences between the proposed method and these methods.
4. Recently, many diffusion-based talking head synthesis methods have been proposed, such as AniPortrait[5], Anitalker[4], Hallo[6], EchoMimic[7], FollowYourEmoji[8], and so on. Most of them operate in an auto-regressive manner. The authors did not compare the overall performance with these methods; therefore, there is no adequate evidence to prove the performance superiority of the proposed method.

Generally, I think the authors should conduct more experiments and propose more reasonable metrics to prove the effectiveness of the proposed method. And considering the main contribution is the non-autoregressive approach, the authors need to adequately discuss its advantages over AR and SAR. Based on the above reasons, I have doubts that this submission meets the bar for publication.

If the authors could discuss the advantages of NAR more thoroughly and supplement with sufficient experiments to prove that the proposed method has better effects compared to existing diffusion methods, I would raise my rating.

[1] Yu, Zhentao, et al. "Talking head generation with probabilistic audio-to-visual diffusion priors." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.  
[2] He, Tianyu, et al. "Gaia: Zero-shot talking avatar generation." arXiv preprint arXiv:2311.15230 (2023).  
[3] Xu, Sicheng, et al. "Vasa-1: Lifelike audio-driven talking faces generated in real time." arXiv preprint arXiv:2404.10667 (2024).  
[4] Liu, Tao, et al. "AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding." arXiv preprint arXiv:2405.03121 (2024).  
[5] Wei, Huawei, Zejun Yang, and Zhisheng Wang. "Aniportrait: Audio-driven synthesis of photorealistic portrait animation." arXiv preprint arXiv:2403.17694 (2024).  
[6] Xu, Mingwang, et al. "Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation." arXiv preprint arXiv:2406.08801 (2024).  
[7] Chen, Zhiyuan, et al. "Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions." arXiv preprint arXiv:2407.08136 (2024).  
[8] Ma, Yue, et al. "Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation." arXiv preprint arXiv:2406.01900 (2024).

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - The latent flow generator is effectively a video-driven talking head generation method. It would have been interesting to see how it perform against similar methods, e.g. [7].

- Notation issue: in eq .6 the input are x_src,y_1N and p_1N. in the conditioning paragraph and in the figure it is stated that instead the inputs are in fact Z_src,a_1N and p_1N. Also p_1N is not defined when it's used in eq.6. Both of these things need to be corrected.

[7]: Y. Wang, D. Yang, F. Bremond and A. Dantcheva, "LIA: Latent Image Animator," in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2024.3449075.
soundness: 3
strengths: - The method works directly on the image by using its own latent flow generator. This is good because the network doesn't need to rely on other intermediate representations (e.g. 3dmm extracted from the video) which can already contain errors leading to bad generation.

- DAWN is able to generate pose and blink directly from the audio which is something not many methods do.

- By generating the head pose and blinking parameters outside the diffusion network and using them as condition instead the authors make their method more controllable.

- Qualitative results look good even if in low resolution.

- The authors provided video results for qualitative evaluation.

- The ablation study is extensive.
summary: The paper presents a new method for talking head generation called DAWN. This networks contains three main parts: a latent flow generator to drive a source image using latent flows, an "audio to latent flow" diffusion network and a pose and blink generation network. The network first predict latent flows from an audio sequence and pose and blink parameters. Those latent flows are then used by the latent flow generator to project the motion to RGB space using a source image containing the new identity. The pose and blink parameters are generated by the pose and blink generation network. DAWN outperforms the method used in the comparison on most metrics quantitatively and also qualitatively. The generated head pose and blink can be replaced by real one extracted from video for better control on the generation. Extensive ablation study is provided.
weaknesses: - The main issue I have with the paper is that the method used as baseline for the comparison are not state of the art: Wav2Lip, MakeItTalk and Audio2Head are from 2020-2021 while SadTalker and DiffusedHeads are from 2023. There are more recent method that could have been used for the quantitative comparison [1,2], there also exist diffusion based method that perform much better than DiffusedHeads [3]. Additionally several recent methods, for which the code is not available, could have been considered only for the qualitative results due to their impressive performances [4,5,6].

- On the same subject DiffusedHeads is known to perform badly on out of distribution data. It was trained on the MEAD dataset and tend to fail if the background is not green.

- The method is able to generate head pose but both datasets used for experiments exhibit limited head poses. It would have been better to use the VoxCeleb or CelebV-HQ datasets to evaluate the quality of the generated head poses.

- Most experiments are performed with a resolution of 128*128 when state of the art usually use 256X256 or higher (notably for HDTF that is often used at 512X512).

- In table 1 the LSEc and LSEd of wav2lips are not in bold/underlined when they should, they are often the best.

- For the video results provided the authors should make it clear whether the head pose was generated or taken from the ground truth. In some examples (e.g. driven_by_music.mp4) it look really impressive but I suspect it come from the ground truth since the results in the comparison video, while good, are not as impressive.

- It is still not clear how the non-autoregressive part is used during inference. Are the sequences generated only at a chosen size (e.g 200 for HDTF)? Or are several sequence generated then put back to back to make long videos (like the ones from the supplementary material). If it is the latter, since the method is non-autoregressive how do the authors avoid discontinuity between sequences with respect to the head pose?

- In table 4 the authors show that PBNET improve lips synchronization (LSEd and LSEc) by a lot but do not explain in details why this is happening. While it's true that the training become more difficult as a result there is still a loss focusing on lips so the lip motion should stay about as good in my opinion.

- I understand why the authors perform two training with different sequence length and the ablation shows that it improves results. However I don't see why they set X_src=X_1 for the first training instead of using a random starting frame like they do in the second training. Could the authors clarify their choice on this?

- The paper mention that a GAN loss is used inside PBNET but does not say how it is applied. Is there a discriminator? This part need to be clarified.

- The paper state that AR and SAR "leads to constrained performance and potential error accumulation, especially in long video sequences" but only cite one paper to support that claim. Most other methods mentioned in the paper or the one I proposed seem fine in that regard. The authors should develop this part with more examples.

[1]: Tan, Shuai, Bin Ji, and Ye Pan. "Style2talker: High-resolution talking head generation with emotion style and art style." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.

[2] S. Wang et al., "StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 6, pp. 4331-4347, June 2024, doi: 10.1109/TPAMI.2024.3357808

[3]: Ma, Y., Zhang, S., Wang, J., Wang, X., Zhang, Y., Deng, Z.: Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767 (2023)

[4]: Xu, Sicheng, et al. "Vasa-1: Lifelike audio-driven talking faces generated in real time." arXiv preprint arXiv:2404.10667 (2024).

[5]: Zhang, Bingyuan, et al. "Emotalker: Emotionally editable talking face generation via diffusion model." ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024

[6]: Tian, Linrui, et al. "Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions." arXiv preprint arXiv:2402.17485 (2024).

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Did the paper conduct a user study? I don’t seem to have seen one. User studies are very important in this field.
soundness: 3
strengths: 1. The performance is decent
2. Talking head generation is a research field with practical value.
summary: The paper introduces DAWN, a system that uses non-autoregressive (NAR) diffusion models to generate talking head videos of varying lengths from portrait images and audio. This method achieves faster processing speeds and high-quality outputs. To overcome limitations in NAR approaches and improve the modeling of longer videos, the system separately controls the movements of the lips, head, and blinking. This provides more precise control over these individual movements. The paper proposes PBNet, a network that generates realistic head poses and blinking sequences directly from audio clips using an NAR approach. The Two-stage Curriculum Learning (TCL) strategy is introduced to train the model effectively in generating lip movements and controlling head poses and blinks accurately, which helps in achieving robust convergence and better extrapolation capabilities.
weaknesses: 1. The motivation for introducing NAR is relatively weak. VASA[1], which uses SAR, already achieves real-time performance with good results. The motivation to use NAR to improve inference speed is weak, as the paper is only faster than diffused head, but the slower speed of diffused head is due to it generating images directly rather than intermediate representations.
2. The novelty of this paper is limited, as many modules are similar to previous methods. For example, the LFG is similar to the FOMM[2] series of work, and pose prediction is similar to SadTalker[3]. Although the paper introduces some novel training techniques like TCL, previous methods have already achieved good results without relying on such fancy techniques. Therefore, the significance of these fancy techniques is questionable.

[1] Xu, Sicheng, et al. "Vasa-1: Lifelike audio-driven talking faces generated in real time." arXiv preprint arXiv:2404.10667 (2024).

[2] Siarohin, Aliaksandr, et al. "First order motion model for image animation." Advances in neural information processing systems 32 (2019).

[3] Zhang, Wenxuan, et al. "Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Meta-review: This paper received mixed scores: one borderline accept, two borderline reject, and one accept. Most reviewers acknowledged the simple and effective solution proposed in the paper. Specifically, the paper introduces the first non-autoregressive diffusion-based approach for talking head generation, which was recognized as a novel contribution by the reviewers. However, some reviewers raised concerns, including the high similarity to existing methods such as Anitalker, TH-PAD, and VASA-1, limited original contributions, and restricted practicality of the approach. The AC has reviewed the paper, the reviews, and the rebuttal. While there have been numerous papers on generative model-based talking face synthesis, such as VASA-1, the application of non-autoregressive diffusion (NAD) to this task is novel and was acknowledged by the reviewers. Furthermore, the paper is well-written and presents its ideas clearly. Therefore, the AC suggests acceptance. It would be even better if the authors addressed all the reviewers' concerns in the final version.

Decision: Accept (Poster)