DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation
Authors: Hanbo Cheng
Limin Lin
Chenyu Liu
Pengcheng Xia
Pengfei Hu
Jiefeng Ma
Jun Du
Jia Pan
Venue: ICLR 2025 Poster (2025)
Keywords: Talking head generation
Non-autoregressive generation
Avatar
Video generation
Diffusion model
Primary area: generative models
TL;DR: We introduce the first non-autoregressive diffusion-based solution for high-quality, fast and general talking head video generation.

Abstract: Talking head generation intends to produce vivid and realistic talking head videos from a single portrait and speech audio clip. Although significant progress has been made in diffusion-based talking head generation, almost all methods rely on autoregressive strategies, which suffer from limited context utilization beyond the current generation step, error accumulation, and slower generation speed. To address these challenges, we present DAWN (\textbf{D}ynamic frame \textbf{A}vatar \textbf{W}ith \textbf{N}on-autoregressive diffusion), a framework that enables all-at-once generation of dynamic-length video sequences. Specifically, it consists of two main components:  (1) audio-driven holistic facial dynamics generation in the latent motion space, and (2) audio-driven head pose and blink generation. Extensive experiments demonstrate that our method generates authentic and vivid videos with precise lip motions, and natural pose/blink movements. Additionally, with a high generation speed, DAWN possesses strong extrapolation capabilities, ensuring the stable production of high-quality long videos. These results highlight the considerable promise and potential impact of DAWN in the field of talking head video generation. Furthermore, we hope that DAWN sparks further exploration of non-autoregressive approaches in diffusion models. Our code will be publicly available at \url{https://github.com/Hanbo-Cheng/DAWN-pytorch}.

Published as a conference paper at ICLR 2025
DAWN: DYNAMIC FRAME AVATAR WITH NON- AUTOREGRESSIVE DIFFUSION FRAMEWORK FOR TALK- ING HEAD VIDEO GENERATION
Hanbo Cheng' Limin Lin' Chenyu Liu?, Pengeheng Xia², Pengfei Hu! Jiefeng Ma', Jun Du Jia Pan"
Uniiersiity of Science and Technology of China, 2iFLYTEK Research
Cttpa::/haabbo-chenggggiii io.ooioo///o DAWN
ABSTRACT
Talking head generation intends to produce vivid and realistic talking head videos from a single portrait and speech audio clip. Although significant progress has been made in diffusion-based based talking head generation, almost all methods rely on autoregressive strategies, which suffer from limited context utilization beyond the current generation step, error accumulation, and slower generation speed. To address these challenges, we present DAWN (Dynamic frame Avatar With Non-autoregressive diffusion), framework that enables all-af-once generation of dynamic-length video sequences. Specifically, it consists of two main components: (1) audio-driven holistic facial dynamics generation in the latent motion space, and (2) audio-driver head pose and blink generation. Extensive experiments demonstrate that our method generates authentic and vivid videos with precise lip motions, and natural pose/blink movements. Additionally, with a high generation speed, DAWN possesses strong extrapolation capabilities, ensuring the stable production of high-quality long videos. These results highlight the considerable promise and potential impact of DAWN the field of talking head video generation. Furthermore, hope that DAWN sparks further exploration of non-autoregressive approaches in diffusion models. Our code will be publicly available at httpp: Chanoe//Chhng//yyyn pytoch
INTRODUCTION
Talking head generation aims synthesizing realistic and expressive talking head from given portrait and audio clip, which is garnering growing interest due to its potential applications in virtual meetings, gaming, and film production. For talking head generation, it essential that the lip motions in the generated vidco precisely match the accompanying speech, while maintaining high overall visual fidelity (Guo al. 2021a). Furthermore, natural coordination between head pose, eye blinking, and the rhythm of the audio is also crucial for convincing output (Liu et al. 2023).
Recently, Diffusion Models (DM) (Ho et al. 2020) have achieved significant success video and image generation tasks (Rombach et al.] 2022 Ho et al. 2022b Peebles & Xie 2023 Ni et al. 2023). However, their application in talking head generation (Shen et al., 2023] Bigioi et al 2024) still faces several challenges. Although many methods yield high-quality results, most of them rely on autoregressive (AR) (Tian et al. 2024 Ma et al.) 2023) or semm-auttoregressive (SAR) (Xu et al. 2024c| He et al. 2023) strategies. In each iteration, AR generates one frame, while SAR generates fixed-length video segment. The two strategies significantly slow down the inference speed and fail to
adequately utilize contextual information from future frames, which leads to constrained performance
and potential error accumulation, especially in long video sequences (Stypulkowski et al. 2024 Tian
al. 2024 Xu al. 2024b).
AAthorr contributed equally to this research, work done as interns at iFLYTEK Research. Conresponding author: Jun Du (jundu @ @ust..ddu.cc...

---

Published as a conference paper at ICLR 2025
To address these challenges, we present DAWN (Dynamic frame Avatar With Non -on-eutoreeressivve diffusioon, a novel approach that significantly improves both the quality and efficiency of talking head generation. Our approach leverages the DM to generate motion representation sequences from given audio and portrait. These motion representations are subsequently used to reconstruct the video. Unlike other methods, approach produces videos of arbitrary length in non-autoregressive (NAR) manner. However, employing the NAR strategy to generate long videos often results in either over-smoothing or significant content inconsistencies due limited extrapolation (Qiu et al. 2023). In the context of talking head generation, we suggest that the model's 's temporal modeling capability significantly hindered by the strong coupling relationship among multiple motions. Typically, the motions in talking head include (1) lip motions and (2) head pose and blink movements. The temporal dependency of head and blink movements extends over several seconds, far longer than that of lip motions (Zhou et al. 2020). Training models to capture these long-term dependencies requires extensive sequences, thus increasing the difficulty and cost of training. Fortunately, head and blink movements can be represented as ow-dimensional vectors (Zhang et al. 2023), enabling the design of a lightweight model that learns these long-term dependencies by training on extended sequences. Thus, to further enhance the temporal modeling and extrapolation capabilities of DAWN, we disentangle the motion components involved in talking head videos. Specifically, we use the Audio-to- Video Flow Diffusion Model (A2V-FDM) to learn the implicit mapping between the lips and audio, while generating the head pose and blinks via explicit control signals. Additionally, we propose lightweight Pose and Blink generation Network (PBNet) trained on long sequences, to generate natural pose/blink movements during inference in NAR manner. In this way, we simplify the training of A2V-FDM as well as achieve the long-term dependency modeling of the pose/blink movement. To further strengthen the convergence and extrapolation capabilities of A2V-FDM, we propose a two-stage training strategy based on curriculum learning to guide the model in generating accurate lip motion and precise pose/blink movement control.
The main contributions of this work are as follows: 1) We present DAWN (Dynamic frame A Avatar With Non- -on-autoregressive diffusion) for generating dynamic- talking head videos from portrait images and audio clips in a non-autoregressive (NAR) manner, achieving faster inference speeds and high-quality results. To the best of our knowledge, this is the first NAR solution based on diffusion models designed for general talking head generation. 2) To compensate for the limitations of extrapolation in NAR strategies and enhance the temporal modeling capabilities for long videos, we decouple the motions of the lips, head, and blink, achieving precise control over these movements. 3) We propose the Pose and Blink generation Network (PBNet) to generate natural head pose and blink sequences exclusively from audio in a NAR manner. 4) We introduce the Two-stage Curriculum Learning (TCL) strategy to guide the model in mastering lip motion generation and precise pose/blink control, ensuring strong convergence and extrapolation ability.
2 RELATED WORKS
Audio-driven talking head generation. Initial approaches for talking head generation employed deterministic models to map audio to video streams (Fan et al. (0066), with later methods introducing generative models such as GANs (Isola et al. 2016a). VAEs (Kingma & Welling 2022). and diffusion models (DMs) (Ho et al. 2020). GAN-related methods (Vougioukas et al. 22200) Pumarola et al. 2018 Hong et al.) 2022) improved visual realism but faced convergence and mode collapse issues (Xia et al.; 2022). Following this, VAEs (Kingma & Welling 2022) generated 3D priors like 3D Morphable Models (3DMM) (Blanz& & Vetter 1999) followed by high-fidelity rendering (Ren et al., 2021) }. which limited the realism and vividness. In contrast, DMs have been introduced into talking head generation due to their good convergence, excellent generation performance, and diversity. Stypuikowski et al. (2024) presented a DM-based talking head generation solution using an AR strategy to generate videos frame-by-frane -ramme iteratively. Subsequently, Tian et al. (2024) improved this AR strategy by incorporating motion conditions extracted by a VAE-based network as priors for each iteration, effectively mitigating degradation issues. Concurrently, Xu et al. (202cc); He et al.) (2023) advocated for motion modeling instead of image modeling. They utilized DMs to iteratively generate latent motion representations over fixed number of frames in SAR manner, subsequently converting these motion representations into video frames. While most diffusion- based methods produce promising results, their AR or SAR strategies incur slow generation speeds and collapse in long-video generation. Although methods like Tian et al. (2024); Xu et al. (2024c) alleviate issues such inconsistencies in content across iterations and long video generation collapse, the risk of
2

---

Published as a conference paper at ICLR 2025
error accumulation remains unresolved. While methods like Du et al. (0233 use ddentity-speciine NAR strategies, to the best of our knowledge, none have addressed NAR talking head generation for arbitrary identities. Consequently, we propose novel NAR dynamic frame generation framework based on DM, which aims to achieve the low-cost and high-quality rapid generation of realistic talking head video through clip of audio and arbitrary portrait.
Audio-driven pose and blink generation. Head pose and blink movements significantly impact the naturalness of talking head videos. However, the mapping from audio to pose and blink movement is one-to-many problem, which presents a significant challenge (Xu et al. 2024a Chen et al. 2020). Early works primarily focused on controlling poses directly using facial landmarks or video references (Zhou et al., 202T; Guo et al. 2021b). However, these approaches require additional guidance information, which impairs the diversity of the results. Later studies considered generating both pose and blink within the context of talking head generation (Zhou et al.] (020). However, simultaneously generating all facial movements can cause interference and ambiguity (Zhang et al., 2023). Therefore, some works attempted to decouple the speaker's s actions into components like lip, head pose, and blink, using discriminative models to predict these conditions separately (Wang et al. 2021 He et al. 2023). Later, researchers recognized that probabilistic modeling is better suited for the one-to-many mapping relationship, leading to the proposal of a VAE- based pose generation framework (Liu et al. 2023). However, most existing pose generation strategies also depend AR or SAR approaches, negatively impacting efficiency, smoothness, and naturalness. To address these issues, we design a VAE-based NAR pose generation method to produce vivid and smooth pose and blink movements while maintaining the NAR generation of the entire framework.
3 METHOD
As shown in Figure 1 DAWN is divided into three main parts: (1) the Latent Flow Generator (LFG); (2) the conditional Audio-to- Video Flow Diffusion Model (A2V-FDM): and (3) the Pose and Blink generation Network (PBNet). First, we train the LFG to estimate the motion representation between different video frames in the latent space. Subsequently, the A2V-FDM is trained to generate temporally coherent motion representation from audio. Finally, PBNet used to generate poses and blinks from audio to control the content in the A2V-FDM. To enhance the model's extrapolation ability while ensuring better convergence, we propose a novel Two-stage Curriculum Learning (TCL) training strategy. We will first discuss preliminaries, then present the specific details of DAWN's three main components, namely LFG, A2V-FDM, and PBNet in Sections 3.2 3.3 and 3.4 respectively.
Finally, we will introduce the TCL strategy in Section 3.5
3.1 PRELIMINARIES
Task definition. The task of talking head generation involves creating natural and vivid talking head video from two inputs: a static single-person portrait, &rse and a speech sequence, V1.N = {yo. ....... yN }. The static image Iam and the speech sequence y can originate from any individual, and the output is 21.N = {āo, i1..... 3N}. }. where N represents the total number of frames.
Diffusion models. Diffussin models generate samples conforming progressively denoising Gaussian noise (Ho et al. 2020). Let 2g represent data sampled from a given distribution (Zx)). In the forward diffusion process, Gaussian noise is progressively added to Zn after T steps, resulting in noisy data ær (Nichol & Dhariwal 2021 Song et al. 2020), and the conditional transition distribution at each step is defined as:
given data distribution by
q<x:(20)= = Na 20, (1 a))
where ar = -I'= 0q The reverse diffusion process gradually recovers the original data from the Gaussian noise Zy ~ (0, I), utilizing a neural network to predict po (x,-1/z;), where 0 represents the parameters of the neural network. The model is trained using the following loss function:
Lsimple B [||c c co (x,,, []
where is the Gaussian noise added to zu in the forward diffusion process to obtain æ., and ea (x t) is the noise predicted by the model. In video-related tasks, the denoising model can be implemented
via 3D U-Net (Ho et al. 2022b Çiçek et al. 2016).
3

---

Published as a conference paper at ICLR 2025
Latent Space
Reperamertiation
AGAN
Fazss
Xarc
Usear
Transformer Encoder
Fan
Posittonal Encoading
Riok
Eneading
4PL
Adri
Wart
ggea
(a) Latent Flow Generator (LFG)
(b) Pose and Blink generation Network (PBNet)
Warp
Image
Asrc
TRAIN
GT pose/bilnk /blin
audio
Deooder
Forward Diffusion Process
Process
P
FaON
Dencking Udde
R
TRAIN
(c) The structure of the Dynumic frame Avalur With Non-Autoregrresive Diffusion Framework (DAWN)
Figure 1: The pipeline of DAWN. First, we train the Latent Flow Generator (LFG) in (a) extract the motion representation from the video. Then the Pose and Blink generation Network (PBNet) in (b) is utilized to generate the head pose and blink sequences of the avatar. Subsequently, the Audio-to- Video Flow Diffusion Model (A2V-FDM) in (c) generates the talking head video from the source image conditioned by the audio and pose/blink sequences provided by the PBNet.
3.2 LATENT FLOWW GENERATOR
The Latent Flow Generator (LFG) is self-supervised training framework designed to model motion information between the source image sue and the driving image zani As illustrated in Figure (a), (a),
LFG consists of three trainable modules: the image encoder E, the flow predictor ',, and the image decoder During training, Zare; Tui Hx are images randomly selected from the same video. The image encoder & encodes the source image into latent code Zue E IR XW, XC,
The flow predictor estimates a dense flow map f and a blocking map m (Siarohin et al. 2021 2020 corresponding to Ese and Edi
f.m = P(I Fan)
The flow map 1 describes the feature-level movement of æai relative Tue in horizontal and vertical directions. The blocking map m E R xW, x1 ranging from 0 to 1, indicates the degree of area blocking the transformation from Base to Taii The flow map f used to perform the affine transformation A, serving map m guides the model in repairing the occlusion area, thereby serving as fine-graines repair. Finally, the image decoder D converts the warped latent code into the target image &gen where the @ the element-wise product:
coarse moarse-grained warping of zxe- Subsequently, the blocking
ggen D(A( Zare f) m)
The LFG is trained in an unsupervised manner and optimized using the following reconstruction loss:
LAFG Iywe (agen Tmmi
(5)
4

---

Published as a conference paper at ICLR 2025
where Lee is multi-resolution reconstruction loss derived from pre-trained VGG-19 19 network, used to evaluate perceptual differences between Egen and Dari (Johnson et al. 2016). We consider the concatenation of m and f xn [f. m to represent the motion of Taii relative to Exe-- In this way, we achieve two objectives: 1) finding an effective explicit motion representation z", which is ddentity-agnostie and well-supported by physical meaning, and 2) reconstructing Vaii from Eae and z" without the need for a full pixel generation.
3.3 CONDITIONAL AUDIO2VIDEO FL.w DIFFUSION MODEL
Through the LFG in Section 3.2 we can specify a Øsre- and extract the identity-agnoste motion representations =I.Ny of the talking head video clip V1:N well as the latent code Aare extracted by from E.e-- Therefore, we design the A2V-FDM to generate the motion representations 2/.N-
[fIN, miN] of each frame relative to Ixrc.
ENN (M(C( VINN: PI:N)
(6)
where the DM refers the diffusion model and PI:N is the pose/blink signal. After generating 2 use the decoder D in LFG to reconstruct the 2 to the target video frames, via the Equation 4 The structure of A2V-FDM is illustrated in Figure (c). The A2V-FDM model includes a 3D U-Net
denoising backhone (Ho et al. 20226). The residue block in the 3D U-Net contains temporal attention and spatial attention modules, which handle frame-level and pixel-level dependencies respectively.
The parameters of the temporal attention module are independent of the input length, so we believe that 3D U-Net can theoretically process video sequences of any length. However, to ease training difficulty, we train on short sequences and aim for Long-sequence inference. To enhance the 3D U-Net's 's extrapolation ability when handling long sequences, we used Rotary Positional Encoding (RoPE) (Su et al. 2023) instead of traditional absolute position embeddings in the temporal attention module.
Conditioning. We incorporate the following conditions control its generative behavior: audio embedding di.x, pose/blink signal PI:N, and source image latent code Zur. The audio embedding a1.,, extracted from the audio y i: using Hubert (Hsu et al. 2021), implicitly controls the lip motion. Due to the strict alignment with video frames, we apply the audio embedding to its corresponding image. Additionally, the avatar's 's pose and blink are controlled via explicit signals. The pose is described by 6D vector Ji et al. (2022). During training, the pose is extracted from video using an open-source tool (Guo et al. (020). For the blink signal, we adopt the aspect ratio of the left and right eyes, following (Zhang et al. 2023). To account for the arbitrary pose and eye-opening degree of the source image Esre. we use the difference between the current frame e; and the source frame Due: Ap. = P; Pare. which models the transition of state rather than the state itself. The model is provided with features zue to supply facial visual details. Each frame's 's latent code performs cross attention with the audio embedding a; and pose/blink information Ap;, respectively. This process injects these conditions into the latent code with different spatial weights. controlling specific regions of the generated content. The image feature z is regarded as a global condition and is concatenated directly with the noisy data as the initial input to the 3D U-Net. We also utilize landmarks to create a face region mask for Øares embedded with a lightweight convolutional network, similar to the approach by Tian et al. (2024). This mask adds to the denoising process in the same manner as Zae-
Loss function. We employed the DM regular denoising loss, Lsimple in Equation to train our model. The synchronization of lip motions with audio is crucial for the talking head task, while the lips often constitute only a small portion of the frame. Consequently, during training, we employed landmarks to isolate the lip region by generating lip mask, miip- We then applied additional weight, thpp to the denoising process of this region, similar to Styputkowski et al. (2024). Ultimately, our loss function is defined as:
L = Limple + Whip (Limpe X maap))
where the symbol & denotes element-wise product.
3.4 AUDIO-DRIVEN POSE AND BLINK GENERATION
To prevent the generated results from exhibiting overly monotonous and minimal movements, we design separate module, namely the Pose and Blink generation Network (PBNet). As shown in
5

---

Published as a conference paper at ICLR 2025
Figure (b), PBNet learns the mapping between audio and pose/blink movements. To maintain the non-autoreerressive (NAR) generation capabilities of A2V-FDM, PBNet employs a transformer based Variational Autoencoder ( (VAE) (Petrovich et al. 2021) to generate variable-length pose and blink sequences. The inputs to PBNet include the initial pose/blink state Pae. the residual pose/blink ApI:N, and the audio embedding d1:N- The transformer encoder & embeds these inputs into Gaussian-distribun and obtain latent code h E Rx through resampling:
μ. logo Ei(psoc API.N d1.N), s.t. Ni,, o))
(8)
We design h to have the same length Ap ensure it can carry sufficient information for variable- length inputs. The transformer decoder Dj generates the final pose/blink sequence A conditioned on a1N and Pare
A = Dh A1., Psre)
To enhance the model's extrapolation capability, we use RoPE the positional encoding in the decoder, consistent with A2V-FDM. During training. we apply an MSE-based reconstruction loss Lree and adversarial loss LGAN guide the model in completing basic reconstruction tasks (Isola jet al., 22016a] Ginosar et al. 2019). Additionally, we employ a KL divergence loss LKI. to ensure that the latent code h closely approximates a standard Gaussian distribution.
3.5 TWO-STAGE CURRICULUM LEARNING FOR TALKING HEAD GENERATION
Empirical evidence indicates that training our A2V-FDM model solely with fixed-length short video clips leads to inaccurate control of poses and blinks, as well as poor generalization to longer vidcos. We argue that one-step training approach impedes the model's convergence an optimal solution and fails to achieve satisfactory results in the complex task of talking head generation. To address these issues, we propose an innovative Two-Stage Curriculum Learning (TCL) strategy inspired by the theory of curriculum learning (Bengio et al., 2009).
Overall, the goal of the A2V-FDM during the training process can be expressed as:
11.N (E VI.N: PL:N))
(10)
In the first stage, we set age = 31, and the sequence length N = K" is a fixed, relatively small constant. This stage primarily focuses enabling the model to generate basic lip motions. However, utilizing æ1 as the source image often exhibits limited variations in poses and blinks, and using short clips result in scarcity of training samples with significant pose or blink movements. Therefore, in the second stage, we set aae E X randomly, where is the set of frames the entire video, to learn control capabilities of large pose transformation. Additionally, differing from stage one, we randomly set N E [Kmin, mmax]. Kmin > K'. This approach aims to enhance control over poses and blinks while maintaining precise lip motions, as longer clips contain more diverse pose and blink movements. Training with lengh sequences also helps the model avoid a bias towards lixethlength sequences, further enhancing the model's extrapolation.
3.6 INFERENCE
Our inference process has four steps: 1) Extract the audio embedding a1.N- 2) Use the source image Due to extract the initial pose/blink state Paee for PBNet. Along with a1:N and a latent space vector biNN sampled from a standard Gaussian distribution, PBNet generates the pose/blink sequences PI:N- 3) Input Uaee (1:N. and PI:N into the A2V-FDM, which generates the motion representation sequences 2T 4) Finally, decode the video sequence ăi.N from Twe and x Both PBNet and A2V-FDM generate sequences of dynamic lengths in single pass, depending input audio length.
Our method leverages non-autoregressive (NAR) generation during the inference process. To enhance extrapolation during inference, we utilize local attention (Luong 2015) in the temporal attention module for both the PBNet decoder and the 3D U-Net in A2V-FDM. which restricts the attention scores local region. This approach effectively models local dependencies in talking head videos. To accommodate the different temporal dependencies of lip motions and pose/blink movements, we use larger window size in the local attention mechanism of PBNet compared to A2V-FDM.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENT
4.1 SETUP
Dataset. Our method is evaluated on two datasets: CREMA (Cao et al. 2014) and HDTF Zhang al. 202T). The CREMA dataset was collected in a controlled laboratory environment and contains 7,442 videos from 91 identities, with durations ranging from I to 5 seconds. The HDTF dataset consists of 410 videos, with an average duration exceeding 100 seconds. These videos are gathered from wild scenarios and feature over 10,000 unique sentences for speech content, along with diverse head pose movements. We partitioned the CREMA dataset into training and testing sets following Stypuikowski et al. (2024). As for the HDTF dataset, we conducted a random split with a 9:1 ratio between the training and testing sets. We resized the videos at a resolution of 128 x 128 and a frame rate of 25 frames per second (fps) without any additional preprocessing.
Implementation details. The architecture of the encoder and decoder in our model aligns with the design proposed by Johnson et al. (2016), while the flow predictor is implemented based MRAA (Siarohin et al.] 2021). The PBNet model is trained using pose and blink movement sequences of 200 frames. During the inference phase of the PBNet model, a local attention mechanism with window size of 400 is employed. For the inference phase of the A2V-FDM model, local attention with window size of 80 is applied. In our evaluation, the length of one-time inference for CREMA is dynamic and depends on the ground truth, while for HDTF, it is fixed at 200 frames for better comparison.
Evaluation metrics. We evaluate the performance of our method using various metrics. Specifically, we employ the Fréchet Inception Distance (FID) (Heusel et al. 2017) to assess the image quality. We utilize the FsD1s and FVDg2 scores, which calculate the Fréchet Video Distance (FVD) based on window sizes of 16 and 32 frames, respectively, to evaluate video quality across different temporal scales. Furthermore, we assess the perception loss of lip shape using the confidence score (LSEc) and distance score (LSEp) (Chung & Zisserman 2017). To evaluate the preservation of speaker identity, we use ArcFace (Deng et al. 2019) to extract features from both the ground truth image and the generated image, and use the cosine similarity (CSIM) between the two as the evaluation metric. Moreover, we employ the Beat Aligm Score (BAS) (Siyao et al. 2022) to evaluate the synchronization of head motion and audio, and calculate the number of blinks per second (blink/s) to assess the liveliness of the eyes. To better illustrate the error accumulation from a quantitative perspective, design a metric to quantify the severity of error accumulation in the image space inspired by [Bian] al. 2022) in the sequential generation task: Degradation Rate (DR), defined as DR FID.,
FIl
DR IS related to the ratio between the FID of the last n frames FID and the first frames FID.t- The motivation for proposing this metric is that when error accumulation occurs, the quality of the generated data at the end of the sequence significantly deteriorates compared to the beginning. A larger DR indicates more severe eтror accumulation. In our experiments, we set n to 25 frames (Is) and 50 frames (2s), denoted as 2R and DR respectively.
4.2 OVERALL COMPARISON
We compared our method with several state-of-the-art methods: Wav2Lip (Prajwal et al. 2020).
Audio2Head (Wang et al. 2021), SadTalker (Zhang et al.) 2023). Diffused Heads (Stypuikowski ct al. 2024). Dream (Ma et al. 2023), Hallo (Xu et al. (2024)) and EchoMimic (Chen et al. 2024 The quantitative experimental results are shown in Table Our method achieves promising performance in FID, FVD1s. FVDsz, CSIM, BAS, and Blink/s metrics for the CREMA and HDTF datasets, as shown in Table It also attains nearly best scores for LSEş and LSEp. is important to note that EchoMimic and Dream are applicable only to face-aligned scenarios, whereas other methods do not require pre-cropping. Additionally, Hallo and EchoMimic are built upon pre-trained Stable Diffusion models (Rombach al.) (202),, inheriting substantial visual generation capabilities. Despite this, our method still achieves comparable or even better performance on the HDTF dataset. Furthermore, our method also outperforms Hallo and Echo.Mimic in terms of generation speed. In Appendix A.4.3) results indicate that our method achieves the fastest or near-fastest generation speed and requires significantly less time than Diffused Heads, Hallo, and EchoMimic.
For the qualitative experiment, as shown in Figure 2 we visualize the generation results for each baseline on different datasets. Our method evidently achieves the visual quality most similar to
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison with several state-of-the-art methods methods on HDTF Zhang et al. 2021) and CREMA (Cao et al., 2014) datasets. We use bold to indicate the best score and underline to represent the second-best score. Wav2Lip generated videos that only contain lip motions, while the rest remain still images. For the sake of rigor, consider it reference for the quality of lip motion and we will include Wav2Lip in the ranking. "T" indicates better performance with higher values, while "L" indicates better performance with lower values. For both BAS and Blink/s, we consider performance to better when they are closer the ground truth.
Method
FID
FVD1gl
FVD324
LSEct
LSEp4
CSIM
BAS
Blink/s
GT
5.88
7.87
0.192
0.24
CREMA
Audio2Head
29.58
188.54
208.44
5.13
7.92
0.660
0.274
0.01
SadTalker
16.05
101,43
158.85
5.57
7.36
0.808
0.244
0.33
Diffused Heads
13.01
64.27
116.18
4.56
9.26
0.673
0.185
0.26
Wav2Lip*
10.23
130.23
242.19
6.08
7.74
0.801
DAWN (ours)
5.77
56.33
75.82
5.77
8.14
0.845
0.231
0.29
GT
7.95
7.33
0.267
0.75
Audio2Head
30.10
122.26
205.42
6.88
7.58
0.705
0.290
0.09
SadTalker
26.11
97.43
187.43
6.27
8.03
0.767
0.297
0.47
HDTF
Wav2Lip*
23.85
166.15
281.73
7.42
7.44
0.701
DreamTalk
58.8
406.58
516.21
6.48
8.43
0.641
0.311
0.032
EchoMimic
32.8
139.00
178.16
6.69
8.27
0.731
0.318
0.121
Hallo
14.2
57.47
100.99
7.16
8.01
0.709
0.301
0.254
DAWN (ours)
9.60
60.34
95.64
6.71
7.94
0.790
0.281
0.86
Table 2: Comparison with other generation strategies. The semi-autoreerresivve (SAR) generation strategy is similar He al. (2023). The two temporal resolution (TTR) generation method is mentioned in Harvey et al. (022). For the DR metric, we consider performance to be better as approaches zero.
Method
Time(s)
FID
FVD164
FVDa24
LSEct
LSE
DR25
DR50
SAR
11.42
13.00
120.33
210.52
4.34
8.29
0.307
0.021
TTR
19.25
9.77
95.42
137.14
4.87
8.68
-0.028
-0.005
Ours
7.32
5.77
56.33
75.82
5.77
8.14
0.044
0.031
the ground truth, showcasing the most realistic and vivid visual effects. Compared to our method, SadTalker relies on the 3DMM prior, which limits its animation capability to the facial region only, resulting in significant artifacts when merging the face with the static torso below the neck. Additionally, SadTalker exhibits unnatural head pose movements and gaze direction, partially due to limited temporal modeling ability. Audio2Head fails to preserve the speaker's 's identity during generation. For the HDTF dataset, the Diffused Heads method collapsed due to the error accumulation. DreamTalk is only applicable to face-aligned scenarios, thus requiring the source image to be cropped. Hallo leads to serious error accumulation the CREMA dataset, resulting in abnormal color patches in the background, indicating robustness defect.
4.3 COMPARISON WITH OTHER GENERATION STRATEGIES
We compared our non-autoregressive generation strategy with two regular video generation strategies: semi-autoregressive (SAR) generation similar He et al. (2023), and 2) two-temporal resolution (TTR), which trains two models with different temporal resolutions (Harvey et al. 2022). The time cost represents the time required to generate an 8-second talking head video. The models were evaluated on the CREMA dataset, and the results shown in Table According the results, non-autoregressive method produces videos with the highest overall quality and fastest speed. In addition, the SAR method results in very high DR, which is at least order of magnitude higher
8

---

Published as a conference paper at ICLR 2025
Target
41
Ours
SadTalker
Audio2Head
Difflusod Heads
DreamTalk
Figure 2: Qualitative comparison with several state-of-the-art methods methods HDTF ZZang et al. 2021) and CREMA (Cao et al. (014) datasets. Our method produces higher-quality results in video quality, lip-sync consistency, identity preservation, and head motions.
Table 3:
The experiment of extrapolation evaluation. "Inference length" refers to the number of frames generated single inference process.
Inference length
FID
FVD1e4
FVDx24
LSEc
LSEp
40
9.35
59.58
94.09
5.76
7.89
100
9.83
61.72
98.80
6.41
7.96
200
9.60
60.34
95.64
6.71
7.94
400
10.36
61.57
97.84
6.63
8.12
600
10.30
60.44
96.62
6.76
8.02
than TTR and NAR. This is because autoregressive methods suffer from degradation issues during iterative generation. Although TTR also successfully alleviates the degradation issue, it compromises generation speed and overall quality. In summary, non-autoregressive method addresses the degradation problem while preserving fast generation speed and high overall quality.
4.4 EXTRAPOLATION VALIDATION
To evaluate the extrapolation ability of our method, we used the HDTF dataset to assess the impact of inference length on model performance, ranging from 40 to 600 frames. FID and FVD metrics remain stable with inference length, while longer audio improves lip movement precision. This suggests that inference with sufficient length helps the model produce more precise lip movement. Furthermore, to better compare the extrapolation of other AR-based DM methods, Hallo was selected for comparison. As shown in Table [ from the Appendix A.3 our method's 's DR metric stabilizes within a certain range, in contrast to Hallo's error accumulation increased inference lengths.

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study TCL and PBNet. The "GT PB" refers whether use ground truth pose/blink signal.
Method
GT PB
FID
FVD164
FVD324
LSEc
LSEp
only stage only stage 2 DAWN
7.95
81.84
126.52
4.38
10.04
13.71
125.75
166.83
6.14
8.43
9.68
52.05
87.11
6.71
7.99
w/o PBNet DAWN
X
15.20
100.94
162.35
5.79
8.36
9.60
60.34
95.64
6.71
7.94
4.5
ABLATION STUDY
Ablation study TCL strategy. Since TCL strategy is used only A2V-FDM, PBNet is excluded by using ground truth pose/blink for evaluation. In this section, we extend the number of training epochs based data throughput when using single-stage training. A2V-FDM trained with either stage or 2 separately shows that stage 1 decreases overall performance except for FID, due to shorter clip training causing minor warping and lower FID scores but poorer FVD. Stage improves LSEc and LSEp but results in worse FID and FVD as the model struggles with simultancous pose/blink control and lip movement. Both stages alone underperform compared to the proposed TCL strategy, underscoring its effectiveness.
Ablation study on PBNet. We evaluate the effectiveness of the PBNet in Table The term "w/o PBNet" indicates that the PBNet module was removed from the architecture, requiring the A2V-FDM to simultaneously generate pose, blink, and lip motions from the audio by itself. The results suggest an overall enhancement of all evaluation metrics with the inclusion of PBNet. This is because modeling the long-term dependency pose and blink movements through PBNet simplifies training for the A2V-FDM. We also visualized the effectiveness of PBNet in Appendix A.4.5
CONCLUSION
We introduce DAWN, innovative architecture that non-autoregressively generates dynamic frames of talking head videos from given portraits and audio. We utilized the LFG to extract motion representations from speech videos. To produce vivid talking head videos, propose PBNet and A2V-LDM. The PBNet generates natural pose/blink movements from speech, while A2V-LDM produces motion representations conditioned generated motion representations are decoded into videos using LFG. We demonstrate on two datasets that our model can generate extended talking head videos with high-quality dynamic frames in single pass, achieving realistic visual effects, accurate lip synchronization, and strong extrapolation capabilities.
audio and pose/blink movements. Finally, these
LIMITATION AND FUTURE WORKS
Our work still has certain limitations. For instance, the model cannot fully comprehend physical common sense during generation, particularly when individuals in portraits are wearing items such as hats, helmets, headpieces. Sometimes, these items do not move with the head, causing artifacts in the results. We aim to inject these physical dependencies into the model without sacrificing vividness in future work. Additionally, our architecture currently requires each sub-module to be trained separately, In future work, we plan to enable joint training to reduce error propagation across modules.
ACKNOWLEDGMENT
This work supported by the National Natural Science Foundation of China under Grant 62171427.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Xiaohang Bian, Bo Qin, Xiaozhe Xin, Jianwu Li, Xuefeng Su, and Yanfeng Wang. Handwritten mathematical expression recognition via attention aggregation based bi-directional mutual learning. In Proceedings of the AAAI conference artificial intelligence, volume 36, pp. 113-121, 2022.
Dan Bigioi, Shubhajit Basak, Michat Styputkowski, Maciej Zieba, Hugh Jordan, Rachel McDonnell, and Peter Corcoran. Speech driven video editing via an andio-conditioned diffusion model. Image and Vision Computing, 142: 104911, 2024.
Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '99. pp. 187-194, USA, 1999. ACM Weess//eddissonWWssey Publishing Co. ISBN 0201485605. 1556. htt.://p////aa..orr//..111////// http://...1111111111111 111.55..11553 311556
doi: 10.
Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390, 2014.
Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. Talking- head generation with rhythmic head motion. In European Conference on Computer Vision, pp. 35-51. Springer, 2020.
Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions, 2024. URL https
7/arxiv. otteiiiorrr///a http:/aas/220..000
Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pp. 251-263. Springer, 2017.
Özgün Çişek, Ahmed Abdulkadir, Soeren S Lienkamp. Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assiste Intervention 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, pp. 424 432. Springer, 2016.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690 4699- 2019.
Chenpeng Du, Qi Chen, Tianyu He, Xu Tan, Xie Chen, Kai Yu, Sheng Zhao, and Jiang Bian. Dae-talker: High fidelity speech-driven talking face generation with diffusion autoencoder. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 4281 4281-4289, 2023.
Bo Fan, Lei Xie, Shan Yang. Lijuan Wang, and Frank K Soong. A deep bidirectional Istm approach for video-realistic talking head. Multimedia Tools and Applications, 75:5287-5309, 2016.
Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learning individual styles of conversational gesture. CoRR, abs/1906.04 0660, 2019. UtLptt/:///aaa org/abs /1966 04160
Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Li. Towards fast, accurate and stable 3d dense face alignment. In European Conference on Computer Vision, pp. 162-168.
Springer, 2020.
Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5784-5794, 2021a.
Yudong Guo, Keyu Chen, Sen Liang, Yong- Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 5784-5794 2021b.
11

---

Published as a conference paper at ICLR 2025
William Harvey, Sacid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Infor- mation Processing Systems, volume 35, pp. 27953-27965. Curran Associates, Inc., wwwripuurucomm wttpiiisscccpppppe wwtpppprppppppceeeeeeeeeeeee. wttprr/felaapppae///0// Conference Coneereeeenccnrecrecccce..e pdff
2022. URL http:::/pproceeeddeeetp
E FieeDBeeleeedd0a000aaEEEE555E3
Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, Han Hu, et al. Gaia: Zero-shot talking avatar generation. arXiv preprint arXiv:2311. 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabillsttc models. Advances in neural information processing systems, 33:6840-653 6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arX1v:2210.02300, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646 2022b.
Fa- Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3397-3406, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460 2021.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-imann translation with conditional adversarial networks. CoRR, abs/1611.00 61.0000 2016a. URL http::t/t///i//// abs/16111 61666110004
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to to-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recog- .rg
nition (CVPR), pp. 5967-5976, 2016b. URL inttpa://aat semant. inemanticschho escho
Corpus II:
ID: 6200260
Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. Eamm: One- shot emotional talking face via audio-based emotion-aware motion model. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and reper-iesolution. CoRR, abs/1603.08155. 2016. URL httt:://aaxw...g////aaaaaa
08155
Diederik P Kingma and Max Welling, Auto-encoding variational bayes, 2022. URL
https
7/arxiv. org/ 6114
Yunfei Liu, Lijian Lin, Fei Yu, Changyin Zhou, and Yu Li. Moda: Mapping-once audio-driven portrait animation with dual attentions. Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 23020 2023.
Minh- Thang Luong. Effective approaches to attention- based neural machine translation. arXiv preprint arXiv: 1508 1500.04025. 2015.
Minh-Thang
12

---

Published as a conference paper at ICLR 2025
Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang. Yingya Zhang, and Zhidong Deng. Dreamtalk: When expressive talking head generation meets diffusion probabilistie models. arXiv preprint arXiv:2312.00767. 2023.
Arsha Nagrani. Joon Son Chung, and Andrew Zisserman. Voxceleb: A large-scale speaker identifica- tion dataset, In Interspecc 2017. ISCA, August 2017. doi: 10.21457/mteeppecc2201---9999 URL 7- 2017-950
http::/dx. dow. ww.. 21 1tt44//nnnerpeeec......
Haomiao Ni, Changhao Shi, Kai Li, Sharon X. Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models, 2023. URL https://arxiv
orc/abs/2233. 13744
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Mathis Petrovich, Michael J. Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer VAE. CoRR, abs/2104.05670 2021. URL https://arxiit org. abs/
2104. 05670
KR Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, and CV Jawahar. lip syne expert all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pp. 492, 2020.
Albert Pumarola, Antonio Agudo, Aleix M. Martinez, Alberto Sanfeliu, and Francese Moreno-Noguer. Ganimation: Anatomically-aware facial animation from a single image. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiv preprint arXiv:2310.1516,, 2023.
Yurui Ren, Ge Li, Yuanqi Chen, Thomas Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 13759-13768, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022.
Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1982-1991, 2023.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. CORR, abs/2003.00116 2020. URL https::/aaaatt
org/abs/ 2003. 00196
Aliaksandr Siarohin, Oliver J. Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. CoRR, abs/2104. 11280, 2021. https://:
org/abs/210s 11280
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 11050-11059, 2022.
2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
13

---

Published as a conference paper at ICLR 2025
Michat Stypuikowski, Konstantinos Vougioukas, Sen He, Maciej Zięba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face face generation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 5091-5100, 2024.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL httpt:tttt//attttt /tt.pp/s//arii..oorg/a////////// 09864
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait geilrrtinnnatingggg expres- sive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17488 2024.
Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial animation with gans. International Journal of Computer Vision (285))1398-1413, 2020.
Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. arXiv preprint arXiv:2107.09293; 2021.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: survey. IEEE transactions on pattern analysis and machine intelligence, 45(3): 3121-3138, 2022.
Chao Xu, Yang Liu, Jiazheng Xing, Weida Wang. Mingze Sun, Jun Dan, Tianxin Huang, Siyuan Li, Zhi-Qi Cheng, Ying Tai, et al. Facechain-imagineid Freely crafting high-fidelity diverse talking faces from disentangled audio. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1292-1302, 2024a.
Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang. Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation, 2024b. htt.::ttt////aattt.
org/ abs /20/6.24666.. 08801
Sicheng Xu, Guojun Chen, Yu-Xiao Guo. Jiaolong Yang. Chong Li. Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404. 2404:4667.6 2024c.
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8652-8666 8661, 2023.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changiie Fan. Flow-guided one-shot talking face generation with high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3661-3670, 2021.
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose- controllable talking face generation by implicitly modularized audio-visual representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4176- 4186. 2021.
Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttllk: speaker-aware talking-head animation. ACM Transactions On Graphics (TOG), 39(6): 1-15, 2020.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 SOCIAL IMPACT CONSIDERATION
DAWN focuses on creating realistic talking head videos with the aim of generating positive social impact. We firmly oppose the malicious misuse of our method, including fraud, creating fake news, and violating portrait rights. Thus, we assert that our open-source code and model should be used exclusively for research purposes. We hope that our technique can provide social benefits in the future, such as promoting education, enabling face-to-face communication for separated people, and treating certain psychological disorders.
A.2 TRAINING DETAILS
Since the HDTF dataset contains fewer videos of longer length compared to the CREMA dataset, each video in the HDTF dataset is sampled 25 times per epoch achieve better I/O performance. The LFG is fine-tuned independently on the training datasets of CREMA and HDTF, using a checkpoint pre-trained on the VoxCeleb (Nagrani et al. (017)) dataset. We fine-tune the LFG with 400 epochs. During the training of A2 we trained with 500 epochs for each stage. In both stages, we select the Whip 1 for the loss function in Equation 7 We freeze the parameters of LFG when training the A2V-FDM. Although end-to-end training can prevent errors in the LFG from affecting the A2V-FDM, the main reason for not adopting end-to-end training is the difficulty in achieving stable convergence. In an end-to-end training setup, the A2V-FDM is likely to receive incorrect supervision signals, which can hinder the training process of the model and make it highly unstable. For PBNet, the MSE-based reconstruction loss can be expressed as:
1 Σ (Ap,. APm))
MMSE =
(11)
where Ap is the predicted pose/blink sequences, and Ap is the ground truth sequences. We also implement KL divergence loss to constrain the latent code h closely approximates standard Gaussian distribution. The adversarial loss LGGAN is defined as:
LGAN arg min max (G,
(12)
where the G is our proposed PBNet and the D is a discriminator implemented based on PatchGAN (Isola et al. (0166b). We use a ID convolution based network to output the probability of each patch in the pose/blink sequence originating from real actions. The discriminator is guided by the BCE loss.
In the training, the loss function of PBNet is set
L Ave Lree + AKL + AGAN
(13)
where we set A = 1, AGAN = 0.6. During the training process, we observed that TRL rapidly converges to approximately zero at the beginning. This convergence impairs the latent state's ability to retain effective information, causing the model to depend almost entirely on the decoder's 's predictive capability for completing the fixed fitting task from audio pose/blink. To ensure diversity in generation, we implemented method of progressively increasing the λKL in training PBNet. The PBNet was trained over 1600 epochs. In the initial 400 epochs, we did not apply the LKI. constraint to the latent code., which helped the model develop basic sequence reconstruction capabilities in the early training phase. From that point until the end of training, AKL was gradually increased to 0.01.
A.3 ADDITIONAL EXPERIMENTS
A.3.1 EXTRAPOLATION COMPARISON
To highlight the advantages of NAR generation, we selected an image-based diffusion method, Hallo, for comparison, due to its representative structure similar to most of the latest AR or SAR methods. We evaluated the degradation rate for both our model and Hallo across inference lengths of 200, 400, and 600, as detailed in Table The results indicate that, in method, both the DR and DR metrics remain stable as inference length increases. In contrast, Hallo's ' DR and DR metrics significantly increase with longer inference lengths, suggesting notable error accumulation in the
15

---

Published as a conference paper at ICLR 2025
Source Image
Generated Frames
Image
Frames
Figure 3: The qualitative study on higher resolution (256 x 256) and different portrait styles.
Table The comparison experiment of error accumulation with Hallo. "Inference length" refers to the number of frames generated in single inference process.
DAWN
Hallo
Inference length
2R
DR50
DR25
5R50
200
0.253
0.043
0,214
0.094
400
0.208
0.164
0.279
0.161
600
0.152
0.152
0.422
0.332
image space. This demonstrates that image-based AR methods, such as Hallo, struggle with error accumulation in longer tasks, while our model exhibits superior extrapolation capabilities and more consistent performance with extended video sequences. These findings underscore the superiority of NAR methods compared to AR and SAR alternatives.
A.4 USER STUDY
The user study evaluates generated videos across four dimensions: 1) L-Syne: Lip-audio synchro- nization; 2) O-Nat: The overall naturalness of the generated talking head :3) M- Viv: The vividness of the head movements; 4) V-Qual: The overall video quality (e.g., presence of artifacts abnormal color blocks). We generated 10 test videos per method, with 23 participants scoring each on 1-5 scale. Users disregarded resolution and cropping when rating. As shown in Table [ our method surpasses existing approaches in lip synchronization, naturalness, and head movement vividness, and comparable EchoMimic in video quality.
A.4.1 POSE/BLINK CONTROLLABLE GENERATION
In addition to generating lifelike avatars, our method also enables the controllable generation pose and blink actions. Users either pose and blink information generated by our PBNet provide these sequences directly, such as by extracting them from a given video. The results, as shown in Figure 4 demonstrate that method not only provides high-precision control over the pose/blink
16

---

Published as a conference paper at ICLR 2025
Table 6: User study.
Method
L-Syne
O-Nat
M-Viv
V-Qual
Audio2Head
3.87
3.67
3.28
3.66
Sadtalker
4.23
3.13
2.81
4.14
Dream
4.38
3.89
3.41
4.42
Hallo
4.40
3.76
3.89
4.03
EchoMimic
4.45
4.30
4.06
4.53
D
4.57
4.41
4.43
4.51
Drive audio, pose, blink
Source image
Generated video
Figure 4: Visualization of cross-identity reenactment. We extract the audio, head pose, and blink signals from the video in the first row, and use them to drive source image, generating the talking head video in the second row.
movements of the generated avatars, but also effectively transfers rich facial dynamics, including expressions, blinks, and lip motions.
A.4.2 EXPERIMENT ON HIGHER RESOLUTION AND DIFFERENT PORTRAIT STYLES
We further investigate the generalization ability of method on nigher-resolution images and different portrait styles. We trained DAWN on the HDTF dataset with a resolution of 256 x 256. Then evaluated reglernne model the HDTF test set. The quantitative results are illustrated in Table 7 The results show that our method still maintains strong competitiveness compared to the latest talking head models. We also our method on multiple out-of-dataset source images featuring diverse styles, as showcased in Figure 3 The results indicate that our method yields promising outcomes high-resolution generation and demonstrates considerable generalization ability across various image styles, including photos, paintings, anime, and sketches.
A.4.3 COMPARISON EXPERIMENT ON GENERATION TIME COST
the time cost of video generation. We generated 8 8-second talking head videos with the same source image and audio, then recorded the time consumption for each method. To ensure fair comparison, we excluded the audio encoding step for all methods. The testing was performed on a single V100 16G GPU. As shown in Figure our method achieves the f'stest near- fastest generation speed and requires significantly less time compared to previous diffusion-based methods, Diffused Heads, Hallo, and EchoMimic.
We experimented
A.4.4 ABLATION STUDY ON THE LOCAL ATTENTION MECHANISM
In our work, we utilized a local attention mechanism to enhance the extrapolation capability of model. We conducted experiments to evaluate the effect of varying the window size of the local attention mechanism A2V-FDM, ranging from 20 to 200, and also assessed the model's performance without the local attention mechanism. To eliminate the influence of the generated pose/blink, we used the ground-truth pose/blink signals to drive the model. The results, presented in Table 8. indicate that the model's performance peaks around window size of 80. This is attributed
17

---

Published as a conference paper at ICLR 2025
Table 7: Quantitative study different resolutions. The "128" indicates method is trained 128 x 128 resolution, and the "256" indicates training on 256 x 256 resolution.
Method
FIDI
FVD164
FVD324
LSEct
LSEpL
CSIM
BAS
Blink/s
GT
7.95
7.33
0.267
0.75
DAWN (128)
9.60
60.34
95,64 105.20
6.71
7.94
0.790
0.281
0.86
DAWN (256)
11.80
68.07
7.20
7.80
0.791
0.278
0.73
1430.11
1400
1200
008.91
800
Time 60
600
Heads
Figure 5: The comparison experiment on generation time cost. The Diffused Heads, Hallo, EchoMimic are existing diffusion-basse methods.
to the maximum length of the video clips during training, which is 40 frames. Consequently. the model learns to process temporal dependencies within 40-frame distance. Thus, using the window size of twice the training clip length takes full advantage of the model's capability. Reducing or increasing the window size degrades performance; smaller window size leads to loss of contextual information, significantly impairing the model's performance, while larger window size or the absence of the local attention mechanism reduces extrapolation ability, also resulting in lower performance. Since extrapolation ability also supported by our TCL. strategy and the intrinsic structure of A2V-FDM, removing the local attention mechanism causes relatively minor damage compared to the loss of contextual information.
A.4.5 ABLATION STUDY ON THE PBNET
We further provide the visualization of the ablation study on PBNet in Figure while the quantitative results are illustrated in Table It suggests that using the PBNet to generate the pose exclusively will provide the pose with more vividness and diversity. However, generating lip, head pose, and blink movement from audio simultaneously will cause relatively static head pose, which severely impacts the vividness and naturalness.
18

---

Published as a conference paper at ICLR 2025
Table 8: Ablation study the local attention mechanism. The "window" means the window size in the local attention operation. The "None" means we use the original attention mechanism instead.
Window
FID4
FVD164
FVDs24
LSEct
LSEp4
20
14.47
159.19
217.54
5.69
8.97
40
10.93
72.93
114.52
6.35
8.33
80
9.68
52.05
87.11
6.71
7.99
200
9.44
53.48
88.84
6.60
7.94
None
9.70
63,95
103.83
6.37
8.15
w/o PBNet
PBNet
GT
Figure 6: The visualization of the ablation study PBNet demonstrates different methodologies. The term "w/o PBNet" indicates "without PBNet", whereby the A2V-FDM is utilized to infer pose and blink movements. Conversely, "w PBNet" signifies the "with PBNet", which directly generates explicit pose and blink signals to control the generation of A2V-FDM.
19


Reviews:
[{"review_id": "shqxbrhlRk", "reviewer": "Reviewer_FboU", "summary": "This work presents a non-autoreregrssive diffusion based approach for talking head generation. This allows the generation of videos with non-fixed length. To enhance the performance the authors also decouple the motion of lips, head, and blinks and also introduce a two stage curriculum learning strategy. The proposed model is evaluated on the CREMA and HTDF datasets achieving state-of-the-art results.", "questions": "Please see above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to follow\n\nFirst non-autoregressive diffusion-based approach for talking head generation", "weaknesses": "Why is the proposed approach compared against wav2lip? Wav2lip only generates the mouth region, therefore the comparison is not really fair. The authors are encouraged to explain in the paper this decision and clearly acknowledge this limitation.\n\nThe authors use the syncnet model for measuring the quality of the generated lip movements. Why not using a lipreading model? It would be a more accurate measure of the lip movement quality. An explanation why syncnet is preferred over lipreading models would be very useful. Alternatively, the authors could use some public lipreading models to evaluate the performance (e.g., AutoAVSR, Raven, AV-Hubert).\n\nThe authors use several quantitative metrics to evaluate the model's performance and this is good. However, a user study is missing. There are no quantitative metrics which correlate highly with human perception, therefore evaluating the performance of generative models via user studies is highly desirable. The paper would be much stronger if a user study is included. Otherwise, the authors should explain why it's not included.\n\nTable 1, why are some results underlined? This should be explained in the captions.\n\nOn which dataset is the LFG trained? Is it trained on each dataset separately? Also, what's the impact of pre-training? Has the option of training the LFG component jointly with the rest of the model been considered (and if yes can some results be presented)? The paper would be stronger if the author provide additional details regarding the above questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KhcRZJNcbh", "reviewer": "Reviewer_fcWS", "summary": "This paper proposes a diffusion-based non-autoregressive talking head synthesis method that operates with input from only a source portrait and an audio sequence. The method is performed in a two-stage manner: a Latent Flow Generator (LFG) serves as the video reenactment model, and a diffusion model together with PBNet serve as the motion generation models. To achieve non-autoregressive synthesis (NAR), PBNet generates the entire blink and pose sequence at once. The authors also propose a two-stage curriculum learning strategy to enhance the efficiency and performance of the diffusion model's learning process.", "questions": "1. I have a question regarding the ablation study on the TCL strategy: when the model was trained only with stage 1 and only with stage 2, did you maintain the same number of training epochs and steps as in the proposed method? I am curious to know whether the effectiveness of the two-stage curriculum learning is due to the training time or the strategy itself.\n\nNew after rebuttal: I raised the soundness score from 1 to 2 and the rating from 3 to 5.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to read. \n2. The proposed method can performed nearly in real time on a single V100 16G GPU.", "weaknesses": "1. I believe the proposed method lacks novelty. The overall architecture and the two-stage approach, which involves training a reenactment model first and then generating motion latents, have already been proposed in many existing works, such as TH-PAD[1], GAIA[2], VASA-1[3], AniTalker[4], and so on.. Although the authors attempted to innovate from a non-autoregressive perspective, I believe that their discussion and argumentation regarding the advantages of NAR are not sufficiently robust, and I will state my reasons below.\n2. I don't think the proposed NAR method and experiments conducted demonstrate the authors' claim:  \na) Generating only blinks and poses through PBNet is not sufficient; the expression-related motion is still generated from the diffusion branch with audio input as a condition.  \nb) The authors claim that there is error accumulation in long video sequences of existing AR and SAR models, but no comparative experiments have been conducted between the proposed NAR method and existing methods to prove this claim.  \nc) Given that the authors consider NAR as the main contribution of this paper, the paper does not adequately discuss the advantages of NAR over AR and SAR, nor does it provide reasonable comparative metrics to substantiate these claims. Merely comparing general metrics such as image quality, lip sync, identity, and motion matching is not sufficient to demonstrate that the performance improvements are due to the adoption of NAR.  \n3. Methods like Anitalker[4], TH-PAD[1], and VASA-1[3], etc., have similar overall architectures with the proposed one, but the authors did not illustrate the differences between the proposed method and these methods.\n4. Recently, many diffusion-based talking head synthesis methods have been proposed, such as AniPortrait[5], Anitalker[4], Hallo[6], EchoMimic[7], FollowYourEmoji[8], and so on. Most of them operate in an auto-regressive manner. The authors did not compare the overall performance with these methods; therefore, there is no adequate evidence to prove the performance superiority of the proposed method.\n\nGenerally, I think the authors should conduct more experiments and propose more reasonable metrics to prove the effectiveness of the proposed method. And considering the main contribution is the non-autoregressive approach, the authors need to adequately discuss its advantages over AR and SAR. Based on the above reasons, I have doubts that this submission meets the bar for publication.\n\nIf the authors could discuss the advantages of NAR more thoroughly and supplement with sufficient experiments to prove that the proposed method has better effects compared to existing diffusion methods, I would raise my rating.\n\n[1] Yu, Zhentao, et al. \"Talking head generation with probabilistic audio-to-visual diffusion priors.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.  \n[2] He, Tianyu, et al. \"Gaia: Zero-shot talking avatar generation.\" arXiv preprint arXiv:2311.15230 (2023).  \n[3] Xu, Sicheng, et al. \"Vasa-1: Lifelike audio-driven talking faces generated in real time.\" arXiv preprint arXiv:2404.10667 (2024).  \n[4] Liu, Tao, et al. \"AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding.\" arXiv preprint arXiv:2405.03121 (2024).  \n[5] Wei, Huawei, Zejun Yang, and Zhisheng Wang. \"Aniportrait: Audio-driven synthesis of photorealistic portrait animation.\" arXiv preprint arXiv:2403.17694 (2024).  \n[6] Xu, Mingwang, et al. \"Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation.\" arXiv preprint arXiv:2406.08801 (2024).  \n[7] Chen, Zhiyuan, et al. \"Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions.\" arXiv preprint arXiv:2407.08136 (2024).  \n[8] Ma, Yue, et al. \"Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation.\" arXiv preprint arXiv:2406.01900 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9bgI06GJkF", "reviewer": "Reviewer_Q1nh", "summary": "The paper presents a new method for talking head generation called DAWN. This networks contains three main parts: a latent flow generator to drive a source image using latent flows, an \"audio to latent flow\" diffusion network and a pose and blink generation network. The network first predict latent flows from an audio sequence and pose and blink parameters. Those latent flows are then used by the latent flow generator to project the motion to RGB space using a source image containing the new identity. The pose and blink parameters are generated by the pose and blink generation network. DAWN outperforms the method used in the comparison on most metrics quantitatively and also qualitatively. The generated head pose and blink can be replaced by real one extracted from video for better control on the generation. Extensive ablation study is provided.", "questions": "- The latent flow generator is effectively a video-driven talking head generation method. It would have been interesting to see how it perform against similar methods, e.g. [7].\n\n- Notation issue: in eq .6 the input are x_src,y_1N and p_1N. in the conditioning paragraph and in the figure it is stated that instead the inputs are in fact Z_src,a_1N and p_1N. Also p_1N is not defined when it's used in eq.6. Both of these things need to be corrected.\n\n[7]: Y. Wang, D. Yang, F. Bremond and A. Dantcheva, \"LIA: Latent Image Animator,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2024.3449075.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method works directly on the image by using its own latent flow generator. This is good because the network doesn't need to rely on other intermediate representations (e.g. 3dmm extracted from the video) which can already contain errors leading to bad generation.\n\n- DAWN is able to generate pose and blink directly from the audio which is something not many methods do.\n\n- By generating the head pose and blinking parameters outside the diffusion network and using them as condition instead the authors make their method more controllable.\n\n- Qualitative results look good even if in low resolution.\n\n- The authors provided video results for qualitative evaluation.\n\n- The ablation study is extensive.", "weaknesses": "- The main issue I have with the paper is that the method used as baseline for the comparison are not state of the art: Wav2Lip, MakeItTalk and Audio2Head are from 2020-2021 while SadTalker and DiffusedHeads are from 2023. There are more recent method that could have been used for the quantitative comparison [1,2], there also exist diffusion based method that perform much better than DiffusedHeads [3]. Additionally several recent methods, for which the code is not available, could have been considered only for the qualitative results due to their impressive performances [4,5,6].\n\n- On the same subject DiffusedHeads is known to perform badly on out of distribution data. It was trained on the MEAD dataset and tend to fail if the background is not green.\n\n- The method is able to generate head pose but both datasets used for experiments exhibit limited head poses. It would have been better to use the VoxCeleb or CelebV-HQ datasets to evaluate the quality of the generated head poses.\n\n- Most experiments are performed with a resolution of 128*128 when state of the art usually use 256X256 or higher (notably for HDTF that is often used at 512X512).\n\n- In table 1 the LSEc and LSEd of wav2lips are not in bold/underlined when they should, they are often the best.\n\n- For the video results provided the authors should make it clear whether the head pose was generated or taken from the ground truth. In some examples (e.g. driven_by_music.mp4) it look really impressive but I suspect it come from the ground truth since the results in the comparison video, while good, are not as impressive.\n\n- It is still not clear how the non-autoregressive part is used during inference. Are the sequences generated only at a chosen size (e.g 200 for HDTF)? Or are several sequence generated then put back to back to make long videos (like the ones from the supplementary material). If it is the latter, since the method is non-autoregressive how do the authors avoid discontinuity between sequences with respect to the head pose?\n\n- In table 4 the authors show that PBNET improve lips synchronization (LSEd and LSEc) by a lot but do not explain in details why this is happening. While it's true that the training become more difficult as a result there is still a loss focusing on lips so the lip motion should stay about as good in my opinion.\n\n- I understand why the authors perform two training with different sequence length and the ablation shows that it improves results. However I don't see why they set X_src=X_1 for the first training instead of using a random starting frame like they do in the second training. Could the authors clarify their choice on this?\n\n- The paper mention that a GAN loss is used inside PBNET but does not say how it is applied. Is there a discriminator? This part need to be clarified.\n\n- The paper state that AR and SAR \"leads to constrained performance and potential error accumulation, especially in long video sequences\" but only cite one paper to support that claim. Most other methods mentioned in the paper or the one I proposed seem fine in that regard. The authors should develop this part with more examples.\n\n[1]: Tan, Shuai, Bin Ji, and Ye Pan. \"Style2talker: High-resolution talking head generation with emotion style and art style.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.\n\n[2] S. Wang et al., \"StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 6, pp. 4331-4347, June 2024, doi: 10.1109/TPAMI.2024.3357808\n\n[3]: Ma, Y., Zhang, S., Wang, J., Wang, X., Zhang, Y., Deng, Z.: Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767 (2023)\n\n[4]: Xu, Sicheng, et al. \"Vasa-1: Lifelike audio-driven talking faces generated in real time.\" arXiv preprint arXiv:2404.10667 (2024).\n\n[5]: Zhang, Bingyuan, et al. \"Emotalker: Emotionally editable talking face generation via diffusion model.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024\n\n[6]: Tian, Linrui, et al. \"Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions.\" arXiv preprint arXiv:2402.17485 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Vd8CtpsCp", "reviewer": "Reviewer_5zi1", "summary": "The paper introduces DAWN, a system that uses non-autoregressive (NAR) diffusion models to generate talking head videos of varying lengths from portrait images and audio. This method achieves faster processing speeds and high-quality outputs. To overcome limitations in NAR approaches and improve the modeling of longer videos, the system separately controls the movements of the lips, head, and blinking. This provides more precise control over these individual movements. The paper proposes PBNet, a network that generates realistic head poses and blinking sequences directly from audio clips using an NAR approach. The Two-stage Curriculum Learning (TCL) strategy is introduced to train the model effectively in generating lip movements and controlling head poses and blinks accurately, which helps in achieving robust convergence and better extrapolation capabilities.", "questions": "Did the paper conduct a user study? I don’t seem to have seen one. User studies are very important in this field.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The performance is decent\n2. Talking head generation is a research field with practical value.", "weaknesses": "1. The motivation for introducing NAR is relatively weak. VASA[1], which uses SAR, already achieves real-time performance with good results. The motivation to use NAR to improve inference speed is weak, as the paper is only faster than diffused head, but the slower speed of diffused head is due to it generating images directly rather than intermediate representations.\n2. The novelty of this paper is limited, as many modules are similar to previous methods. For example, the LFG is similar to the FOMM[2] series of work, and pose prediction is similar to SadTalker[3]. Although the paper introduces some novel training techniques like TCL, previous methods have already achieved good results without relying on such fancy techniques. Therefore, the significance of these fancy techniques is questionable.\n\n[1] Xu, Sicheng, et al. \"Vasa-1: Lifelike audio-driven talking faces generated in real time.\" arXiv preprint arXiv:2404.10667 (2024).\n\n[2] Siarohin, Aliaksandr, et al. \"First order motion model for image animation.\" Advances in neural information processing systems 32 (2019).\n\n[3] Zhang, Wenxuan, et al. \"Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)