DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation

Authors: Hanbo Cheng, Limin Lin, Chenyu Liu, Pengcheng Xia, Pengfei Hu, Jiefeng Ma, Jun Du, Jia Pan
Primary area: generative models
Keywords: Talking head generation, Non-autoregressive generation, Avatar, Video generation, Diffusion model

Abstract:
Talking head generation intends to produce vivid and realistic talking head videos from a single portrait and speech audio clip. Although significant progress has been made in diffusion-based talking head generation, almost all methods rely on autoregressive strategies, which suffer from limited context utilization beyond the current generation step, error accumulation, and slower generation speed. To address these challenges, we present DAWN (\textbf{D}ynamic frame \textbf{A}vatar \textbf{W}ith \textbf{N}on-autoregressive diffusion), a framework that enables all-at-once generation of dynamic-length video sequences. Specifically, it consists of two main components:  (1) audio-driven holistic facial dynamics generation in the latent motion space, and (2) audio-driven head pose and blink generation. Extensive experiments demonstrate that our method generates authentic and vivid videos with precise lip motions, and natural pose/blink movements. Additionally, with a high generation speed, DAWN possesses strong extrapolation capabilities, ensuring the stable production of high-quality long videos. These results highlight the considerable promise and potential impact of DAWN in the field of talking head video generation. Furthermore, we hope that DAWN sparks further exploration of non-autoregressive approaches in diffusion models. Our code will be publicly available at \url{https://github.com/Hanbo-Cheng/DAWN-pytorch}.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729238815090

thread_type:
official_review

<thread_object>
creator: reviewer_5zi1

<content>
summary:
The paper introduces DAWN, a system that uses non-autoregressive (NAR) diffusion models to generate talking head videos of varying lengths from portrait images and audio. This method achieves faster processing speeds and high-quality outputs. To overcome limitations in NAR approaches and improve the modeling of longer videos, the system separately controls the movements of the lips, head, and blinking. This provides more precise control over these individual movements. The paper proposes PBNet, a network that generates realistic head poses and blinking sequences directly from audio clips using an NAR approach. The Two-stage Curriculum Learning (TCL) strategy is introduced to train the model effectively in generating lip movements and controlling head poses and blinks accurately, which helps in achieving robust convergence and better extrapolation capabilities.

strengths:
1. The performance is decent
2. Talking head generation is a research field with practical value.

weaknesses:
1. The motivation for introducing NAR is relatively weak. VASA[1], which uses SAR, already achieves real-time performance with good results. The motivation to use NAR to improve inference speed is weak, as the paper is only faster than diffused head, but the slower speed of diffused head is due to it generating images directly rather than intermediate representations.
2. The novelty of this paper is limited, as many modules are similar to previous methods. For example, the LFG is similar to the FOMM[2] series of work, and pose prediction is similar to SadTalker[3]. Although the paper introduces some novel training techniques like TCL, previous methods have already achieved good results without relying on such fancy techniques. Therefore, the significance of these fancy techniques is questionable.

[1] Xu, Sicheng, et al. "Vasa-1: Lifelike audio-driven talking faces generated in real time." arXiv preprint arXiv:2404.10667 (2024).

[2] Siarohin, Aliaksandr, et al. "First order motion model for image animation." Advances in neural information processing systems 32 (2019).

[3] Zhang, Wenxuan, et al. "Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

questions:
Did the paper conduct a user study? I don’t seem to have seen one. User studies are very important in this field.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729770561499

thread_type:
official_review

<thread_object>
creator: reviewer_q1nh

<content>
summary:
The paper presents a new method for talking head generation called DAWN. This networks contains three main parts: a latent flow generator to drive a source image using latent flows, an "audio to latent flow" diffusion network and a pose and blink generation network. The network first predict latent flows from an audio sequence and pose and blink parameters. Those latent flows are then used by the latent flow generator to project the motion to RGB space using a source image containing the new identity. The pose and blink parameters are generated by the pose and blink generation network. DAWN outperforms the method used in the comparison on most metrics quantitatively and also qualitatively. The generated head pose and blink can be replaced by real one extracted from video for better control on the generation. Extensive ablation study is provided.

strengths:
- The method works directly on the image by using its own latent flow generator. This is good because the network doesn't need to rely on other intermediate representations (e.g. 3dmm extracted from the video) which can already contain errors leading to bad generation.

- DAWN is able to generate pose and blink directly from the audio which is something not many methods do.

- By generating the head pose and blinking parameters outside the diffusion network and using them as condition instead the authors make their method more controllable.

- Qualitative results look good even if in low resolution.

- The authors provided video results for qualitative evaluation.

- The ablation study is extensive.

weaknesses:
- The main issue I have with the paper is that the method used as baseline for the comparison are not state of the art: Wav2Lip, MakeItTalk and Audio2Head are from 2020-2021 while SadTalker and DiffusedHeads are from 2023. There are more recent method that could have been used for the quantitative comparison [1,2], there also exist diffusion based method that perform much better than DiffusedHeads [3]. Additionally several recent methods, for which the code is not available, could have been considered only for the qualitative results due to their impressive performances [4,5,6].

- On the same subject DiffusedHeads is known to perform badly on out of distribution data. It was trained on the MEAD dataset and tend to fail if the background is not green.

- The method is able to generate head pose but both datasets used for experiments exhibit limited head poses. It would have been better to use the VoxCeleb or CelebV-HQ datasets to evaluate the quality of the generated head poses.

- Most experiments are performed with a resolution of 128*128 when state of the art usually use 256X256 or higher (notably for HDTF that is often used at 512X512).

- In table 1 the LSEc and LSEd of wav2lips are not in bold/underlined when they should, they are often the best.

- For the video results provided the authors should make it clear whether the head pose was generated or taken from the ground truth. In some examples (e.g. driven_by_music.mp4) it look really impressive but I suspect it come from the ground truth since the results in the comparison video, while good, are not as impressive.

- It is still not clear how the non-autoregressive part is used during inference. Are the sequences generated only at a chosen size (e.g 200 for HDTF)? Or are several sequence generated then put back to back to make long videos (like the ones from the supplementary material). If it is the latter, since the method is non-autoregressive how do the authors avoid discontinuity between sequences with respect to the head pose?

- In table 4 the authors show that PBNET improve lips synchronization (LSEd and LSEc) by a lot but do not explain in details why this is happening. While it's true that the training become more difficult as a result there is still a loss focusing on lips so the lip motion should stay about as good in my opinion.

- I understand why the authors perform two training with different sequence length and the ablation shows that it improves results. However I don't see why they set X_src=X_1 for the first training instead of using a random starting frame like they do in the second training. Could the authors clarify their choice on this?

- The paper mention that a GAN loss is used inside PBNET but does not say how it is applied. Is there a discriminator? This part need to be clarified.

- The paper state that AR and SAR "leads to constrained performance and potential error accumulation, especially in long video sequences" but only cite one paper to support that claim. Most other methods mentioned in the paper or the one I proposed seem fine in that regard. The authors should develop this part with more examples.

[1]: Tan, Shuai, Bin Ji, and Ye Pan. "Style2talker: High-resolution talking head generation with emotion style and art style." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.

[2] S. Wang et al., "StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 6, pp. 4331-4347, June 2024, doi: 10.1109/TPAMI.2024.3357808

[3]: Ma, Y., Zhang, S., Wang, J., Wang, X., Zhang, Y., Deng, Z.: Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767 (2023)

[4]: Xu, Sicheng, et al. "Vasa-1: Lifelike audio-driven talking faces generated in real time." arXiv preprint arXiv:2404.10667 (2024).

[5]: Zhang, Bingyuan, et al. "Emotalker: Emotionally editable talking face generation via diffusion model." ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024

[6]: Tian, Linrui, et al. "Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions." arXiv preprint arXiv:2402.17485 (2024).

questions:
- The latent flow generator is effectively a video-driven talking head generation method. It would have been interesting to see how it perform against similar methods, e.g. [7].

- Notation issue: in eq .6 the input are x_src,y_1N and p_1N. in the conditioning paragraph and in the figure it is stated that instead the inputs are in fact Z_src,a_1N and p_1N. Also p_1N is not defined when it's used in eq.6. Both of these things need to be corrected.

[7]: Y. Wang, D. Yang, F. Bremond and A. Dantcheva, "LIA: Latent Image Animator," in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2024.3449075.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729863890830

thread_type:
official_review

<thread_object>
creator: reviewer_fcws

<content>
summary:
This paper proposes a diffusion-based non-autoregressive talking head synthesis method that operates with input from only a source portrait and an audio sequence. The method is performed in a two-stage manner: a Latent Flow Generator (LFG) serves as the video reenactment model, and a diffusion model together with PBNet serve as the motion generation models. To achieve non-autoregressive synthesis (NAR), PBNet generates the entire blink and pose sequence at once. The authors also propose a two-stage curriculum learning strategy to enhance the efficiency and performance of the diffusion model's learning process.

strengths:
1. The paper is well written and easy to read. 
2. The proposed method can performed nearly in real time on a single V100 16G GPU.

weaknesses:
1. I believe the proposed method lacks novelty. The overall architecture and the two-stage approach, which involves training a reenactment model first and then generating motion latents, have already been proposed in many existing works, such as TH-PAD[1], GAIA[2], VASA-1[3], AniTalker[4], and so on.. Although the authors attempted to innovate from a non-autoregressive perspective, I believe that their discussion and argumentation regarding the advantages of NAR are not sufficiently robust, and I will state my reasons below.
2. I don't think the proposed NAR method and experiments conducted demonstrate the authors' claim:  
a) Generating only blinks and poses through PBNet is not sufficient; the expression-related motion is still generated from the diffusion branch with audio input as a condition.  
b) The authors claim that there is error accumulation in long video sequences of existing AR and SAR models, but no comparative experiments have been conducted between the proposed NAR method and existing methods to prove this claim.  
c) Given that the authors consider NAR as the main contribution of this paper, the paper does not adequately discuss the advantages of NAR over AR and SAR, nor does it provide reasonable comparative metrics to substantiate these claims. Merely comparing general metrics such as image quality, lip sync, identity, and motion matching is not sufficient to demonstrate that the performance improvements are due to the adoption of NAR.  
3. Methods like Anitalker[4], TH-PAD[1], and VASA-1[3], etc., have similar overall architectures with the proposed one, but the authors did not illustrate the differences between the proposed method and these methods.
4. Recently, many diffusion-based talking head synthesis methods have been proposed, such as AniPortrait[5], Anitalker[4], Hallo[6], EchoMimic[7], FollowYourEmoji[8], and so on. Most of them operate in an auto-regressive manner. The authors did not compare the overall performance with these methods; therefore, there is no adequate evidence to prove the performance superiority of the proposed method.

Generally, I think the authors should conduct more experiments and propose more reasonable metrics to prove the effectiveness of the proposed method. And considering the main contribution is the non-autoregressive approach, the authors need to adequately discuss its advantages over AR and SAR. Based on the above reasons, I have doubts that this submission meets the bar for publication.

If the authors could discuss the advantages of NAR more thoroughly and supplement with sufficient experiments to prove that the proposed method has better effects compared to existing diffusion methods, I would raise my rating.

[1] Yu, Zhentao, et al. "Talking head generation with probabilistic audio-to-visual diffusion priors." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.  
[2] He, Tianyu, et al. "Gaia: Zero-shot talking avatar generation." arXiv preprint arXiv:2311.15230 (2023).  
[3] Xu, Sicheng, et al. "Vasa-1: Lifelike audio-driven talking faces generated in real time." arXiv preprint arXiv:2404.10667 (2024).  
[4] Liu, Tao, et al. "AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding." arXiv preprint arXiv:2405.03121 (2024).  
[5] Wei, Huawei, Zejun Yang, and Zhisheng Wang. "Aniportrait: Audio-driven synthesis of photorealistic portrait animation." arXiv preprint arXiv:2403.17694 (2024).  
[6] Xu, Mingwang, et al. "Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation." arXiv preprint arXiv:2406.08801 (2024).  
[7] Chen, Zhiyuan, et al. "Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions." arXiv preprint arXiv:2407.08136 (2024).  
[8] Ma, Yue, et al. "Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation." arXiv preprint arXiv:2406.01900 (2024).

questions:
1. I have a question regarding the ablation study on the TCL strategy: when the model was trained only with stage 1 and only with stage 2, did you maintain the same number of training epochs and steps as in the proposed method? I am curious to know whether the effectiveness of the two-stage curriculum learning is due to the training time or the strategy itself.

New after rebuttal: I raised the soundness score from 1 to 2 and the rating from 3 to 5.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676756680

thread_type:
official_review

<thread_object>
creator: reviewer_fbou

<content>
summary:
This work presents a non-autoreregrssive diffusion based approach for talking head generation. This allows the generation of videos with non-fixed length. To enhance the performance the authors also decouple the motion of lips, head, and blinks and also introduce a two stage curriculum learning strategy. The proposed model is evaluated on the CREMA and HTDF datasets achieving state-of-the-art results.

strengths:
The paper is well written and easy to follow

First non-autoregressive diffusion-based approach for talking head generation

weaknesses:
Why is the proposed approach compared against wav2lip? Wav2lip only generates the mouth region, therefore the comparison is not really fair. The authors are encouraged to explain in the paper this decision and clearly acknowledge this limitation.

The authors use the syncnet model for measuring the quality of the generated lip movements. Why not using a lipreading model? It would be a more accurate measure of the lip movement quality. An explanation why syncnet is preferred over lipreading models would be very useful. Alternatively, the authors could use some public lipreading models to evaluate the performance (e.g., AutoAVSR, Raven, AV-Hubert).

The authors use several quantitative metrics to evaluate the model's performance and this is good. However, a user study is missing. There are no quantitative metrics which correlate highly with human perception, therefore evaluating the performance of generative models via user studies is highly desirable. The paper would be much stronger if a user study is included. Otherwise, the authors should explain why it's not included.

Table 1, why are some results underlined? This should be explained in the captions.

On which dataset is the LFG trained? Is it trained on each dataset separately? Also, what's the impact of pre-training? Has the option of training the LFG component jointly with the rest of the model been considered (and if yes can some results be presented)? The paper would be stronger if the author provide additional details regarding the above questions.

questions:
Please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732028319583

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments to All Reviewers

comment:
We sincerely thank all reviewers for their valuable suggestions and efforts. We have earnestly responded to each suggestion and accordingly revised our paper. If you find our answers responsive to your concerns, we would be grateful if you would consider increasing your score.

In response to the reviewers' suggestions and comments, we have revised the manuscript, with all changes marked in blue. These revisions include: 
- Main Paper: In Equation 6 (line 226), we correct some of the imprecise expressions. (Reviewer  Q1nh )
- Main Paper: In Section 3.6 (line 311), emphasize both the PBNet and A2V-FDM generate the sequence with dynamic length non-autoregressively, and the generation length depends on input audio length. (Reviewer Q1nh)
- Main Paper: In Table 1 (line 381), we add the details for the comparison with Wav2Lip, and explain why Wav2Lip didn't participate in ranking. (FboU, Q1nh) Also, we explain the meaning of \underline{underline} (line 380). (Reviewer  FboU)
- Main Paper: In Table 1 (line 398) and Figure 2 (line 449), we add **three** latest diffusion-based methods to reinforce the experiment in the overall comparison section. (Reviewer  fcWS, Q1nh)
- Main Paper: In Section 4.4 (line 349), Table 2 (line 409) and Table 6 (line 834), we add the **Degradation  Rate** for evaluation of error accumulation, to better prove the superiority of our non-autoregressive method. At the same time, we have added multiple citations (line 46) to support the assertion that AR methods lead to error accumulation. (Reviewer  fcWS, Q1nh, 5zi1)
- Main Paper: In Section 4.5 (line 499), we add the **user study** to reinforce our experiment by human evaluation. (Reviewer  FboU, 5zi1)
- Appendix: In Appendix A.2 (line 768), we add the Training Details to better explain the configuration in training (including LFG, PBNet and A2V-FDM). (Reviewer FboU, fcWS, Q1nh)
- Appendix: In Appendix A.3.3 (line 858), we add the quantitative study on our method with higher resolution. (Reviewer Q1nh)
- Appendix: In Appendix A.3.4 (line 933), we add two latest diffusion-based methods for image latent into generation speed comparison. (Reviewer 5zi1)

- Main Paper: We specify that the FID-based Degradation Rate proposed in the paper is specifically designed for detecting the degradation problem of image-based AR diffusion methods, as it focuses on the degradation detection in the image space. (Main Paper line 349, Appendix line 801, 807) (Reviewer Q1nh)

Please kindly check our updated manuscript. We welcome further discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received mixed scores: one borderline accept, two borderline reject, and one accept. Most reviewers acknowledged the simple and effective solution proposed in the paper. Specifically, the paper introduces the first non-autoregressive diffusion-based approach for talking head generation, which was recognized as a novel contribution by the reviewers. However, some reviewers raised concerns, including the high similarity to existing methods such as Anitalker, TH-PAD, and VASA-1, limited original contributions, and restricted practicality of the approach. The AC has reviewed the paper, the reviews, and the rebuttal. While there have been numerous papers on generative model-based talking face synthesis, such as VASA-1, the application of non-autoregressive diffusion (NAD) to this task is novel and was acknowledged by the reviewers. Furthermore, the paper is well-written and presents its ideas clearly. Therefore, the AC suggests acceptance. It would be even better if the authors addressed all the reviewers' concerns in the final version.