ConsistentAvatar: Learning to Diffuse Fully Consistent Talking Head Avatar with Temporal Guidance

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 09:12:10

Abstract: Diffusion models have shown impressive potential on talking head generation. While plausible appearance and talking effect are achieved, these methods still suffers from temporal, 3D or expression inconsistency due to the error accumulation and inherent limitation of single-image generation ability. In this paper, we propose ConsistentAvatar, a novel framework for fully consistent and high-fidelity talking avatar generation. Instead of directly employing multi-modal conditions to the diffusion process, our method learns to first model the temporal representation for stability between adjacent frames. Specifically, we propose a Temporally-Sensitive Detail (TSD) map containing high-frequency feature and contour that vary significantly along time axis. Using a temporal consistent diffusion module, we learn to align TSD of the initial result to that of the video frame ground truth. The final avatar is generated by a fully consistent diffusion module, conditioned on the aligned TSD, rough head normal, and emotion prompt embedding. We find that the aligned TSD, which represents the temporal patterns, constrains the diffusion process to generate temporally stable talking head. Further, its reliable guidance complements the inaccuracy of other conditions, suppressing the accumulated error while improving the consistency on various aspects. Extensive experiments demonstrate that ConsistentAvatar outperforms the state-of-the-art methods on the generated appearance, 3D, expression and temporal consistency.

Main paper: 
 introduction: 
 In the realm of virtual reality and its associated applications, the creation of lifelike and animatable character avatars poses a significant challenge. Early approaches [5, 7, 34, 47, 49] often involve generating head avatars by fitting 3D Morphable Models (3DMMs) [2] , utilizing a parameterized representation to describe the shape, texture, and details of the human face. By adjusting these parameters, precise control over facial features can be achieved, thereby demonstrating relatively strong 3D controllability. However, 3DMMs typically rely on certain assumptions, such as linear shape spaces, which may not fully capture the complex variations in facial shape, texture, and details. Consequently, the resulting facial renderings may lack realism (see Fig. 1 (a) ).To address these issues, some methods [11, 16, 36, 39, 45, 46] combine StyleGANs [15] with the priors of 3DMMs to improve the fidelity of generated avatars. For example, SadTalker [45] generates 3D motion coefficients of 3DMMs from audio and implicitly adjusts a novel 3D-aware face rendering. Thanks to the powerful generative capabilities of GANs, these methods often yield realistic images. However, these methods fundamentally rely on a singleimage 2D generation process, where the image rendering process is intricately entangled and facial fitting capability is limited. Consequently, these methods encounter difficulties in achieving consistent 3D control (see Fig. 1 (b) ). To overcome the limitations of 2D GAN-based methods, recent researches have tended to utilize Neural Radiance Field (NeRF) or similar implicit representations to generate face avatars [8, 17, 24, 26, 47] , or integrate NeRF into GANs for 3D-aware dynamic face synthesis [17, 24, 26] . Typically, these methods learn dynamic NeRF based on input expressions as the conditions to represent the deformed 3D space. For example, IMAvatar [47] optimizes the implicit function on the mesh template to achieve more accurate 3D control. However, as the volume rendering involves manual approximation on the real-world imaging process, it has potential to lose fidelity on the rendered images. Additionally, NeRF struggles to decouple temporal information from 3D representation, resulting in temporal inconsistencies in such methods. As illustrated in Fig. 1 (a), INSTA [49] and IMAvatar [47] suffer from inaccurate expressions or appearance degradation. With the development of diffusion models, more recent studies [12, 18, 19, 35] have adopted this strategy to enhance the quality of face generation and editing. For example, DiffTalk [35] models the generation of talking heads as a denoising process driven by audio. Due to the powerful generation capabilities of diffusion models, these methods often achieve highly realistic results. However, these methods are essentially 2D-aware, so that they also lack sufficient 3D consistency.Based on the above discussion, it seems that using more widelycovered conditions lead to fully consistent generated avatars in diffusion models. Actually, several efforts [6, 21] have integrated 3D-aware conditions like face normal or depth to guide their diffusion models. To further analyse relative strategies, we build a baseline diffusion model to generate avatars, conditioned on the low-resolution face image / normal generated from pre-trained INSTA [49] and emotion label embedding from CLIP [30] . As illustrated in Fig. 1 (d), we observe that DiffusionRig [6] and our baseline model both suffer from high temporal error and instability, revealing that employing more conditions for diffusion models may not always lead to superior results. We argue the reason behind is two-fold: 1) the inaccuracy within these conditions disturbs the diffusion process and accumulates to the final result; 2) the based diffusion model [33] is image but not video generation model, and thus temporal consistency cannot be modeled or guaranteed. Note that, given a video as training data, the ground truth of temporal patterns has been implicitly contained. This motivate us to learn these patterns and constrain other noisy conditions for talking avatar generation.To this end, we introduce ConsistentAvatar, a novel framework that combines diffusion models with the temporal, 3D-aware, and emotional conditions, for generating fully consistent and highfidelity head avatars. Instead of directly generating avatar, we learn to first generate temporally consistent representation as a constraint to guide other conditions. Concretely, our method starts from the efficient INSTA method [49] and utilize its outputs as the initial results. In order to model the temporal consistency, we propose a Temporally-Sensitive Detail (TSD) generated by Fourier transformation, containing high-frequency information and contours that change significantly between frames. We extract TSD from coarse RGB output of INSTA and the target video frame, and propose a temporal consistency diffusion model to align the input TSD to the precise one. Besides, we utilize the coarse normal output of INSTA as the 3D-aware condition, and propose an emotion selection module to generate emotion embedding for each frame in an unsupervised manner. With the aligned TSD, normal, and emotion embedding as conditions, we then propose a fully consistent diffusion model to generate the final avatars. In this way, we guarantee the temporal consistency during the avatar generation, and complement other conditions to contribute to a fully consistent and high-quality avatar generation result. In summary, our contributions are as follows:• We propose ConsistentAvatar, a diffusion-based neural renderer that generates temporal, 3D, and expression consistent talking head avatars. • We learn to align a novel Temporally-Sensitive Details (TSD) to maintain the stability between generated frames, and complement rough normal and emotion conditions for high-fidelity generation.• Extensive experiments demonstrate that ConsistentAvatar outperforms the state-of-the-art methods on the generated appearance quality, details, expression and temporal consistency. 
related work 2.1 3d face animation: 
 Early methods [4, 32, 45] use 3DMM priors to instruct the generator. SadTalker [45] , for instance, generates head poses and expressions of 3DMMs from audio and implicitly modulate a 3D-aware face synthesis model. Recent methods [1, 8, 10, 27, 47, 49] , combined 3D representation techniques with 3DMMs to control the expressions and poses of avatars. For example, NHA [10] and IMAvatar [47] refine the mesh topology of FLAME [22] to achieve more realistic mesh-based avatars. INSTA [49] and RigNeRF [1] utilize 3DMMs to construct radiance fields. Additionally, other 3D representation methods , such as GaussianAvatars [29] , construct dynamic 3D representations based on 3D Gaussian splats rigged to a parametric morphable face model. More recent works [6, 19, 21, 35] have leveraged the powerful generation capabilities of 2D diffusion models. For instance, DiffusionAvatars [21] utilizes the 3D priors of the recent Neural Parametric Head Model (NPHM) [9] combines with a 2D rendering network for high-quality image synthesis. In our work, we integrate 3D representation with diffusion models, simultaneously leveraging 2D and 3D priors for consistent head avatar synthesis.Denoising Diffusion Probabilistic Models (DDPMs) [13] integrate image generation with a sequential denoising process of isotropic Gaussian noise. In this process, the model is trained to anticipate noise levels from the input image. Due to their remarkable ability to generate 2D content, they have gradually found application in face-related tasks. Many studies [6, 19, 21, 35] , refine pre-trained diffusion models by introducing additional control factors like landmarks or depth. For example, DiffusionRig [6] suggests conditioning the diffusion model on rasterized grids, considering factors such as normals and albedo. Conversely, DiffTalk [35] utilizes audio-driven and landmark-based conditioning to improve identity generalization. Despite the good controllability and visual quality of these methods, they often lack accurate 3D representations or precise 2D detail priors. As a result, they struggle to maintain consistent 3D rendering across various viewpoints and temporal consistency in video rendering simultaneously.Undoubtedly, emotions play a crucial role in shaping the authenticity of the synthesized portrait. Recently, there has been a growing effort to incorporate emotions into the synthesis process for better control over the final output. For instance, EMOCA [3] introduces a novel deep perceptual emotion consistency loss, ensuring alignment between the reconstructed 3D expression and the depicted emotion in the input image. Meanwhile, GMTalker [41] proposes a Gaussian mixture-based Expression Generator (GMEG), enabling the creation of a continuous and multimodal latent space for more versatile emotion manipulation. However, approaches like EVP [14] and EMMN [38] , which directly infer emotions from labeled audio, may encounter accuracy issues due to the inherent complexity of emotional expression. In contrast, methods such as MEAD [40] and the work by Sinha et al. [37] implicitly learn the intrinsic relationship between emotions and facial expressions through the use of emotion labels. In our methodology, we leverage the MEAD [40] dataset to assign emotion labels to the experimental dataset based on emotional similarities, facilitating precise control over the emotions of the resulting portraits. 
preliminary: 
 INSTA. INSTA [49] is based on a dynamic neural radiance field composed of neural graphics primitives embedded around a parametric face model. It is capable of reconstructing photorealistic digital avatars instantaneously in less than 10 minutes, while allowing for interactive rendering of novel poses and expressions. We choose INSTA to get a proxy of talking head under a target expression and head pose. The proxy, i.e., initial result used as input for our method is 512 × 512 rendered by volume rendering process, containing limited appearance quality and expression accuracy.Our method is capable of lifting the proxy to a high-fidelity and consistent avatar.Denoising Diffusion Probabilistic Models. Denoising Diffusion Probabilistic Models (DDPMs) [13, 25, 31] belong to a class of generative models that take random noise images as input and progressively denoise them to produce photorealistic images. This process can be viewed as the reverse of the diffusion process, which gradually adds noise to images. The core component of DDPMs is a denoising network denoted as 𝑓 𝜃 , During training, it receives a noisy image 𝑥 𝑡 and a timestep 𝑡 (1 ≤ 𝑡 ≤ 𝑇 ), and predicts the noise at time 𝑡 : 𝜖 𝑡 . More formally, the predicted noise at time 𝑡is ε𝑡 = 𝑓 𝜃 (𝑥 𝑡 , 𝑡), where 𝑥 𝑡 = 𝛼 𝑡 𝑥 0 + √︃ 1 -𝛼 2 𝑡 𝜖 𝑡 ,𝜖 𝑡 is a random, normally distributed noise image, and 𝛼 𝑡 is a hyperparameter that gradually increases the noise level of 𝑥 𝑡 with each step of the forward process. The loss is computed based on the distance between 𝜖 𝑡 and ε𝑡 . Therefore, the trained model can generate images by taking a random noise image as input and progressively denoising it to achieve photorealism. 
method: 
 In this section, we introduce ConsistentAvatar, a framework specifically designed for generating high-quality avatars while maintaining full consistency. An overview of this framework is illustrated in Fig. 2 . Ensuring temporal consistency in generating portraits remains a challenge as the diffusion model struggles to directly learn time-related information from images. Therefore, we learn to generate temporally consistent representations as constraints to guide other conditions. We define Temporally-Sensitive Details (TSD), a detail map 𝐼 𝑇 𝑆𝐷 generated by Fourier transformation from the coarse RGB output of INSTA [49] and the target video frame. Furthermore, our experiments reveal that directly using the extracted TSD as a condition does not yield satisfactory temporal consistency, we propose a temporal consistency diffusion model to align the input TSD to the precise one (Sec. 4.1). After ensuring temporal consistency during the avatar generation process, we utilize the coarse normal output of INSTA and emotional text embeddings as conditions to construct a fully consistent diffusion model (Sec. 4.2). Finally, to expedite and further optimize our model, we draw inspiration from LCM [23] and SDXL [28] , significantly reducing the necessary inference steps and further enhancing the quality of image generation (Sec. 4.3).As mentioned above, to ensure temporal consistency in generating portraits, we extract Temporally-Sensitive details (TSD) generated by Fourier transformation from the coarse RGB output of INSTA [49] and the target video frame. We then align these details through a diffusion model. This approach complements other conditions and contributes to achieving fully consistent and high-quality avatar generation results. Given a monocular RGB video containing 𝐾 frames {𝐼 1 , 𝐼 2 , . . . , 𝐼 𝐾 } , camera pose 𝑃 𝑐 , tracked FLAME [22] meshes 𝑴 = {𝑀 𝑖 } with corresponding expressions 𝑬 = {𝐸 𝑖 } ∈ R 1×100 and poses 𝑷 = {𝑃 𝑖 } ∈ R 1×6 . We obtain RGB output and normal using INSTA [49] . The process is described as follows:EQUATIONwhere F 𝑖𝑛𝑠𝑡𝑎 is the INSTA model. After obtaining the RGB output, we utilize Fourier transform and its inverse transform to extract TSD from the RGB output. We apply the Fourier transform to shift an image from its spatial domain to the frequency domain, revealing various frequency components. Retaining these high-frequency elements enables the Fourier transform to efficiently extract image details, which are represented as:EQUATIONwhere the equation represents content extraction at frequency 𝑤 from the image 𝐼 𝑟𝑔𝑏 𝑖 , with 𝑒 -𝑗 𝑤𝑡 as the orthogonal basis and 𝑊 as the frequency set of the image. Experimentally, this paper sets 𝑤=10. Then performing inverse Fourier transform to convert the frequency domain back to images, which are represented as:EQUATIONAs illustrated in Fig. 2 , the TSD of a video frame contains highfrequency information and contours that represent crucial feature of expression, head pose and details for generating the avatar. However, TSD also varies significantly between adjacent frames. This inspires us to predict stable TSD of a frame from inaccurate 𝐼 𝑇 𝑆𝐷 𝑖 . Performing the same operation on the ground truth video frames allows us to obtain the ground truth of TSD. Therefore, we propose employing a Temporally-Sensitive Diffusion Model (TSDM) to align 𝐼 𝑇 𝑆𝐷 𝑖 . Specifically, inspired by [6] , we first encode the obtained 𝐼 𝑇 𝑆𝐷 𝑖 ∈ R 512×512×3 to obtain the global latent code E (𝐼 𝑇 𝑆𝐷 𝑖 ) ∈ R 1×256 . Then we pass 𝑃 𝑖 and 𝐸 𝑖 through a linear layer to obtain the same scale as E (𝐼 𝑇 𝑆𝐷 𝑖 ) and concatenate them to get 𝑓 𝑐 ∈ R 3×256 . We add new cross-attention layers to the U-Net, following IPAdapter [43] . Let 𝑍 be the intermediate feature map computed by an existing cross-attention operation in the pretrained LDM [33] : 𝑍 = 𝐴𝑇𝑇 𝐸𝑁𝑇 𝐼𝑂𝑁 (𝑄, 𝐾, 𝑉 ). Then, we perform direct conditioning by adding another cross-attention layer:EQUATIONThe more specific denoising process can be represented as follows:EQUATIONwhere 𝑥 𝑡 is the noisy image at timestep 𝑡 ∈ {1, 2, . . . ,𝑇 }, 𝑥 0 is the ground truth latent image, ε𝑡 is the predicted noise, and F represents the denoising model. The optimization objective of the entire denoising process is as follows: EQUATIONIn Sec. 4.1, after passing TSD through the diffusion model, we obtain precise and temporally stable TSD. This compensates for other conditions that may not be particularly accurate. Therefore, by adding normal condition and emotion condition, we alleviate 3D consistency and expression consistency.Emotion Condition: For emotion condition, we propose utilizing the MEAD [40] dataset to acquire emotion labels for our experimental dataset. As described in Fig. 3 , the MEAD comprises numerous monocular video clips {𝑉 1 , 𝑉 2 , . . . , 𝑉 𝑀 }, each clips corresponds to an emotional label 𝐿 𝑚 ∈ {𝐿 1 , 𝐿 2 , . . . , 𝐿 𝑀 } (such as anger, happy, etc.). Leveraging this rich emotion dataset, we construct a facial expression database. Subsequently, employing DECA [7] , we compute facial expression vectors for each frame in every clip 𝐸 𝑚𝑛 , 𝑛 represents the frame index of the clip. And then determine the similarity between the current frame's facial expression vector 𝐸 𝑖 and those in the database. Finally, we assign the emotion label corresponding to the facial expression vector with the highest similarity in the database as the emotion label for the current frame. The process is described as follows:EQUATIONwhere 𝐼 𝑖 is an image from our dataset, and 𝑉 𝑚𝑛 is a frame from the MEAD dataset, 𝑐𝑠 () represents the cosine similarity function, which is used to measure the similarity between two expression vectors. 𝑎𝑟𝑔𝑚𝑎𝑥 () retrieves the image corresponding to the maximum similarity, which belongs to the MEAD dataset. After acquiring emotional labels corresponding to image through EMO (𝐼 𝑖 ), we use the text encoder in CLIP [30] to extract textual features of emotion labels, obtaining the final emotion code 𝑇 𝑖 ∈ R 1×768 . After acquiring STD, normal, and emotion conditions, we perform the same linear operation on 𝑇 𝑖 as on 𝐸 𝑖 and 𝑃 𝑖 , and then concatenate them to obtain 𝑓 𝑑 ∈ R 4×256 . At that time, it can be utilized as a condition through cross-attention operation. Finally, we construct a Fully Consistent Stable Diffusion model (FCSD). Specifically, we utilize the pretrained Stable Diffusion model (SD) [33] as the backbone network to expedite training and enhance generation quality. Additionally, we incorporate a fully consistency module, effectively integrating consistency-related information into the backbone network, to construct a ControlNet [44] . The final rendering is achieved through an iteratively denoising the full noise 𝑥 𝑇 with our fullyconsistent neural renderer S conditioned on 𝑧 𝑐 :EQUATION)where C is the ControlNet architecture and 𝑧 𝑐 is the concatenation of the normal condition and the TSD condition. During training, we minimize the following loss:EQUATIONSimilar to Sec. 4.1, the final denoised result x0 is then upsampled to the pixel space with the pretrained image decoder 𝐼 𝑔𝑒𝑛 𝑖 = D 𝑝𝑜𝑟𝑡𝑟𝑎𝑖𝑡 ( x0 ), where 𝐼 𝑔𝑒𝑛 𝑖 ∈ R 512×512×3 is the reconstructed face image. This process provides the network with additional multimodal conditions, helping it control the 3D and expression consistency of the characters under the well-guided aligned TSD, thereby assisting in generating portrait animations with more precise motion.It is indisputable that both generation speed and the quality of portrait generation are pivotal criteria for task assessment. We closely follow the development of diffusion models and, based on this foundation, further accelerate efficiency and enhance image quality.Latent Consistency Model (LCM). To ensure our model maintains high-quality generation while minimizing the required steps, we employ LCM [23] to expedite the generation process. The core concept of this model lies in redefining the fundamental logic of traditional diffusion models (such as DDPM [13] ) in the generation process. Traditional diffusion models generate final results by gradually reducing noise in images, typically through iterative and time-consuming processes. In contrast, LCM [23] transforms traditional numerical ordinary differential equation (ODE) solvers into neural network-based solvers, enabling direct prediction of the final clear image and thereby reducing intermediate steps, significantly enhancing efficiency. By employing LCM [23] to enhance our model, we have reduced the required steps for predicting the final image from around 1000 steps to approximately 10 steps.Refiner. Inspired by Stable Diffusion XL (SDXL) [28] , we have further elevated the generation quality of our model. SDXL represents the latest optimized version of Stable Diffusion, comprising a two-stage cascaded diffusion model consisting of a Base model and a Refiner model. Integrating the Refiner model with our own, after the Base model generates latent features of the image, we utilize the Refiner model to conduct minor noise reduction and enhance detail quality on these latent features. 
experiment 5.1 setup: 
 Dataset: In our experiments, we utilize four datasets. Initially, we employ the dataset released by INSTA [49] to train the primary framework. This dataset comprises 10 monocular videos, each capturing the performance of an individual actor. These videos undergo cropping and resizing, achieving a resolution of 512 2 , effectively removing extraneous elements from the facial region through background subtraction. Additionally, for equitable comparison with other methods, we utilize two datasets from IMAvatar [47] and Ner-Face [8] which share the same data format as INSTA [49] . Finally, during the emotional labeling phase, we utilize the Multi-view Emotional Audio-visual Dataset (MEAD) [40] , which includes a dialogue video corpus featuring 60 actors conversing with 8 different emotions. High-quality audio-visual clips from seven different perspectives are captured in a strictly controlled environment.Evaluation Protocol: We utilize the final 350 frames of each video in our dataset for testing purposes, while the remaining frames are allocated for training. To assess image quality, we employ metrics consistent with state-of-the-art methods [47, 49] , including L2 loss, structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and the perceptual metric LPIPS. Additionally, to evaluate the temporal consistency of the generated videos, we utilize optical flow to compute the degree of change between adjacent frames.Implementation Details: Our framework was implemented using PyTorch on a machine equipped with an RTX 3090 GPU. The training process is divided into three stages. In the first stage, all experimental configurations match those of INSTA [49] , yielding RGB and normal outputs for the portraits. Moving to the second stage, we utilize the temporally unstable TSDs obtained from the RGB outputs of the first stage to train an TSDs generator with ground truth supervision. We utilize the Adam optimizer [20] with a learning rate of 10 -5 and conduct 3000 iterations with a batch size of 4. Finally, in the third stage, we train a fully consistent portrait generator based on the outputs of the first two stages. Once again, we use the Adam optimizer with a learning rate of 10 -4 and conduct 15000 iterations with a batch size of 4.Baseline setting: To illustrate the role of TSD in temporal consistency more clearly, we use Stage 1 combined with Stage 3 as our baseline. Here, TSD is not learned through Stage 2, but is directly used as a condition for ControlNet [44] .Image quality evaluation: We conduct comprehensive experiments on our dataset, assessing the quality and consistency of the synthetic digital human avatars generated by our method, and comparing them with state-of-the-art methods such as IMAvatar [47] , DiffusionRig [6] and INSTA [49] . As shown in Fig. 4 , our approach yields more realistic results. We capture facial details such as wrinkles and eyeglass frames well. In contrast, other methods Table 1 : Quantitative analyses between our method and stateof-the-art models.like INSTA [49] fail to capture wrinkle information and other details. Tab. 1 presents quantitative results, demonstrating that our method achieves state-of-the-art performance across three different datasets. Particularly noteworthy is the substantial improvement in both PSNR and L2 loss compared to INSTA [49] . Taken together, these metrics indicate that our generated portraits are closer to ground truth and more realistic. Additionally, our ability to better capture facial expressions contributes to the overall enhancement of the results. Additional. We are well aware of the importance of real-time performance, so as mentioned in Sec. 1, we have focused on improving speed by introducing LCM [23] on top of our model. As shown in the data in the Tab. 1, we have significantly reduced the time. Note that the above results were obtained with a time step setting of 10 during the denoising process, while the default diffusion model step size is set to 1000. The incorporation of LCM [23] significantly reduces the time step. 3D consistency results: 3D consistency measures whether the character maintains the correct appearance from different viewpoints. As shown in Fig. 1 (b), methods like SadTalker [45] do not naturally exhibit good 3D consistency since they do not involve 3D operations during learning and rely on a single-image 2D generation process. From Fig. 5 , we can also observe the advantage of our method, with the angles of head rotation being closer to the ground truth and the results appearing more realistic. For quantitative results, we utilize DECA [7] to estimate the pose coefficients of the generated portraits, calculating the error between the predicted pose coefficients and the ground truth, referred to as Pose Error Expression consistency results: The expression consistency measures how well the generated results fit the target expressions. As shown in Fig. 6 , our method is capable of achieving more accurate expressions. For quantitative analysis, similar to PE, we employ DECA [7] to evaluate the expression coefficients of the generated portraits, calculating the disparity between the predicted expression coefficients and the ground truth, referred to as Expression Error (EE). As depicted in Tab. 2, our approach surpasses both Diffusionrig [6] and INSTA [49] , exhibiting minimal error relative to the ground truth.Temporal consistency results: The evaluation of temporal consistency evaluates the stability of the generated video, focusing on the absence of jitter or significant fluctuations between frames, ensuring smooth playback. We compare our method with state-ofthe-art approaches like INSTA [49] , SadTalker [45] and DiffusionRig [6] in various categories. We utilize optical flow to compute the degree of change between two frames. Naturally, we consider the normal variation between video frames. Therefore, we compare it with the ground truth as well. From the Fig. 7 , it's evident that our approach closely approximates the ground truth.We conduct a series of ablation studies to analyze the different components of our method. Specifically, we focus on 1) the impact of unaligned coarse TSD versus aligned TSD through the diffusion model on other conditions and on temporal consistency; 2) the impact of the emotion condition on the final results; 3) the impact of the normal condition on the final 3D consistency;Aligned TSD and Unaligned TSD. From Fig. 1 (d), it is evident that the temporal consistency of our baseline is inferior to that of our final method. Since the baseline directly adopts unaligned TSD conditions, this proves that aligned TSD can provide better  
limitation:: 
 Although our work achieves realistic and fully consistent portraits through the diffusion model and the utilization of TSD, normal, and emotion conditions, there are still limitations. Our method often lacks accurate modeling of teeth and eyeball due to the lack of geometric constraints, as depicted in Fig. 9 .  
conclusion: 
 ConsistentAvatar presents a novel framework for generating talking avatars with full consistency and high fidelity. We introduce a Temporally-Sensitive Detail (TSD) map containing high-frequency features and contours that exhibit significant variation over time. Utilizing a temporal consistent diffusion module, we align the TSD of the initial result with the ground truth of the video frame. The final avatar is generated using a fully consistent diffusion module, conditioned on the aligned TSD, rough head normal, and emotion prompt embedding. Aligned TSD, representing temporal patterns, guides the diffusion process to produce temporally stable talking heads. Its reliable guidance supplements the inaccuracies of other conditions, thereby reducing accumulated errors and enhancing consistency across various aspects.  
 
 Appendix:  


Decision: The paper proposes a new method for high-fidelity talking head generation. Generally, I find this paper quite impressive to some extent $-$ (1) it provided sufficient qualitative evidence (both on paper and through the supplemental demo clips) which was quite convincing; (2) there is a measured amount of analysis and discussion afforded to the aspect of temporal consistency, which bodes well with the theme of the paper. It must be said that achieving a temporally consistent generation with diffusion models is a rather challenging task, and this paper sets out to address this critical issue. Comparisons with state-of-the-art methods were convincing and showed the promise of the proposed idea. It would be great if the authors could demonstrate the generation w.r.t emotions ($L_m$) and if the expression consistencies ($deca$) are better for certain emotion types than others.

Indeed, I concur with the reviewers' assessment that the authors did a thorough job of addressing their concerns in the rebuttal. I find the rebuttal concise yet meticulous. I hope the authors will incorporate these changes into the final version of the paper to increase the quality of the final paper. 

I'm happy to recommend the acceptance of this paper for oral presentation.

Reviews: review: paper_summary: paper_summary: The paper addresses the challenges faced by existing diffusion models in generating talking heads, particularly issues related to temporal, 3D, and expression inconsistencies. These issues arise from error accumulation and the limitations inherent in single-image generation capabilities. To overcome these challenges, the authors propose ConsistentAvatar, a novel framework that ensures fully consistent and high-fidelity talking avatar generation.
strength_weakness: strength_weakness: 1. The proposed Temporally-Sensitive Detail (TSD) map helps maintain stability between adjacent frames, reducing flickering and inconsistencies over time. 
2. The use of rough head normal from INSTA and emotion prompt embeddings from CLIP, in conjunction with the TSD, results in more lifelike and expressive avatars.
3. ConsistentAvatar achieves a higher level of detail and expression accuracy in the generated avatars.
limitations: limitations: 1. The baseline is a combination of INSTA and ControlNet, which is incremental. As shown in Table 1, the enhancement provided by TSD is not significant, limiting the overall contribution of this paper.
2.  ConsistentAvatar trains INSTA, a diffusion model, and a ControlNet. the efficiency of training is a problem. It is better to make the training efficiency clear.
3. The comparison in the ablation study is not convincing. More metrics should be presented to effectively evaluate the performance of different components, such as metrics reported in Table 1.
4. Poor writing. There are quite a few grammar mistakes and awkward phrases with unclear definitions. These problems make this paper harder to read and understand. For instance:
-Abstract (line 45): “these methods still suffer” -> “these methods still suffer”
-Abstract (line 54): “high-frequency feature and contour” -> “high-frequency features and contours”
-Abstract (line 55): “vary significantly along time axis” -> “vary significantly along the time axis”
-Introduction (line 141): “recent researches” -> “recent research” or “recent studies”
-Introduction (line 176): “the reason behind is two-fold”->”the reasons behind are two-fold”
-Introduction (line 182 and 183) “This motivate … constrain…”-> “This motivates … constrains…”
-Abstract (line 104): “the inaccuracy of other conditions” which kind of inaccuracy conditions?
-And so on…
5. The figures are a little messy, especially Figures 2 and 3. In Figure 3, the phrase “if x is…” is used, but “x” is not defined. The figures should be cleaned up to enhance the paper's readability and make it suitable for publication.
6. SadTalker is not a perfect baseline for this problem. More emotional talking-head generation methods[1,2] should be discussed. 
[1] Emote Portrait Alive: Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions. Ji et al. ACM SIGGRAPH 2022
[2] Efficient emotional adaptation for audio-driven talking-head generation. Gan et al. ICCV 2023
paper_summary: paper_summary: This paper proposes ConsistentAvatar, a diffusion-based head avatar generation framework, which renders avatar videos initialized by a Nerf-based approach INSTA to generate high-fidelity talking face videos.
To overcome the inconsistency of diffusion models, this paper introduces Temporal-Sensitive Details (TSD), which extract high-frequency temporal information from head video by Fourier transformation and utilizes TSD as a condition for video generation.
To generate consistent avatar videos, ConsistentAvatar proposes a fully-consistent module, where multiple conditions, including normal map, TSD, and initial avatar videos are inputted into the diffusion model. Additionally, emotion text embedding is used to control the avatar's emotions.
Experiments demonstrate that ConsistentAvatar performs well on the generated appearance, 3D, expression, and temporal consistency.
strength_weakness: strength_weakness: The idea of generating temporally consistent representations as conditions is interesting. To achieve this idea, this paper extracts temporal high-frequency features as TSD, and the refined stable TSD is utilized as a condition for the diffusion model. The experiments demonstrate the effectiveness of TSD.
The full pipeline for avatar video generation is complete, where multiple conditions are considered for consistency, and emotion labels are embedded for control. Additionally, LCM and refiner are proposed for further accelerate efficiency and enhance image quality.
The provided videos in supplementary materials show better consistency than the diffusion-based method, diffusionrig.
limitations: limitations: The current experimental results should all be based on reconstruction. Could you compare the non-reconstructed poses as well?

In Table 2, why does the PE metric only compare with SadTalker, but not Insta, DiffusionRig, PointAvatar, or ImAvatar?

Although optical flow can be used to measure temporal consistency, can other common metrics like FVD also be used for evaluation?
paper_summary: paper_summary: This paper focuses on talking head generation and introduces a diffusion-based neural render method (namely ConsistentAvatar) with high temporal, 3D and expression consistency. Specifically, this paper suggests a Temporally-Sensitive Detail (TSD) map containing high-frequency information that vary significantly in different frames. The authors inject TSD into diffusion models to improve the temporal stability. Comprehensive experiments showcase the superiority of the proposed method compared to state-of-the-art methods.
strength_weakness: strength_weakness: + This paper in well organized and easy to understand.
+ The overall paper is technically sound.
+ The proposed method shows non-trivial performance improvement compared to several prior methods. The videos in supplementary material clearly show temporal consistency of the proposed method.
limitations: limitations: - The novelty of this paper is marginal. Most techniques are from existing methods, such INSTA [49], ControlNet [44] and LCM [23]. Only the usage of high-frequency information (TSD) to enhance temporal consistency is somewhat new in the talking head generation task. However, I cannot find the results of the proposed method without TSD in Table 1. 
- Some design choices are confusing. I am wondering why stage 2 can improve the performance. Although the authors claim necessity of temporally consistent module (stage 2), this module should marginal performance improvement as shown in Table 1. 
- In Table 1, I am wondering why the baseline (i.e., the final model without stage 2) has the same time costs.
- Some typos should be fixed, such as ‘We learn to align a novel Temporally-Sensitive Details’ in Line 211.
paper_summary: paper_summary: The paper "ConsistentAvatar: Learning to Diffuse Fully Consistent Talking Head Avatar with Temporal Guidance" introduces a novel framework that tackles temporal, 3D, and expression inconsistencies in talking head avatar generation. By leveraging Temporally-Sensitive Details (TSD) and optical flow for temporal guidance, the method achieves significant improvements in consistency across various aspects. Comparative analysis with real data demonstrates the effectiveness of the proposed approach. The experimental results showcase superior performance compared to state-of-the-art methods in terms of appearance, 3D consistency, expression, and temporal consistency. The paper also provides clear implementation details and optimization techniques, such as the use of LCM for faster inference speed and the Refiner for improved detail quality. Overall, the paper makes a solid contribution to the field of avatar generation by addressing multiple consistency issues simultaneously and demonstrating promising results through extensive experiments.
strength_weakness: strength_weakness: 1.	The paper introduces a novel framework, ConsistentAvatar, which addresses temporal, 3D, and expression inconsistencies in talking head avatar generation simultaneously.
2.	The method proposes a novel Temporally-Sensitive Details (TSD) module to maintain the stability between generated frames, which can further aid the rough normal and emotion conditions for high-fidelity avatar generation.
3.	The paper provides a comprehensive comparative analysis with real data, demonstrating significant mitigation of temporal inconsistencies by the proposed method. Extensive experiments showcase that the proposed method outperforms state-of-the-art methods in terms of appearance, 3D consistency, expression, and temporal consistency.
4.	The paper provides clear implementation details, including the training process stages and optimization techniques like LCM and Refiner which improve efficiency and image quality respectively.
limitations: limitations: Weaknesses  
1.	The paper demonstrated about the improvement on generation speed by the incorporation of LCM in Table 1. However, apart from the generation quality, rendering speed is also crucial as stated in the paper. The proposed method seems cannot reach the real-time requirement. More discussion needs to be addressed to reach the goal.
2.	The paper may benefit from discussing potential real-world applications or scenarios where the ConsistentAvatar framework could be utilized.

Questions   
1.	How does the ConsistentAvatar framework handle the trade-off between computational efficiency and maintaining high-quality avatar generation results, especially when incorporating the LCM and Refiner?
2.	The LCM can generate high-quality results with faster speed, however, sometimes the time step might be not enough for generating a realistic result, how does ConsistentAvatar address this issue?

Limitations  
The paper acknowledges limitations in accurately modeling teeth and eyeballs due to the lack of geometric constraints. These limitations might impact the overall realism and fidelity of the generated avatars. Are there potential avenues for improvement in future research to address these challenges?