FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model

Authors: ziyu  yao
zhiqi  huang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-11 15:48:34

Abstract: Talking head generation is a significant research topic that still faces numerous challenges. Previous works often adopt generative adversarial networks or regression models, which are plagued by generation quality and average facial shape problem. Although diffusion models show impressive generative ability, their exploration in talking head generation remains unsatisfactory. This is because they either solely use the diffusion model to obtain an intermediate representation and then employ another pre-trained renderer, or they overlook the feature decoupling of complex facial details, such as expressions, head poses and appearance textures. Therefore, we propose a Facial Decoupled Diffusion model for Talking head generation called FD2Talk, which fully leverages the advantages of diffusion models and decouples the complex facial details through multi-stages. Specifically, we separate facial details into motion and appearance. In the initial phase, we design the Diffusion Transformer to accurately predict motion coefficients from raw audio. These motions are highly decoupled from appearance, making them easier for the network to learn compared to high-dimensional RGB images. Subsequently, in the second phase, we encode the reference image to capture appearance textures. The predicted facial and head motions and encoded appearance then serve as the conditions for the Diffusion UNet, guiding the frame generation. Benefiting from decoupling facial details and fully leveraging diffusion models, extensive experiments substantiate that our approach excels in enhancing image quality and generating more accurate and diverse results compared to previous state-of-the-art methods.

Main paper: 
 introduction: 
 Talking head generation is a task that creates a digital representation of a person's head and facial movements synchronized with the audio signal. This technology serves as a cornerstone with farreaching applications, including virtual reality, augmented reality, and entertainment industries such as film production [17, 54, 55] .With the development of deep learning [11, 52, 53] , it has recently attracted numerous researchers and achieved impressive results. Prevailing methodologies for talking head generation can be broadly divided into two paradigms. One approach involves using a GAN-based framework [9, 12, 16, 22, 29, 43] , which simultaneously optimizes a generator and a discriminator. However, due to the inherent flaws of GANs themselves and the suboptimal framework designs, this often results in unsatisfactory results, such as unnatural faces and inaccurate lip movements. The other approach utilizes regression models [5, 15, 18, 23, 46, 61] to map audio to facial movements, ensuring better temporal consistency. Nonetheless, regression-based methods encounter challenges in generating natural movements with individualized characteristics, leading to issues with average facial shapes and less diverse results.Recently, the rise of diffusion models [21, 32, 39] has marked a new era in generative tasks. Due to their stable generation process and relative ease of training, diffusion models offer a promising avenue for the advancement of talking head technology. While some previous works [13, 36, 40] have attempted to apply diffusion models to talking head generation, their generated results still suffer from low image quality, unnaturalness, and insufficient lip synchronization. We analyze that there are two main issues in current methods. 1) Some approaches [27] apply diffusion models solely to predict facial intermediate representations, such as 3DMM coefficients. However, they still rely on pre-trained renderers for rendering the final faces, resulting in low-quality in the generated images. 2) Other approaches [36, 40] directly generate faces through pixel-level denoising, globally conditioned on the audio and reference image. Nevertheless, they overlook the fact that faces contain rich information, such as expressions, poses, texture, etc. Previous methods couple these facial details, significantly complicating denoising generation and yielding unsatisfactory results.To address the above issues, we propose the Facial Decoupled Diffusion model for Talking head generation, named FD2Talk. Our FD2Talk leverages the generative advantages of diffusion models to generate high-quality, diverse and natural talking heads videos. As illustrated in Fig. 1 , the proposed FD2Talk model is a multi-stage framework that decouples complex facial details into motion and appearance information. The first phase focuses on motion information generation, while the second phase is dedicated to driving frame synthesis. 1) Motion Generation. Motion information, including lip movements, expressions, and head poses, is highly related to the given audio and is more decoupled from facial appearance, making it easier to learn. In the first stage, we design novel Diffusion Transformers to extract motion-only information, i.e., 3DMM expression and head pose coefficients, from the raw audio. Through the denoising process, we generate natural and accurate motions, thereby enhancing the realism of our final outputs. Additionally, predicting the head pose coefficients at this stage enables us to produce more diverse motions compared to previous methods.2) Frames Generation. Moving on to the second stage, we first encode the reference image to capture appearance information, including human identity and texture characteristics. Combining this appearance information with the previously learned motion, we obtain a comprehensive facial representation related to the final RGB faces. Unlike previous methods that utilize a pre-trained face renderer to render final frames, we design a conditional Diffusion UNet and utilize motion and appearance as conditions to guide higher-quality and more natural animated frame generation.Our two-stage approach not only maintains motion plausibility and accuracy, but also enhances texture fidelity. Moreover, by focusing on generating appearance-independent information in the first stage, we can enhance the generalization ability of our FD2Talk. This is because we can obtain pure motion coefficients from the audio signal without being influenced by the portrait domains. The contribution can be summarized as follows:• Our proposed FD2Talk is a multi-stage framework that effectively decouples facial motion and appearance, enabling accurate motion modeling, superior texture synthesis, and improved generalization. • Our approach fully leverages the generative power of diffusion models in both motion and frames generation stages, thus enhancing the quality of the results.• Extensive experiments demonstrate that our method excels at generating accurate and realistic talking head videos, achieving state-of-the-art performance. By incorporating head pose modeling, our FD2Talk produces significantly more diverse results compared to previous methods. 
related works: 
 Audio-Driven Talking Head Generation. Previous methods have attempted to utilize generative adversarial networks [4, 6, 29, 41, 44, 45, 59, 60] and regression models, such as RNN [42] , LSTM [18, 46, 61] and Transformer [1, 15] to synthesis talking head videos based on audio signals. Among GAN-based methods, [29] proposed a novel lip-synchronization network that generates talking head videos with accurate lip movements across different identities by learning from a powerful lip-sync discriminator. [59] disentangled person identity and speech information through adversarial learning, leading to improved talking head generation. [44] introduced a temporal GAN with three discriminators focused on achieving detailed frames, audio-visual synchronization, and realistic expressions, capable of generating lifelike talking head videos. On the other hand, in regression-based methods, [18] adopts LSTM for better temporal consistency using explicit and implicit keypoints as the intermediate representation. Additionally, [15] proposed a Transformer-based autoregressive model that encodes long-term audio context and autoregressively predicts a sequence of animated 3D face meshes. Despite significant progress, the unrealistic results in GAN-based generation and the average facial shape problem in regression-based models remain unresolved.Diffusion Models for Talking Head Generation. Diffusion models have demonstrated the remarkable ability across multiple generative tasks, such as image generation [30, 34, 35] , image inpainting [24, 50, 51] , and video generation [3, 20, 25] . Recently, some studies [13, 36, 40] have delved into using diffusion models for talking head generation. However, these studies still face challenges in producing natural and accurate faces. On one hand, they [27] generate intermediate representations using diffusion models but rely on pre-trained face renderers for synthesizing the final frames. On the other hand, they [36, 40] globally utilize audio features to condition the generation of faces, which couples the complex facial motion and appearance. To fully leverage the advantages of the diffusion model and disentangle the complex facial information, we utilize the diffusion model in both motion generation and frame generation, thereby achieving better performance. 
method: 
 Given a reference image I ∈ R 3×𝐻 ×𝑊 and a corresponding audio input, our model is designed to synthesize a realistic talking head video V ∈ R 3×𝐹 ×𝐻 ×𝑊 with lip movements synchronized with the audio signal. Here, the symbols 𝐹 , 𝐻 , and 𝑊 denote the frame numbers, frame height and frame width respectively.Our FD2Talk framework consists of two stages that decouple facial information into motion and appearance, thus enhancing the modeling of facial representation. We employ powerful diffusion models in both stages, making FD2Talk a fully diffusion-based approach that produces high-quality talking head results. Specifically, we start by using Diffusion Transformers to predict expressions and pose motions from the audio input. In the subsequent stage, we utilize a Diffusion UNet to generate final RGB images, conditioned on the previously predicted motion information along with appearance texture information extracted from a reference image.3.1.1 3D Morphable Model. To generate high-quality talking heads, we integrate 3D information into our method, specifically employing the 3D Morphable Model (3DMM) [10] to decouple the facial representation from a given face image. This allows us to describe the 3D face space (3D mesh) using Principal Component Analysis:EQUATIONHere, S ∈ R 3𝑁 (where 𝑁 represents the number of vertices of a face, and 3 represents the axes 𝑥, 𝑦, and 𝑧) denotes a 3D face, while S is the mean shape. 𝜶 ∈ R 𝐷 3.1.2 Diffusion Model. Diffusion models are formulated as timeconditional denoising networks that learn the reverse process of a Markov Chain with a length 𝑇 . Specifically, starting from the clean signal 𝒙 0 , the process of adding noise can be denoted as follows:EQUATIONHere, 𝜖 𝑡 ∼ N (0, 1) denotes random Gaussian noise, while 𝛼 𝑡 represents the hyper-parameter for the diffusion process. 𝒙 𝑡 refers to the noisy feature at step 𝑡, where 𝑡 ∈ [1, . . . ,𝑇 ]. During inference, the 𝑇 -step denoising process progressively denoise random Gaussian noise N (0, 1) to estimate the clean signal 𝒙 0 . In our work, all diffusion-based models are designed to predict signal itself rather than noise. Thus, the overall goal can be described as follows:EQUATIONwhere 𝜃 represents the diffusion model and 𝑐 represents conditional guiding. We utilize the 𝐿2 error between the estimated signal and the ground truth 𝒙 0 .Early diffusion-based methods [36, 40] globally utilize audio signals as a condition for the pixel-level denoising process. However, this approach combines motion and appearance, making it challenging for overall training convergence. In contrast, in the first stage, our method focuses on generating motion-only information from the audio signal, specifically 3DMM expression and head pose coefficients. These coefficients exclusively represent facial and head motion, which are highly decoupled from the appearance textures and greatly influence lip synchronization and motion diversity. Furthermore, compared to high-dimensional RGB faces, low-dimensional 3DMM coefficients are considerably easier for the model to learn.To ensure smooth continuity between different frame motions and fully leverage the diffusion models, we introduce sequence-tosequence Diffusion Transformers for generating both expression and pose coefficients. Meanwhile, to effectively address the one-tomany mapping problem and accurately predict lip movements and diverse head poses, we decouple the prediction of expression and pose coefficients using an Expression Transformer 𝜃 𝑒𝑥𝑝 and a Pose Transformer 𝜃 𝑝𝑜𝑠𝑒 , which is illustrated in Fig. 2 . We decouple the motion into expression and head poses, both of which are predicted by our designed DiTs. The audio guides the generation through cross-attention layers, utilizing an alignment mask to ensure accurate lip movements. Furthermore, the pre-trained lip expert also enhances the lip synchronization.Specifically, we initialize the noisy expression sequence 𝜷 𝑇 ∈ R 𝐹 ×𝐷 𝛽 and noisy pose sequence 𝒑 𝑇 ∈ R 𝐹 ×𝐷 𝑝 from random Gaussian noise N (0, 1), where 𝐹 represents the number of frames aligned with the final video. We then denoise the 𝜷 𝑇 and 𝒑 𝑇 conditioned on audio features through 𝑇 loops to estimate denoised sequence 𝜷 0 and 𝒑 0 . Here, the length of audio clip 𝑨 is aligned with 𝐹 , and we adopt the state-of-the-art self-supervised pre-trained speech model, Wav2Vec 2.0 [2] , to extract the audio features.Taking 𝜃 𝑝𝑜𝑠𝑒 as an example, at each timestep 𝑡, we concatenate the embedding from the timestep and audio features to obtain the condition 𝑐. We then project 𝑐 to an intermediate representation 𝜏 (𝑐) ∈ R 𝐹 ×𝐷 𝜏 using a linear layer. Then, 𝜏 (𝑐) is fused into 𝜃 𝑝𝑜𝑠𝑒 via the cross-attention layer, where the query (Q) is derived from 𝒑 𝑡 , while the key (K) and value (V) are obtained from 𝜏 (𝑐). Meanwhile, we design an alignment mask M to ensure the consistency of generated coefficients and the audio signal, so that 𝜏 (𝑐) for the 𝑖 𝑡ℎ timestamp attends to 𝒑 𝑡 at the 𝑗 𝑡ℎ timestamp only if 𝑗 -𝑘 ≤ 𝑖 ≤ 𝑗 +𝑘. For FD2Talk, we empirically set 𝑘 = 3. The M can be denoted as:EQUATIONIn our diffusion process, we directly estimate the original signal. Therefore, after 𝐿-layer Pose Transformer, we obtain p0 . Subsequently, we can calculate the single-step denoising result 𝒑 𝑡 -1 :EQUATIONwhere 𝜎 𝑡 is the Gaussian covariance at the 𝑡 𝑡ℎ timestep. The Exp Transformer 𝜃 𝑒𝑥𝑝 and Pose Transformer 𝜃 𝑝𝑜𝑠𝑒 share the same architecture, and the denoising process for 𝜷 𝑡 is identical to that for 𝒑 𝑡 . Therefore, after 𝑇 iterations, we obtain 𝜷 0 and 𝒑 0 as the final values for the expression and pose coefficients.Previous methods primarily employ pre-trained face renderers [31, 47] to generate final RGB faces, whose performance sets an upper bound on talking face generation. Therefore, we design a conditional Diffusion UNet 𝜃 𝑢𝑛𝑒𝑡 to generate the final frame based on previously predicted 3DMM coefficients, aiming to utilize the diffusion models to achieve diverse and realistic faces generation.To reduces computational overhead and accelerates convergence, we introduce a pair of encoder E and decoder D [32] to transition the frame generation into the latent space. Suppose the downsampling factor is 𝑓 = 𝐻 /ℎ = 𝑊 /𝑤, then we can encode the reference image I into the reference latent code 𝑥 = E (I) ∈ R 𝑑 ×ℎ×𝑤 .As shown in Fig. 3 , we initialize the noisy latent image 𝑱 𝑇 ∈ R 𝑑 ×ℎ×𝑤 from N (0, 1), then we progressively denoise it conditioned on both reference latent code 𝑥 and 3DMM coefficients 𝜷 0 and 𝒑 0 .Here, 𝑥 encompasses the appearance texture of the reference image, while 𝜷 0 and 𝒑 0 includes the driving facial and head motions.An intuitive approach is to directly concatenate the 𝑥, 𝜷 0 and 𝒑 0 to obtain the conditions. However, we observe that this operation leads to difficulties in training convergence, because there exists a gap between the image domain and the motion coefficients domain.To address the impact of domain gap, we use two cross-attention layers to introduce these two conditions respectively. Specifically, both the encoder and decoder of Diffusion UNet consist of two cross-attention layers, denoted as 𝜙 1 and 𝜙 2 . The coefficients 𝜷 0 and 𝒑 0 are concatenated, following with a linear projection, to form the condition for 𝜙 1 . The calculation of 𝜙 1 can be defined as:EQUATIONwhere the query (Q) is from 𝑱 𝑡 , and the key (K) and value (V) are from the condition {𝜷 0 , 𝒑 0 }. Then, in the second layer 𝜙 2 , we utilize the reference latent code 𝑥 as the condition to guide this process:EQUATIONwhere the query (Q) is from 𝒎 1 , and the key (K) and value (V) are derived from 𝑥. Here, the 𝑥 is reshaped into sequence, and positional encoding is also introduced. This decoupling of conditions enhances the denoising stability, leading to higher-quality results. Similar to that in the first stage, at each diffusion timestep 𝑡, we predict J 0 from 𝑱 𝑡 , and then calculate the corresponding 𝑱 𝑡 -1 using the Eq. ( 5). After 𝑇 iterations, this process generates the accurate denoised latent image 𝑱 0 . The reference latent code and the denoised latent image are further concatenated as the input of decoder D, allowing us to generate the RGB image V 𝑖 , which serves as each frame for the talking head video V = {V 𝑖 } 𝐹 1 . Moreover, as we denoise in the latent space, we can easily extend to higherresolution talking head synthesis by adjusting the downsampling factor 𝑓 , thereby further enhancing our generation quality.Our training process consists of two stages. In the first stage, we train the Exp Transformer and Pose Transformer to generate accurate expression and pose coefficients. Using these accurate coefficients as a foundation, we then train the Diffusion UNet in the second stage to generate natural and diverse RGB frames. For the Exp Transformer 𝜃 𝑒𝑥𝑝 , by adding random Gaussian noise, the 𝜷 0 can become 𝜷 𝑡 using Eq. ( 5). The 𝜃 𝑒𝑥𝑝 estimates β0 = 𝜃 𝑒𝑥𝑝 (𝜷 𝑡 , 𝑡, 𝐴), and the objective can be defined as follows:EQUATIONSimilar to the 𝜃 𝑒𝑥𝑝 , the objective of the Pose Transformer 𝜃 𝑝𝑜𝑠𝑒 is:EQUATIONWhile the random noise introduced in the diffusion model can effectively facilitate the diverse generation, it also leads to inaccurate mouth shape generation to some extent. Therefore, we utilize a pre-trained lip expert [29] to guide this denoising process and generate more accurate mouth shape. Specifically, we first obtain the identity coefficients from the reference image, and then calculate the 3D meshes using these identity coefficients along with the predicted expression coefficients β0 via Eq. (1). From these 3D meshes, we select vertices in the mouth area to represent lip motion [26] . The pre-trained lip expert calculates the cosine similarity between mouth motion embedding 𝑣 and audio embedding 𝑎 as follows:EQUATION)where 𝜖 is a small number for avoiding the division-by-zero. Then, the 𝜃 𝑒𝑥𝑝 minimizes the synchronous loss as follows: EQUATIONEQUATIONwhere 𝜆 𝑒𝑥𝑝 , 𝜆 𝑝𝑜𝑠𝑒 and 𝜆 𝑠𝑦𝑛𝑐 are the weight factors to control the three losses in the same numeric scale.We utilize the pre-trained [14] encoder E and decoder D as the foundation for learning in the latent space. Given that the input channel for the decoder in our method is 2 × 𝑑, we opt to substitute the first convolution layer of the decoder. Subsequently, we fine-tune both the encoder and decoder using frames from the training set. Specifically, in each iteration, we randomly select two frames 𝐹 1 and 𝐹 2 from a single video and then calculate the reconstruction loss as follows:EQUATION)Meanwhile, we introduce the perceptual loss [56] to enforce E and D to accurately reconstruct the frames in the image space:EQUATIONwhere 𝜙 represents the perceptual feature extractor [56] . Then the overall objective of encoder E and decoder D can be defined as:EQUATIONwhere 𝜆 𝑟𝑒𝑐 and 𝜆 𝑝𝑒𝑟 control the numeric scales.During the training of Diffusion UNet 𝜃 𝑢𝑛𝑒𝑡 , we randomly extract a video clip along with its corresponding audio clip. The first frame from this video clip serves as the reference image I. Utilizing the trained encoder E, we obtain the reference latent code 𝑥, as well as the ground truths for each latent image 𝑱 0 . Subsequently, we employ the trained Exp Transformer and Pose Transformer to acquire the β0 ∈ R 𝐹 ×𝐷 𝛽 and p0 ∈ R 𝐹 ×𝐷 𝑝 . Different from the sequence-tosequence Diffusion Transformer in the first stage, our Diffusion UNet generate each RGB frames one by one, so we extract the coefficient β ∈ R 𝐷 𝛽 and p ∈ R 𝐷 𝑝 for each frame. The training of our Diffusion UNet is facilitated by a tuple denoted as (𝑱 0 , 𝑡, β, p, 𝑥). Specifically, we add the random Gaussian noise on 𝑱 0 to obtain the noisy latent image 𝑱 𝑡 at the 𝑡-th timestep. We then optimize 𝜃 𝑢𝑛𝑒𝑡 using the following objective function:EQUATION) 
experiments 4.1 experimental setup: 
 Datasets. We use HDTF [58] and VFHQ [49] datasets to train our FD2Talk. HDTF is a large in-the-wild high-resolution and highquality audio-visual dataset that consists of about 362 different videos spanning 15.8 hours. The resolution of the face region in the video generally reaches 512 × 512. VFHQ is a large-scale video face dataset, which contains over 16000 high-fidelity clips of diverse interview scenarios. However, since VFHQ lacks audio components, it is exclusively utilized during the second phase of training. All videos are clipped into small fragments and cropped [37] to obtain the face region. Then we use Deep3d [10] , a single-image face reconstruction method, to recover the facial image and extract the relevant coefficients. Both HDTF and VFHQ are split 70% as the training set, 10% as the validation set, and 20% as the testing set. Moreover, we introduce VoxCeleb [28] to further evaluate our method, which contains over 100𝑘 videos of 1251 subjects.Implementation Detail. We train the model on video frames with 256 × 256 resolution. In the first stage, the 6-layer Exp Transformer and Pose Transformer are trained with a batch size of 1 and a generated sequence length of 25. In the second stage, we first finetune the pre-trained [14] encoder and decoder, and then train the Diffusion UNet with a batch size of 32, and the resolution of the latent image is 64 × 64. The two-stage framework is trained with the Adam [8] optimizer separately and can be inferred in an end-to-end fashion. The diffusion step is set to 1000 and 50 during training and inference, respectively. Our two-stage model is trained for approximately 8 and 32 hours using 8 NVIDIA 3090 GPUs.Baselines. We compare our method with several previous methods of audio-driven talking head generation, including Wav2Lip [29] , MakeItTalk [61] , SadTalker [57] , DiffTalk [36] , and DreamTalk [27] . We provide a reference image and audio signal as input for all methods. Note that Wav2Lip requires additional videos to offer head pose information, so we also fixed the head pose of our method for a fair comparison in quantitative evaluation.Evaluation Metrics. To evaluate the superiority of our proposed method, we consider three aspects: 1) Lip synchronization is assessed using two metrics: LSE-C [29] and SyncNet [7] . LSE-C measures the confidence score of perceptual differences in mouth shape from Wav2Lip, while the SyncNet score assesses the audio-visual synchronization quality. 2) Motion diversity is evaluated by extracting head motion feature embeddings using Hopenet [33] and calculating their standard deviations. Additionally, we use the Beat Align Score [38] to measure alignment between the audio and generated head motions. 3) Generated image quality is assessed using widely recognized metrics: FID [19] , PSNR, and SSIM [48] .We compare our method with previous state-of-the-art methods qualitatively. The results are visualized in Fig. 4 . While Wav2Lip can generate accurate lip movements, it falls short in producing high-quality images due to blurriness issues in the mouth region. Moreover, Wav2Lip focuses solely on animating the lips, neglecting other facial areas and resulting in a lack of motion diversity. MakeItTalk and SadTalker attempt to address some weaknesses of Wav2Lip, such as enhancing motion diversity. However, they still struggle to synthesize detailed facial features like apple cheeks and teeth due to generative limitations in GANs and regression models. For diffusion-based methods, DiffTalk combines appearance and motion during denoising, leading to inaccurate lip movement generation. DreamTalk, on the other hand, neglects head pose modeling and still relies on pre-trained render models, resulting in synthesized results with unreasonable head poses and slightly distorted facial regions. In contrast, our FD2Talk fully leverages powerful diffusion models in both stages and effectively separates appearance and motion information. These operations result in accurate lip movements, diverse head poses, and high-quality, lifelike talking head videos.We further quantitatively analyze the comparison between FD2Talk and previous state-of-the-art methods in lip synchronization, motion diversity, and image quality, on HDTF and VoxCeleb datasets.Our approach surpasses MakeItTalk, SadTalker, DiffTalk, and DreamTalk in terms of lip synchronization. We attribute this improvement to the alignment mask used during cross-attention in the Exp and Pose Transformer. This mask enables the predicted coefficients consistency with the corresponding audio signal. Additionally, the accurate lip movements are further enhanced by the lip synchronization loss with a well-pretrained lip expert. It is worth noting that although Wav2Lip achieves the highest lip accuracy, it neglects the overall naturalness and diversity of the results.When considering the three metrics of image quality, i.e., FID, PSNR, and SSIM, our approach significantly outperforms previous methods, which can be attributed to two aspects: 1) Our method maximizes the potential of diffusion models to generate more natural results compared to previous works using GANs, regression models, or partial diffusion models. 2) We disentangle complex facial information through two stages, enabling accurate motion prediction, and the creation of natural, high-fidelity appearance textures, ultimately resulting in superior and high-quality results.Moreover, our work surpasses previous methods in the diversity of head motions and achieves the best performance in Diversity and Beat Align Score. This achievement is attributed to our Pose Transformer, which predicts the head pose coefficients through the denoising process. The introduced random noise facilitates the generation of richer and more diverse pose results compared to previous methods.We also test the generalization of our FD2Talk model for out-ofdomain portraits. As demonstrated in Fig. 5 , whether the provided faces are paintings, cartoon portraits, or oil paintings, our FD2Talk can animate them using audio signals, ensuring lip synchronization while preserving the appearance details of the reference face with high fidelity, thus enhancing image quality. Moreover, the generated results include rich head poses, demonstrating excellent motion diversity as expected. This generalization ability stems from the decoupling of facial representation. In the first stage, we focus on generating appearance-independent motion information, which is solely linked to the audio signal and remains robust across various portrait domains.4.5.1 Decoupling Expressions and Head Poses. In the first stage, we decouple the Diffusion Transformers for the prediction of expressions and poses to address the one-to-many mapping issue. We compare this approach with a baseline where a Diffusion Transformer is used to jointly predict expression and pose coefficients.As shown in Tab. 2 and Fig. 6 , this baseline exhibits a noticeable decrease in lip synchronization and motion diversity. This is because lip movements are heavily influenced by facial expressions but have little correlation with head pose. On the other hand, motion diversity is closely related to predicted pose coefficients. Jointly learning these coefficients leads to mutual interference and makes training more challenging. Therefore, we choose to decouple the prediction of expression and pose coefficients using Exp and Pose Transformers, respectively.In the second stage, the predicted motion information and encoded appearance texture are passed through distinct cross-attention layers to guide the Diffusion UNet. We verify its effectiveness by comparing it with a baseline where we directly concatenate these two conditions and guide the denoising process. As demonstrated in Tab. 2 and Fig. 6 , concatenating motion and appearance leads to a decrease in each metric, particularly image quality, as we can observe the distortion in faces. We analyze that appearance textures constitute image-domain information, which is much higher than coefficient-domain motion. Therefore, decoupling them using two distinct cross-attention layers can significantly enhance the robustness of overall diffusion models and ensure convergence. We conduct user studies with 20 participants to evaluate the performance of all methods. We generate 30 test videos covering different genders, ages, styles, and expressions. For each method, participants are required to choose the best one based on three metrics: 1) lip synchronization, 2) head motion diversity, and 3) overall image quality. As demonstrated in Tab. 4, our work outperforms previous methods across all aspects, particularly in motion diversity and image quality. We attribute this to the decoupling of motion and appearance, as well as adopting diffusion models to generate higher-quality frames. 
conclusion: 
 Talking head generation is an important research topic that still faces great challenges. Considering the issues of previous works, such as reliance on generative adversarial networks (GANs), regression models, and partial diffusion models, and neglecting the disentangling of complex facial representation, we propose a novel facial decoupled diffusion model, called FD2Talk, to generate highquality, natural, and diverse results. Our FD2Talk fully leverages the strong generative ability of diffusion models and decouples the high-dimensional facial information into motion and appearance. We firstly utilize Diffusion Transformers to predict the accurate 3DMM expression and head pose coefficients from the audio signal, which serves as the decoupled motion-only information. Then these motion coefficients are fused into the Diffusion UNet, along with the appearance texture extracted from the reference image, to guide the generation of final RGB frames. Extensive experiments demonstrate that our approach surpasses previous methods in generating more accurate lip movements and yielding higher-quality and more diverse results. 
 
 Appendix: Proceedings of the 32nd ACM International Conference on Multimedia (MM '24), October 28-November 1, 2024, Melbourne, VIC, Australia. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3664647.3681238 


Decision: All reviewers agree that the proposed method is interesting and innovative. All reviewers comment that combining the advantages of diffusion models and decoupling complex facial information is a novel approach to talking head generation and contribute significantly to the increased performance of the presented approach. The reviewers are completely satisfied with the presented experimental study and that the rebuttal addressed a large majority of additionally raised questions. The authors are advised to include the relevant explanation in the final version of the paper. Given a general appreciation of the work by the reviewers, I believe that the paper will be of interest to a wider audience attending ACM-MM and would recommend a presentation of the work as a poster.

Reviews: review: paper_summary: paper_summary: This work proposes FD2Talk, a Facial Decoupled Diffusion model for Talking head generation, which addresses the limitations of previous approaches by decoupling facial details and leveraging diffusion models. Through extensive experiments, FD2Talk demonstrates enhanced image quality and the generation of more accurate and diverse results compared to previous state-of-the-art methods.
strength_weakness: strength_weakness: 1. The proposed FD2Talk model introduces a novel approach to talking head generation by leveraging the advantages of diffusion models and decoupling complex facial information. This unique combination sets it apart from previous works and contributes to its improved results.
2. The proposed approach is extensively evaluated through experiments, substantiating its effectiveness. The experiments demonstrate the superiority of FD2Talk over previous state-of-the-art methods in terms of image quality enhancement and generation accuracy.
limitations: limitations: 1. This work does not mention the generalizability of the proposed approach to different datasets or domains. It would be valuable to discuss any potential limitations or challenges that may arise when applying FD2Talk to other datasets or scenarios.
2. What about the super-resolution images/ videos?
paper_summary: paper_summary: In addressing the Talking Head Generation task, this paper proposes a Diffusion-based generation method leveraging audio decoupling, which includes two main innovations: 1. Utilizing a DiT structure to decouple audio into Expression and Head Pose components, and 2. Using the Exp Coefficients and Pose Coefficients obtained in the first stage, along with Appearance, to guide facial generation.
strength_weakness: strength_weakness: 1. The paper is well-written and logically structured.
2.The proposed audio decoupling method is rational, well-suited to the task, and easily comprehensible.
limitations: limitations: 1. While the method shows improvements over some metrics compared to methods from 2023 (although using FID and similar metrics for generative tasks can be inaccurate), it falls short when compared to some of the latest approaches, such as AniPortrait and VASA1, which are exceptionally well-executed. 
2. The condition injection methods proposed in Section 3.3 are quite common, relying on cross attention, and may lack deeper insights specific to the Talking Head Generation task. 
3. Additionally, the paper does not address temporal processing, which is often crucial for video generation.
paper_summary: paper_summary: This manuscript proposes a new method to improve the image quality and accuracy of talking head generation by leveraging the advantages of diffusion models and decoupling the complex facial information (motion and appearance) through multi-stages. Experimental results showcase the effectiveness and outperformance of the proposed method.
strength_weakness: strength_weakness: (1). The method has a certain novelty on decoupling facial information before diffusion model and is technically sound.
(2). The experimental evaluation is comprehensive and convincing.
(3). This manuscript is well-structured and presented in a clear and comprehensible manner.
limitations: limitations: (1). In Motion Generation stage, the authors utilize 3DMM's expression and head pose coefficients as output. Thus 3DMM's performance sets an upper bound on Motion Generation. Do the authors have any study on it? And how to deal with the accumulated error that 3DMM brings in?
(2). In frame generation stage, are the parameters in Encoder and Decoder trainable? If not, where to use L𝑒&𝑑(equation 15)?
(3). No comparisons with sota NeRF-based methods, such as ER-NeRF, GeneFace++, SyncTalk.
(4). Annotations in some figures are incomplete, e.g. no annotations of 𝜃𝑒𝑥𝑝 and 𝜃𝑝𝑜𝑠𝑒 in Fig.2, no annotation of reference latent code 𝑥 in Fig.3.
paper_summary: paper_summary: The paper proposes a facial decoupled diffusion model for talking head generation. The model first leverage Diffusion Transformer for head pose and expression modeling. Then, the frame synthesis diffusion model use appearance latent code and with head pose and expression code for conditional image synthesis. Experiments on HDTF and VoxCeleb shows the advantage over previous methods.
strength_weakness: strength_weakness: 1. The proposed decoupled diffusion model for head pose and expression is novel.
2. Experiments on HDTF and VoxCeleb shows the advantage over previous method in lip synchronization, motion diversity and image quality.
3. The OOD evaluation is a spotlight, which shows the excellent generalization ability of the model.
limitations: limitations: 1. My main concern is the efficiency. How is the inferencing speed, FLOPS, and total parameters of the model? Since 3 diffusion models with a lip-expert model are used, the efficiency may be an obstacle for practical use.
2. The performance of decoupled diffusion models may be limited with the accuracy of predicted 3DMM coefficients. How does the model with GT coefficients perform on the datasets?
3. The implementation details are unclear for the cross attention in frame synthesis diffusion model. Will the code be released?