AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 07:35:30

Abstract: The paper introduces AniTalker, an innovative framework designed to generate lifelike talking faces from a single portrait. Unlike existing models that primarily focus on verbal cues such as lip synchronization and fail to capture the complex dynamics of facial expressions and nonverbal cues, AniTalker employs a universal motion representation. This innovative representation effectively captures a wide range of facial dynamics, including subtle expressions and head movements. AniTalker enhances motion depiction through two self-supervised learning strategies: the first involves reconstructing target video frames from source frames within the same identity to learn subtle motion representations, and the second develops an identity encoder using metric learning while actively minimizing mutual information between the identity and motion encoders. This approach ensures that the motion representation is dynamic and devoid of identity-specific details, significantly reducing the need for labeled data. Additionally, the integration of a diffusion model with a variance adapter allows for the generation of diverse and controllable facial animations. This method not only demonstrates AniTalker’s capability to create detailed and realistic facial movements but also underscores its potential in crafting dynamic avatars for real-world applications. Synthetic results can be viewed at https://anitalker.github.io.

Main paper: 
 introduction: 
 Integrating speech signals with single portraits [13, 18, 31, 42, 44, [56] [57] [58] to generate talking avatars has greatly enhanced both the entertainment and education sectors, providing innovative avenues for interactive digital experiences. While current methodologies [34, 44, 54, 58, 59 ] have made notable strides in achieving synchronicity between speech signals and lip movements, thus enhancing verbal communication, they often neglect the critical aspect of nonverbal communication. Nonverbal communication encompasses the transmission of information without the use of words, including but not limited to specific head movements, facial expressions, and blinking. Research [33] indicates that these nonverbal cues are pivotal in communicating.The primary challenge lies in the inadequacy of existing models to encapsulate the complex dynamics associated with facial motion representation. Existing approaches predominantly employ explicit structural representations such as blendshapes [3, 13, 32] , landmark coefficients [18, 45, 57] , or 3D Morphable Models (3DMM) [7, 14, 25] to animate faces. Designed initially for single-image processing, these methods offer a constrained approximation of facial dynamics, failing to capture the full breadth of human expressiveness. Recent advancements [11, 24] have introduced trainable facial motion encoders as alternatives to conventional explicit features, showing significant progress in capturing detailed facial movements. However, their deployment is often tailored for specific speakers [11] or limited to the mouth region [24] , highlighting a gap in fine-grained motion representation that captures all varieties of facial dynamics.A universal and fine-grained motion representation that is applicable across different characters remains absent. Such a representation should fulfill three key criteria: capturing minute details, such as minor mouth movements, eye blinks, or slight facial muscle twitching; ensuring universality, making it applicable to any speaker while removing identity-specific information to maintain a clear separation between appearance and motion; and incorporating a wide range of nonverbal cues, such as expressions, head movements, and posture.In this paper, we introduce AniTalker. Our approach hinges on a universal motion encoder designed to grasp the intricacies of facial dynamics. By adopting the self-supervised learning paradigm, we mitigate the reliance on labeled data, enabling our motion encoder to learn robust motion representations. This learning process operates on dual levels: one entails understanding motion dynamics through the transformation of a source image into a target image, capturing a spectrum of facial movements, from subtle changes to significant alterations. Concurrently, the use of identity labels within the dataset facilitates the joint optimization of an identity recognition network in a self-supervised manner, further aiming to disentangle identity from motion information through mutual information minimization. This ensures that the motion representation retains minimal identity information, upholding its universal applicability.To authenticate the versatility of our motion space, we integrate a diffusion model and a variance adapter to enable varied generation and manipulation of facial animations. Thanks to our sophisticated representation and the diffusion motion generator, AniTalker is capable of producing diverse and controllable talking faces.In summary, our contributions are threefold:(1) We have developed universal facial motion encoders using a self-supervised approach that effectively captures facial dynamics across various individuals. These encoders feature an identity decoupling mechanism to minimize identity information in the motion data and prevent identity leakage. (2) Our framework includes a motion generation system that combines a diffusion-based motion generator with a variance adapter. This system allows for the production of diverse and controllable facial animations, showcasing the flexibility of our motion space. (3) Extensive evaluations affirm our framework's contribution to enhancing the realism and dynamism of digital human representations, while simultaneously preserving identity. 
related works: 
 Speech-driven Talking Face Generation refers to creating talking faces driven by speech, We categorize the models based on whether they are single-stage or two-stage. Single-stage models [34, 55, 58] generate images directly from speech, performing end-toend rendering. Due to the size constraints of rendering networks, this method struggles with processing longer videos, generally managing hundreds of milliseconds. The two-stage type [3, 11, 13, 18, 24, 31, 57] decouples motion information from facial appearance and consists of a speech-to-motion generator followed by a motion-to-video rendering stage. As the first stage solely generates motion information and does not involve the texture information of the frames, it requires less model size and can handle long sequences, up to several seconds or even minutes. This two-stage method is known to reduce jitter [3, 11, 24] , enhance speech-tomotion synchronization [11, 13, 31, 57] , reduce the need for aligned audio-visual training data [3, 24] , and enable the creation of longer videos [18] . Our framework also employs a two-stage structure but with a redesigned motion representation and generation process.Motion Representation serves as an essential bridge between the driving features and the final rendered output in creating talking faces. Current methods predominantly utilize explicit structural representations, such as blendshapes [3, 13, 30] , 3D Morphable Models (3DMMs) [25] , or landmarks [45, 57] . These formats offer high interpretability and facilitate the separation of facial actions from textures, making them favored as intermediary representations in facial generation tasks. However, due to the wide range of variability in real-world facial movements, they often fail to capture the subtle nuances of facial expressions fully, thus limiting the diversity and expressiveness of methods dependent on these representations. Our research is dedicated to expanding the spectrum of motion representation by developing a learned implicit representation that is not constrained by the limitations of explicit parametric models.Self-supervised motion transfer approaches [29, 38, 41, 45, 46, 48, 51] aim to reconstruct the target image from a source image by learning robust motion representations from a large amount of unlabeled data. This significantly reduces the need for labeled data. A key challenge in these methods is separating motion from identity information. They primarily warp the source image using predicted dense optical flow fields. This approach attempts to disentangle motion from identity by predicting distortions and transformations of the source image. However, information leakage occurs in practice, causing the target image to contain not just motion but also identity information. Building on this observation, we explicitly introduce identity modeling and employ the Mutual Information Neural Estimation (MINE) [1, 4] method to achieve a motion representation independent of identity.Diffusion Models [19] have demonstrated outstanding performance across various generative tasks [12, 17, 21, 36] . Recent research has utilized diffusion models as a rendering module [2, 11, 24, 27, 37, 40, 42] . Although diffusion models often produce higher-quality images, they require extensive model parameters and substantial training data to converge. To enhance the generation process, several approaches [18, 25, 26, 30, 52] employ diffusion models for generating motion representations. Diffusion models excel at addressing the one-to-many mapping challenge, which is crucial for speech-driven generation tasks. Given that the same audio clip can lead to different actions (e.g., lip movements and head poses) across different individuals or even within the same person, diffusion models provide a robust solution for managing this variability. Additionally, the training and inference phases of diffusion models, which systematically introduce and then remove noise, allow for the incorporation of noise during generation to foster diversity. We also use diffusion in conjunction with our motion representation to further explore diversity in talking face generation. 
anitalker framework 3.1 model overview: 
 AniTalker contains two critical components: (1) Training a motion representation that can capture universal face dynamics, and (2) Based on the well-trained motion encoder from the previous step, the generation or manipulation of the motion representation using the user-controlled driving signal to produce the synthesised talking face video.Our approach utilizes a self-supervised image animation framework, employing two RGB images from a video clip: a source image 𝐼 𝑠 and a target image 𝐼 𝑡 (𝐼 ∈ R 𝐻 ×𝑊 ×3 ), to serve distinct functions: 𝐼 𝑠 provides identity information, whereas 𝐼 𝑡 delivers motion details. The primary aim is to reconstruct 𝐼 𝑡 . Due to the random selection of frames, occasionally adjacent frames are chosen, enabling the network to learn representations of subtle movements. As depicted in Figure 2 (a), both the source and target images originate from the same video clip. Through this self-supervised learning method, the target image's encoder is intended to exclusively capture motion information. By learning from frame-to-frame transfer, we can acquire a more universal representation of facial motion. This representation includes verbal actions such as lip movements, as well as nonverbal actions, including expressions, posture, and movement.To explicitly decouple motion and identity in the aforementioned processes, we strengthen the self-supervised learning approach by incorporating Metric Learning (ML) and Mutual Information Disentanglement (MID). Specifically: Metric Learning. Drawing inspiration from face recognition [8, 43] and speaker identification [9] , metric learning facilitates the generation of robust identity information. This technique employs a strategy involving pairs of positive and negative samples, aiming to minimize the distance between similar samples and maximize it between dissimilar ones, thereby enhancing the network's ability to discriminate between different identities. This process can also proceed in a self-supervised fashion, with each iteration randomly selecting distinct identities from the dataset. Specifically, the approach establishes an anchor (𝑎) and selects a positive sample (𝑝) and a negative sample (𝑛)-corresponding to faces of different identities-with the goal of reducing the distance (𝑑) between the anchor and the positive sample while increasing the distance between the anchor and the negative samples. This optimization, depicted in Figure 2 (b), involves randomly selecting a different identity from a list of candidates not belonging to the current person as the negative sample. The optimization goal for this process is as follows:L 𝑀𝐿 = max (0, 𝑑 (𝑎, 𝑝) -𝑑 (𝑎, 𝑛) + margin)Here, the margin is a positive threshold introduced to further separate the positive and negative samples, thus improving the model's ability to distinguish between different identities.Mutual Information Disentanglement. Although metric learning effectively constrains the identity encoder, focusing solely on this encoder does not adequately minimize the identity information within the motion encoder. To tackle this issue, we utilize Mutual Information (MI), a statistical measure that evaluates the dependency between the outputs of the identity and motion encoders. Given the challenge of directly computing MI between two variables, we adopt a parametric method to approximate MI estimation among random variables. Specifically, we use CLUB [4] , which estimates an upper bound for MI. Assuming the output of the identity encoder is the identity latent 𝑧 𝑖𝑑 and the motion encoder's output is the motion latent 𝑧 𝑚 , our goal is to optimize the mutual information 𝐼 (E (𝑧 𝑖𝑑 ); E (𝑧 𝑚 )), where E denotes the learnable Multi-Layer Perceptron (MLP) within CLUB. This optimization ensures that the motion encoder primarily captures motion, thereby preventing identity information from contaminating the motion space. This strategy is depicted in Figure 2 (c) .In summary, by leveraging Metric Learning and Mutual Information Disentanglement, we enhance the model's capacity to accurately differentiate between identity and motion while reducing reliance on labeled data.Hierarchical Aggregation Layer (HAL). To enhance the motion encoder's capability to understand motion variance across different scales, we introduce the Hierarchical Aggregation Layer (HAL). This layer aims to integrate information from various stages of the image encoder, each providing different receptive fields [23] . HAL processes inputs from all intermediate layers of the image encoder and passes them through an Average Pooling (AvgPool) layer to capture scale-specific information. A Weighted Sum [50] layer follows, assigning learnable weights to effectively merge information from these diverse layers. This soft fusion approach enables the motion encoder to capture and depict movements across a broad range of scales. Such a strategy allows our representations to adapt to faces of different sizes without the need for prior face alignment or normalization.Specifically, the features following the AvgPool layer are denoted as [𝑚 1 , 𝑚 2 , . . . , 𝑚 𝑛 ], representing the set of averaged features, with [𝑤 1 , 𝑤 2 , . . . , 𝑤 𝑛 ] as the corresponding set of weights, where 𝑛 symbolizes the number of intermediate layers in the image encoder. These weights undergo normalization through the softmax function to guarantee a cumulative weight of 1. The equation for the weighted sum of tensors, indicating the layer's output, is formulated as m = 𝑛 𝑖=1 𝑤 𝑖 • 𝑚 𝑖 . The softmax normalization process is mathematically articulated as 𝑤 𝑖 = 𝑒 𝑊 𝑖 𝑛 𝑗 =1 𝑒 𝑊 𝑗 , ensuring the proportional distribution of weights across the various layers. Subsequently, m is fed into the motion encoder for further encoding.Learning Objective. The main goal of learning is to reconstruct the target image by inputting two images: the source and the target within the current identity index. Several loss functions are utilized during the training process, including reconstruction loss 𝐿 𝑟𝑒𝑐𝑜𝑛 , perceptual loss 𝐿 𝑝𝑒𝑟𝑐𝑒𝑝 , adversarial loss 𝐿 𝑎𝑑𝑣 , mutual information loss 𝐿 𝑀𝐼 , and identity metric learning loss 𝐿 𝑀𝐿 . The total loss is formulated as follows:𝐿 𝑚𝑜𝑡𝑖𝑜𝑛 = 𝐿 𝑟𝑒𝑐𝑜𝑛 + 𝜆 1 𝐿 𝑝𝑒𝑟𝑐𝑒𝑝 + 𝜆 2 𝐿 𝑎𝑑𝑣 + 𝜆 3 𝐿 𝑀𝐼 + 𝜆 4 𝐿 𝑀𝐿Once the motion encoder and image renderer are trained, at the second stage, we can freeze these models. The motion encoder is used to generate images, then video-driven or speech-driven methods are employed to produce motion, and finally, the image renderer carries out the final frame-by-frame rendering.Video driving, also referred to face reenactment, leverages a driven speaker's video sequenceI 𝑑 = [𝐼 𝑑1 , 𝐼 𝑑 2 , . . . , 𝐼 𝑑 𝑇 ] to animate a source image 𝐼 𝑠 , resulting in a video that accurately replicates the driven poses and facial expressions. In this process, the video sequence I 𝑑 is input into the motion encoder, previously trained in the first phase, to extract the motion latent. This latent, along with 𝐼 𝑠 , is then directly fed, frame by frame, into the image renderer for rendering. No additional training is required. The detailed inference process, where the orange lines represent the data flow during video-driven inference, is depicted in Figure 2 (e).Pipeline. Unlike video-driven methods that use images, the speech-driven approach generates videos consistent with the speech signal or other control signals to animate a source image 𝐼 𝑠 . Specifically, we utilize a combination of diffusion and variance adapters: the former learns a better distribution of motion data, while the latter mainly introduces attribute manipulation.Diffusion Models. For generating motion latent sequences, we utilize a multi-layer Conformer [16] . During training, we incorporate the training process of diffusion, which includes both adding noise and denoising steps. The noising process gradually converts clean Motion Latent M into Gaussian noise M 𝑇 , where 𝑇 represents the number of total denoising steps in the diffusion process. Conversely, the denoising process systematically eliminates noise from the Gaussian noise, resulting in clean Motion Latents. This iterative process better captures the distribution of motion, enhancing the diversity of the generated results. During the training phase, we adhere to the methodology described in [19] for the DDPM's training stage, applying the specified simplified loss objective, as illustrated in Equation 1, where 𝑡 represents a specific time step and C represents the control signal, which refers to either speech or speech perturbed by a Variance Adapter (to be discussed in the following section). For inference, considering the numerous iteration steps required by diffusion, we select the Denoising Diffusion Implicit Model (DDIM) [39] -an alternate non-Markovian noising process-as the solver to quicken the sampling process.EQUATIONVariance Adapter. The Variance Adapter [35] is a residual branch connected to audio features, allowing optional control over the speech signal. Originally proposed to mitigate the one-to-many problem in Text-to-Speech (TTS) tasks, its architecture includes a predictor and an encoder that use speech signals to predict attribute representations. A residual connection is then applied between the encoder output and the speech signals. During the Training Stage, the encoder processes speech features in collaboration with the predictor to minimize the L2 loss against a ground truth control signal. This includes incorporating an attribute extractor for targeting specific attributes, such as employing a pose extractor (yaw, pitch, roll) to control head posture during the audio generation process. In the Inference Stage, the trained encoder and predictor can flexibly synthesize speech with controlled attributes or operate based on speech-driven inputs. The detailed structure is depicted in Figure 3 . Our approach extends previous works [11, 18] by incorporating LSTM [15] for improved temporal modeling and introducing additional cues such as head position and head scale, which we refer to as camera parameters. The architecture is detailed in Figure 3 .Learning Objective. The total loss comprises diffusion loss and variance adapter loss, where 𝐾 represents the number of attributes:𝐿 gen = 𝐿 diff + 𝜆 𝐾 ∑︁ 𝑘=1 𝐿 var 𝑘 
experiments 4.1 experimental settings: 
 We utilizes three datasets: VoxCeleb [28] , HDTF [56] , and VFHQ [49] . Due to different processing approaches across these datasets, we re-downloaded the original videos and processed them in a unified way. Specifically, our processing pipeline included filtering out blurred faces and faces at extreme angles. It is noted that we did not align faces but instead used a fixed detection box for each video clip, allowing for natural head movement. This effort resulted in a dataset containing 4,242 unique speaker IDs, encompassing 17,108 video clips with a cumulative duration of 55 hours. Details of this filtering process are provided in the supplementary material. Each video in these datasets carries a unique facial ID tag, which we used as labels for training our identity encoder. We also reserved some videos from HDTF for testing, following the test split in [55] .Scenario Setting We evaluate methods under two scenarios: video-driven and speech-driven, both operating on a one-shot basis with only a single portrait required. The primary distinction lies in the source of animation: image sequences for video-driven and audio signals for speech-driven scenarios. The detailed data flow for inference is illustrated in Figure 2 . Additionally, each scenario is divided into two types: self-driven, where the source and target share the same identity, and cross-driven, involving different identities. In speech-driven tasks, if posture information is needed, it is provided from the ground truth. Moreover, for our motion generator, unless specified otherwise, we use a consistent seed to generate all outcomes. To ensure a fair comparison, the output resolution for all algorithms is standardized to 256 × 256.Implementation Details In training the motion representation, our self-supervised training paradigm is primarily based on LIA [46] . Both the identity and motion encoders employ MLPs. Our training targets use the CLUB1 for mutual information loss, in conjunction with AAM-Softmax [43] . This robust metric learning method utilizes angular distance and incorporates an increased number of negative samples to enhance the metric learning loss. In the second phase, the speech encoder and the Motion Generator utilize a four-layer and a two-layer conformer architecture, respectively, inspired by [11, 24] . This architecture integrates the conformer structure [16] and relative positional encoding [6] . A pre-trained HuBERT-large model [20] serves as the audio feature encoder, incorporating a downsampling layer to adjust the audio sampling rate from 50 Hz to 25 Hz to synchronize with the video frame rate. The training of the audio generation process spans 125 frames (5 seconds). Detailed implementation specifics and model structure are further elaborated in the supplementary materials.Evaluation Metric For objective metrics, we utilize Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [47] , and Learned Perceptual Image Patch Similarity (LPIPS) [53] to quantify the similarity between generated and ground truth images. Cosine Similarity (CSIM)2 measures facial similarity using a pretrained face recognition. Lip-sync Error Distance (LSE-D) [5] assesses the alignment between generated lip movements and the corresponding audio. Regarding subjective metrics, we employ the Mean Opinion Score (MOS) as our metric, with 10 participants We benchmarked our approach against several leading face reenactment methods [29, 38, 41, 45, 46, 51] , all employing variations of self-supervised learning. The results are presented in Table 1 . Due to the inherent challenges and the absence of frame-by-frame ground truth in Cross-Reenactment (using another person's video for driving), the overall results tend to be lower compared to Self-Reenactment (using the current person's video). In Self-Reenactment, our algorithm achieved superior results for image structural metrics such as PSNR, SSIM, and LPIPS, validating the effectiveness of our motion representation in reconstructing images. Additionally, using the CSIM metric to measure face similarity, we observed that the similarity between the reconstructed face and the original portrait was the second highest, slightly behind MTIA [41] , illustrating our model's identity preservation capabilities. For Cross-Reenactment, where the portrait serves as ground truth and considering cross-driven deformations, we focused on high-level metrics: SSIM and LPIPS. Our method demonstrated commendable performance. We also evaluated CSIM, which, unlike self-reenactment, showed a significant improvement, achieving the best results among these datasets. This highlights our algorithm's outstanding ability to disentangle identity and motion when driving with different individuals.Qualitative Results To highlight comparative results, we conducted a cross-reenactment scenario analysis with different algorithms, as presented in Figure 4 . The objective was to deform the source portrait using the actions of the target. Each row in the figure represents a driving case. We observed that baseline methods exhibited varying degrees of identity leakage, where the identity information from the target contaminated the source portrait's identity. For example, as demonstrated in the fourth row, the slim facial structure of the driving portrait led to slimmer outcomes, which was unintended. However, our results consistently preserved the facial identity. Additionally, in terms of expression recovery, as evident in the first and third rows, our approach replicated the action of opening the eyes in the source portrait accurately, creating a natural set of eyes. In contrast, other algorithms either produced slight eye-opening or unnatural eyes. These qualitative findings highlight the advantage of decoupling ability. We compare our method against existing state-of-the-art speechdriven approaches, including MakeItTalk [59] , PC-AVS [58] , Au-dio2Head [44] , and SadTalker [54] . Quantitative results are presented in Table 2 . From the subjective evaluation, our method consistently shows improvements in fidelity, lip-sync accuracy, naturalness, and a reduction in motion jittering, particularly noted for the enhanced naturalness of movements. These advancements can be attributed to our sophisticated universal motion representation. The objective evaluation involves driving the image with its audio. Compared to these methods, our approach shows significant improvements in SSIM and CSIM. However, our Sync-D metric shows a decrease, which we believe is due to two main reasons: (1) we do not use this metric as a supervisory signal, and (2) the Sync-D metric focuses on short-term alignment and does not adequately represent long-term information that is more crucial for the comprehensibility of generated videos. This is also corroborated by the qualitative results shown in Figure 5 , highlighting our model's ability to produce convincingly synchronized lip movements to the given phonetic sounds. To further validate the effectiveness of our disentanglement between motion and identity, we conducted tests using various methods. Initially, to evaluate the performance of developing a reliable identity encoder using only Metric Learning (ML) without Mutual Information Disentanglement (MID), we assessed both Triplet loss [10] and AAM-Softmax [43] .Our results indicate that AAM-Softmax, an angle-based metric, achieves superior outcomes in our experiments. Additionally, by incorporating a mutual information decoupling module alongside AAM-Softmax, we noted further improvements in results. This enhancement encouraged the motion encoder to focus exclusively on motion-related information. These findings are comprehensively detailed in Table 3 . [22] (136 dimensions) for the landmark representation and expression parameters (50 dimensions) from EMOCA [7] for the 3DMM representation. In self-reenactment scenarios, all rendering methods were kept consistent, and different features were used to generate driven images. We observed several key points: (1) As shown in Table 4 , our learned representation exhibits a more compact dimensionality, indicating a more succinct encoding of facial dynamics. (2) Our video comparisons show that, unlike these explicit representations, our implicit motion representation maintains frame stability without the need for additional smoothing. This can be attributed to our self-supervised training strategy of sampling adjacent frames, which effectively captures subtle dynamic changes while inherently ensuring temporal stability.  
discussion: 
 Discussion on Universal Motion Representation Our investigations into the model's ability to encode facial dynamics have highlighted a universal representation of human facial movements. As depicted in Figure 7 , we observed that different individuals maintain consistent postures and expressions (such as turning the head left, speaking with homophones, and closing eyes) at each point within our motion space, demonstrating that our motion space forms a Motion Manifold. This manifold facilitates the representation of a continuous motion space, enabling the precise modeling of subtle facial feature variations and allowing for smooth transitions. Additionally, by integrating perturbations through diffusion noise, our model can simulate random, minute motion changes that align with fundamental movement patterns, thus enhancing the diversityTurn Head Left of generated expressions. These findings demonstrate that our motion representation has a robust capacity to capture and represent a wide array of human facial movements.Discussion on Generalization Ability Although our model is trained on real human faces, it demonstrates the ability to generalize to other images with facial structures, such as cartoons, sculptures, reliefs, and game characters. This underscores the model's excellent scalability. We primarily attribute this capability to the complete decoupling of identity and motion, which ensures that the model grasps the intrinsic nature of facial movements, thereby enhancing its generalization capability. 
conclusion: 
 The AniTalker framework represents a significant advancement in the creation of lifelike talking avatars, addressing the need for a fine-grained and universal motion representation in digital human animation. By integrating a self-supervised universal motion encoder and employing sophisticated techniques like metric learning and mutual information disentanglement, AniTalker effectively captures the subtleties of both verbal and non-verbal facial dynamics. The resulting framework not only achieves enhanced realism in facial animations but also demonstrates strong generalization capabilities across different identities and media. AniTalker sets a new benchmark for the realistic and dynamic representation of digital human faces, promising broad applications in entertainment, communication, and education.Limitation and Future Work While AniTalker shows promise in generalizing motion dynamics, it still faces challenges. Our rendering network generates frames individually, which can lead to inconsistencies in complex backgrounds. Additionally, limited by the performance of the warping technique, extreme cases where the face shifts to a large angle may result in noticeable blurring at the edges. Future work will focus on improving the temporal coherence and rendering effects of the rendering module.  
 
 Appendix: https://github.com/Linear95/CLUB/https://github.com/dc3ea9f/vico_challenge_baseline 


Decision: AniTalker proposes a novel framework for generating talking faces from a single portrait and audio input. The reviewers acknowledge the clarity of writing, promising results (dkjp, Yf2S), effectiveness of decoupled motion representation (1mt1, dkjp), and well-discussed related work (Yf2S). However, there are concerns regarding the completeness of explanations, particularly for the second rendering pipeline, and the lack of comparison with recent methods (1mt1, Yf2S).

**Strengths:**

* Clear and well-organized writing (dkjp, Yf2S).
* High-quality talking face generation (dkjp, Yf2S).
* Effective decoupling of identity and motion information (1mt1, dkjp).
* Comprehensive overview of related work (Yf2S).

**Weaknesses:**

* **Second Rendering Pipeline:**
    * Lack of explanation for training and denoising (Yf2S).
    * Unclear justification for necessity (Yf2S).
    * Unclear presentation in Figure 2(e) (Yf2S). 
* **Comparisons:** Limited comparison with recent speech-driven methods (1mt1).

While AniTalker presents a promising approach, the need for a clearer explanation of the second rendering pipeline and a more comprehensive comparison with recent work prevent its acceptance for oral presentation. However, the overall quality of the research and the reviewers' generally positive evaluations make it suitable for poster presentation. This will allow the authors to showcase their work and receive feedback from the conference attendees.

The authors should consider revising their paper to address the following before the conference:

* Provide a comprehensive description of the training process for the diffusion-based module, including the denoising methods used.
* Enhance the clarity of Figure 2(e) to better represent the second pipeline's architecture.
* Include comparisons with recent speech-driven talking face generation methods in the final version of the paper (supplemental material is acceptable).

Reviews: review: paper_summary: paper_summary: This paper proposes a new speech driven talking face generation method AniTalker which employs a universal motion representation to capture a wide range of facial dynamics, including subtle expressions and head movements.
strength_weakness: strength_weakness: - The new universal facial motion encoders are a good way to disentangle identity and motion from input portrait images, especially the introduction of MI and MID modules. 
- Paper writing is good to follow and the content is well organized. Paper contains many details about training, data cleaning, module design, etc.
limitations: limitations: - What are the flow fields, how are they computed and what about their functions in the image renderer module?
- In the image renderer, looks like it inputs only the target image's motion latent from the motion encoder, without the source image's motion latent? If so, how to measure the motion difference between source and target image?
- Does this paper use the pretrained CLUB model to compute MI between identity code and motion code? If so, can the CLUB method handle any general type and any dimensional data? Why is this measurement reliable on the identity code and motion code originally encoded from images?
- Figure 6 and Section 4.4.3 analyzes the impact of HAL layers. From them, I can only find the different impacts from different layers, but why does it prove HAL is helpful for dynamic representation? And why can the normalized weights' magnitude show the impact of the layer? 
- One suggestion is to add some descriptions about reconstruction loss, perceptual loss, adversarial loss used in training, since they are also important parts of the final loss. I can see from supplemental material that they just follow the definition of LIA, but still it worth noticing how they are constructed. 
- How are head poses introduced into the network? Does the network use original groundtruth head poses or predict poses from input audio/video directly? There are some descriptions from Section B2 in supplemental material, but still no information about how to use head poses in the main pipeline. Meanwhile, from supplemental videos I can see the results have the same head poses for different images input. Are these poses some groundtruth poses or predicted?
- Experiments are quite thorough but miss some recent speech-driven methods to compare. Most of the compared method in this paper are a little old. Here I added some new ones as below. And actually DiffTalk [37] is also a recent method, but authors only cite it without detailed comparisons.
- Supplemental videos don't contain comparisons with previous methods. Impossible to see if AniTalker surpasses others or not. 

References:
- [CVPR 2023] PD-FGC: Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis
- [AAAI 2023] StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles
- [ICCV 2023] Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation
- [CVPR 2023] TalkLip: Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
paper_summary: paper_summary: This paper presents an innovative method for talking face generation. Given a target face image and a motion control signal (video or speech), the method enables high-quality talking face generation. By learning a universal motion representation, the method can capture diverse facial motions.
strength_weakness: strength_weakness: 1. The supplementary videos are excellent, which will contribute to the research on talking face generation.
2. The paper is written in a clear and logical manner. Sufficient experimental results are shown in the paper to demonstrate the validity of the proposed method.
3. A training strategy that effectively decouples identity information from motion information is proposed in the paper. By utilising metric learning and mutual information decoupling, the strategy can effectively suppress the leakage of identity information into motion information.
limitations: limitations: 1. The generated result is $256 \times 256$ resolution, which can be extended to $512 \times 512$ or even higher resolution in the future research.
2. For some examples of complex backgrounds, the generated results still show background shaking, which can be further optimized.
3. In the introduction to metric learning in Section 3.2 (lines 360-364), the method requires the estimation of an anchor, but it is not stated that the anchor is an image with the same identity as the positive sample? In addition, there are similarities between the method and triplet loss, please elaborate on the differences if possible.
paper_summary: paper_summary: The paper introduces AniTalker, a framework for generating lifelike talking faces from a single portrait. Unlike existing models that focus primarily on lip synchronization and overlook the complexity of facial expressions and nonverbal cues, The proposed work employs a universal motion representation to capture a wide range of facial dynamics, including subtle expressions and head movements.

AniTalker enhances motion depiction through two self-supervised learning strategies: first, by employing metric learning (ML) to encode identity-related information from the source face image frame, and second, by developing a motion encoder using Mutual Information Disentanglement to separate spatial information within the same identity. To ensure that motion representation remains dynamic and free of identity-specific details while reducing reliance on labeled data.

Additionally, the work integrates a diffusion model with a variance adapter, enabling the generation of diverse and controllable facial animations from the audio as a reference source.
strength_weakness: strength_weakness: 1. The related work is well-discussed, providing a comprehensive overview of existing models and their limitations in capturing facial dynamics.
2. The rendered results exhibit high fidelity and realism, indicating the effectiveness of the proposed work in generating lifelike facial animations.
3. The encoder modules utilized for extracting identity and spatial information are well-described, with additional support from the Hierarchical Aggregation Layer (HAL), enhancing the understanding of the underlying mechanisms driving motion depiction
limitations: limitations: 1. The second pipeline utilized for rendering frames from the audio sequence lacks proper description and explanation. There is insufficient detail provided on how the training of the diffusion module is conducted, how denoising is performed to render the sequence of frames.
2. The presentation of the second pipeline in Figure 2 (e), representing the framework of the model, is unclear and incomplete. It fails to provide a comprehensive overview of the second pipeline's architecture, leading to confusion regarding its purpose and necessity.
3. The necessity of introducing the second pipeline for rendering frames from the audio sequence is not adequately justified. There is a lack of clarity on how this pipeline enhances the overall performance or capabilities of the model, raising questions about its relevance and contribution to the proposed framework.
4. The absence of detailed explanations and clear presentation of the second pipeline contributes to the complexity and confusion surrounding its implementation and effectiveness. This can hinder the understanding of the model's operation and impede its usability and reproducibility.
5. To address these weaknesses, the authors should provide comprehensive descriptions of the training process for the diffusion-based module, elucidate the denoising methods utilized.  Additionally, enhancing the presentation of the second pipeline in visual representations such as figures or diagrams can improve its clarity and facilitate better understanding for readers.