DEITalk: Speech-Driven 3D Facial Animation with Dynamic Emotional Intensity Modeling

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:35:05

Abstract: Speech-driven 3D facial animation aims to synthesize 3D talking head animations with precise lip movements and rich stylistic expressions. However, existing methods exhibit two limitations: 1) they mostly focused on emotionless facial animation modeling, neglecting the importance of emotional expression, due to the lack of high-quality 3D emotional talking head datasets, and 2) several latest works treated emotional intensity as a global controllable parameter, akin to emotional or speaker style, leading to over-smoothed emotional expressions in their outcomes. To address these challenges, we first collect a 3D talking head dataset comprising five emotional styles with a set of coefficients based on the MetaHuman character model and then propose an end-to-end deep neural network, DEITalk, which conditions on speech and emotional style labels to generate realistic facial animation with dynamic expressions. To model emotional saliency variations in long-term audio contexts, we design a dynamic emotional intensity (DEI) modeling module and a dynamic positional encoding (DPE) strategy. The former extracts implicit representations of emotional intensity from speech features and utilizes them as local (high temporal frequency) emotional supervision, whereas the latter offers abilities to generalize to longer speech sequences. Moreover, we introduce an emotion-guided feature fusion decoder and a four-way loss function to generate emotion-enhanced 3D facial animation with controllable emotional styles. Extensive experimental results demonstrate that our method outperforms existing state-of-the-art methods. We recommend watching the video demo provided in our supplementary material for detailed results.

Main paper: 
 introduction: 
 Speech-driven 3D facial animation aims to synthesize realistic facial movements of 3D characters based on arbitrary speech input. This capability has significant applications in various industries such as film production, video games, and virtual reality [9, 32, 38, 46] . Thus, it has garnered significant attention from both academia and industry recently. Learning-based methods [5, 11, 15] for speechdriven 3D facial animation have become a focal point of current research. Compared to traditional methods that require professional animators to manually create animations based on a set of rules, these data-driven approaches can greatly reduce annotation and time costs while achieving realistic 3D character facial animations.However, recent methods still suffer from two major limitations. First, due to the absence of suitable datasets, many speech-driven 3D efforts [6, 10, 31, 39] focused mainly on synthesizing highly synchronized lip movements, as well as reconstructing natural upper facial movements. Moreover, they neglect the significance of emotions in facial animation expression, leading to the results falling short of realism. Secondly, some works view emotional information as a style embedding and represent them in the form of one-hot vectors [16, 20, 37] . Although such a strategy provides an effective means of controlling emotional expression in animation, it tends to steer the learned results towards the mean of different emotional styles, i.e., losing personalized expression of emotions.We argue that facial animation consists primarily of two distinct components: speech and emotional expression. Lip movements, driven by speech, represent a high-frequency phenomenon that requires precise synchronization with speech. Conversely, emotions can be characterized by two sub-components: emotional style, a consistent, low-frequency pattern, and emotional intensity, a relatively higher-frequency descriptor influenced by the level of emotional significance at various moments. However, estimating dynamic emotional intensity from audio or video using data-driven methods remains a challenging field. This challenge mainly arises from the difficulty of constructing large-scale datasets containing emotional intensity annotations for audio or video content, which requires expert evaluation of data across temporal dimensions and incurs significant labor and time costs. Furthermore, emotional intensity perception is inherently subjective and complex, influenced by individual expression variances and cultural disparities, thus making it difficult to establish a unified standard for evaluation. Certain approaches treat emotional intensity as a global control condition, enabling emotional intensity editing [7, 28] , while they may restrict the expressive range of the model for emotions.To synthesize realistic emotional 3D talking head animation, we propose a novel speech-driven 3D facial animation method (shown in Figure 1 ) called DEITalk, where a dynamic emotional intensity modeling module is proposed as our key contribution (illustrated in Figure 2 ). This module learns a shared latent space for audio and visual modalities via contrastive learning, which can be used to model the saliency of emotions temporally, i.e., dynamic emotional intensity. In addition, we design an emotion-guided feature fusion decoder, in which multiple different types of features are decoded by a Temporal Convolutional Network (TCN) [23] and output 185 emotion-enhanced control rig coefficients for facial expression rendering. Furthermore, a dynamic positional encoding strategy and a diversified loss function are further proposed to better enhance the model's capability for emotional expression.To address the lack of a publicly available 3D emotional talking head dataset, we adopt a combination to capture 3D facial movements: the MetaHuman character model [13] , developed by Epic, and the Live Link Face application, available in the iOS App Store. Subsequently, 185 control rig coefficients are acquired through associated MetaHuman plugins, where each coefficient corresponds to a group of facial muscles. Using the aforementioned methods, we establish a novel 3D emotional talking head dataset, termed ETH-3D. In contrast to existing 3D scan datasets such as VOCASET [6] and BIWI [12] , ETH-3D employs low-dimensional data (i.e., control rig) to depict rich and intricate facial motions. This data serves as a universal facial motion representation that can be generalized to any predefined MetaHuman character.In summary, the primary contributions of our work include:• First, we propose a novel approach, DEITalk, for speechdriven 3D facial animation, which enables the synthesis of realistic expressions with a specified emotional style while adapting to the dynamic emotional intensity in the speech. • Second, we introduce a dynamic emotional intensity modeling module that employs a contrastive learning strategy to learn the correlation between speech and facial movements in emotional saliency. The module generates joint embeddings which are then transformed into dynamic emotional intensity features. These features guide the model in synthesizing convincing facial expression changes. • Third, we capture a 3D emotional talking head dataset, ETH-3D, comprising five distinct emotional styles. The dataset includes audio-visual pairs that demonstrate reasonable dynamic variations in emotional intensity. • Finally, we are among the pioneering developers of generation models based on the MetaHuman control rig. In comparison to mesh-based models, our approach not only synthesizes more detailed and natural facial animations but is also better suited for integration into industrial pipelines. 
related work 2.1 speech-driven 3d facial animation: 
 With the rapid development and widespread adoption of 3D games and virtual reality technologies, traditional speech-driven or imagedriven 2D facial animation methods [4, 14, 17, 18, 34, 43, 44] cannot be directly applied to 3D character models. Consequently, speechdriven 3D facial animation has recently garnered significant attention [3, 5, 6, 10, 11, 15, 16, 19, 20, 27, 28, 31, 33, 35, 39, 45] .Most previous methods have significant limitations due to the lack of high-quality publicly available 3D datasets for various emotions. For instance, both VOCA [6] and Faceformer [10] are only capable of achieving decent lip movements but exhibit limited upper facial movements because they are trained on the VOCASET dataset [6] , which mainly focuses on descriptions of lip movements. Although Meshtalk [31] enhances diversity in upper facial movements by learning a categorical latent space, it fails to convey different emotions. Substantially, the above methods focus solely on achieving different speaking styles among individuals for animation, controlled by inserting style vectors of training individuals, while overlooking an important factor that affects the perception of animation: emotion.Recently, Emotalk [28] introduced an emotion disentanglement mechanism to extract emotional information from speech input. However, it relies on curated training data, limiting its effectiveness in extracting emotions in zero-shot settings. Furthermore, since it only offers control over the intensity of emotional expression rather than the emotional type, users face challenges in obtaining animations representing desired emotional categories. 3D-TalkEmo [37] utilizes emotional style labels to adjust facial animation's emotional expression. However, due to the lack of description regarding emotional expression intensity and treating emotional style labels as global control information, the resulting animations for the same emotional style are highly similar, lacking high-frequency details related to speech input.In contrast to existing works, this paper conceptualizes the intensity of emotion as a high-frequency implicit learnable attribute that exhibits correlations with both speech and facial movements. Through the integration of this implicit attribute with explicit emotional styles controlled by users, the model is guided to generate more nuanced and intricate facial expressions.Unlike the significant advancements witnessed in recent years in deep learning models [26, 40] for speech emotion recognition (SER) tasks, estimating the intensity of emotional reactions in audio or video using data-driven approaches remains a highly challenging endeavor. This is primarily attributed to the inherently subjective nature of human perception regarding the intensity of emotional expression and the difficulty in establishing universally applicable quantitative assessment metrics for emotional intensity.A task of emotion reaction intensity (ERI) estimation was proposed by the fifth Affective Behavior Analysis in the wild (ABAW) competition [21] . ABAW released an audio-visual dataset called Figure 2 : Overview of our DEITalk, which takes a speech sequence A 1:𝑇 and emotional style e 𝑠𝑡 𝑦 as input. The speech sequence undergoes initial encoding using Wav2Vec 2.0, followed by a linear interpolation layer for speech feature resampling. Subsequently, the resampled speech feature is passed through the dynamic emotional intensity modeling module to extract emotional intensity features. Next, a dynamic positional encoding strategy is applied to the speech feature. Finally, the speech feature, along with the emotional intensity feature and the emotional style embedding, is decoded by an emotion-guided feature fusion decoder. This decoder outputs control rig coefficients, which can then be applied to manipulate MetaHuman characters, resulting in emotive facial animations.Hume-Reaction, where individuals' reactions to emotional stimuli are annotated across seven emotional expression intensities. In response, numerous deep learning models [22, 24, 29, 42] have been proposed for this ERI task. However, the emotional intensity annotations in the Hume-Reaction dataset are globally unique for each audiovisual sample. Such annotations may prove inadequate for analyzing long-context audiovisual data, thereby limiting the applicability of these methods within our particular domain.In this paper, we regard emotional intensity as a continuous attribute evolving over time, which can be implicitly modeled. Therefore, a dynamic emotional intensity modeling module (detailed in Section 3.3) is proposed to extract this information. 
method: 
 Problem Formulation. Let C 1:𝑇 = (c 1 , . . . , c 𝑇 ), c 𝑡 ∈ R 185 be a T -length sequence of control rig coefficients, where each frame c 𝑡 has 185 control rig data. Also, we define A 1:𝑇 = (a 1 , . . . , a 𝑇 ) as a sequence of speech snippets, each of which a 𝑡 ∈ R 𝑑 has d samples to align with the corresponding visual frame c 𝑡 . Moreover, e 𝑠𝑡 𝑦 ∈ R 5 denotes a user-controllable emotional style which is one-hot encoding. Then, our goal is to predict emotion-enhanced coefficients Ĉ1:𝑇 = (ĉ 1 , . . . , ĉ𝑇 ) conditioned on A 1:𝑇 and e 𝑠𝑡 𝑦 .The whole pipeline of our approach for the speech-driven 3D emotional facial animation task is revealed in Figure 2 . To address existing limitations, such as lack or improper handling of emotional intensity modeling, we carefully design two modules: the Dynamic Emotional Intensity Modeling Module (DEI, detailed in Section 3.3) and the Emotion-guided Feature Fusion Decoding Module (detailed in Section 3.4). The former extracts high time-frequency representations of emotional intensity from speech features, while the latter maps audio to facial coefficients using emotional information derived from emotional style input and DEI. Moreover, we design a dynamic positional encoding (DPE) strategy to provide temporalorder information for speech sequences of varying lengths. In summary, DEITalk takes input with a speech sequence A 1:𝑇 and an emotional style label e 𝑠𝑡 𝑦 , and produces emotion-enhanced control rig coefficients Ĉ1:𝑇 . These predicted coefficients can be utilized to animate MetaHuman characters, enhancing their emotional expression. Formally, we define our DEITalk as: EQUATIONwhere 𝜃 indicates the model parameters. In the following sections, we will introduce the details of our method.Audio Feature Extraction. Before feeding the raw speech signal X to our model, it will be converted into a speech sequence A 1:𝑇 via the Librosa library [25] at a sampling frequency of 16kHz. Then, we employ Wav2Vec 2.0 [2] to encode A 1:𝑇 , which represents the state-of-the-art self-supervised pre-trained speech model for ASR [41] task. Wav2Vec 2.0 consists of several temporal convolutional layers and transformer encoder layers, which, respectively, convert the audio input into feature vectors sampled at 50Hz and further encode them into contextualized speech representations S 1:𝑇 = (s 1 , . . . , s 𝑇 ), s 𝑡 ∈ R 768 . Formally, it can be defined as E 𝐴 (A 1:𝑇 ) → S 1:𝑇 , where E 𝐴 is the Wav2Vec2.0 architecture. In addition, a linear interpolation layer is applied to resample the speech feature at 30Hz to match the frequency f c of the control rig coefficients. Moreover, to maximize data utilization, we employ large overlap sliding windows to segment speech sequences during the training phase.Dynamic Positional Encoding. Facial movements for emotional expression exhibit longer temporal dependencies. Utilizing sliding windows for speech segmentation could potentially result in the loss of positional information regarding the original complete speech sequences (depicted in Figure 3 ). Thus, endeavoring to reconstruct precise emotional expressions exclusively from speech segments may introduce ambiguity within the system. Additionally, the sinusoidal position encoding method employed in transformer architectures [36] exhibits limited generalization capabilities [8] when confronted with sequences longer than those encountered during the training phase.To address this issue, we introduce a dynamic positional encoding (DPE) strategy that can effectively handle sequences of different Figure 4 : Mechanism of DPE. We capture three equally sized segments from two speech sequences of different lengths using sliding windows. Segments 2 and 3 are both located at the tail, which means their corresponding facial expressions should be more similar during the silence, while segment 1 is located in the middle, meaning its facial expressions during the silence should be more different from those of segments 1 and 2. Although these three segments share very similar waveform characteristics, DPE effectively provides positional information relative to the original speech sequences(e.g., despite being located at different indices within their respective sequences, segments 1 and 2 exhibit a high degree of similarity in their DPE results).lengths and describe the positional information of each time step relative to the complete sequence. Specifically, we modify the original sinusoidal positional encoding method to enable adaptive position encoding tailored to speech sequences of varying lengths T :DPE (𝑡,2𝑖 ) = sin 20𝜋 • 𝑡 𝑇 • 10000 2𝑖/𝑑 DPE (𝑡,2𝑖+1) = cos 20𝜋 • 𝑡 𝑇 • 10000 2𝑖/𝑑(2)Figure 5 : We learn a speech-expression joint embedding space using contrastive learning. The speech feature sequence is converted to a compressed sequence S 𝑒𝑚𝑜 1:𝑇 , which is then aggregated into an embedding vector s 𝑚𝑎𝑥 via max pooling. Similarly, the captured control rig coefficients are processed by a motion encoder, producing a feature sequence S 𝑒𝑚𝑜 1:𝑇 and the corresponding embedding vector m 𝑚𝑎𝑥 . To ensure that S 𝑒𝑚𝑜 1:𝑇 can capture pertinent emotional information, a classifier is introduced to predict emotional categories. This module is trained using a contrastive loss that maximizes the similarity between the embeddings s 𝑚𝑎𝑥 and m 𝑚𝑎𝑥 of paired speeches and control rig coefficients.where t is the current time step in the sequence, d is the dimension of the model, and i is the dimension index. Instead of assigning a unique position identifier for each time step, the proposed DPE strategy injects position information into each time step relative to the entire sequence (as shown in Figure 4 ). DPE is applied to the speech features S 1:𝑇 to provide temporal order information. For each feature vector s 𝑡 in S 1:𝑇 : ŝ𝑡 = s 𝑡 + DPE(𝑡,𝑇 ).(3)In light of the potential limitations associated with employing onehot vectors to characterize the intensity of facial movements, our goal is to learn an implicit representation of emotional intensity from the extracted speech features and to represent it as a spatiallocal, high-temporal-frequency feature. This feature serves as one of the conditions that guide the decoder in synthesizing facial animation with rich expressions. Speech-Expression Joint Embeddings. The many-to-many mapping between speech content and facial expression challenges in generating semantically correct facial movements. Inspired by the Contrastive-Language-Image-Pretraining (CLIP) model [30] and GestureDiffuCLIPmodel [1] , we integrate contrastive learning with a temporal aggregation mechanism to establish a joint embedding space for speech and facial movements in emotional expression. This space offers emotional saliency cues that steer the system toward learning the correspondence of emotional intensity across these two distinct modalities.Architecture. As shown in Figure 5 , a linear projection layer is applied to map the speech feature S 1:𝑇 into a lower-dimensional feature S 𝑒𝑚𝑜 1:𝑇 = (s 𝑒𝑚𝑜 1 , . . . , s 𝑒𝑚𝑜 𝑇 ), s 𝑒𝑚𝑜 𝑡 ∈ R 128 . Meanwhile, we designed a motion encoder E 𝑀 that comprises a four-layer TCN along with a linear layer to extract the ground truth of the control rig coefficients C 1:𝑇 , which can be defined asE 𝑀 (C 1:𝑇 ) → M 𝑒𝑚𝑜 1:𝑇 = (m 𝑒𝑚𝑜 1 , . . . , m 𝑒𝑚𝑜 𝑇 ), m 𝑒𝑚𝑜 𝑡 ∈ R 128 .To mitigate confusion during the training phase caused by incomplete temporal alignment between these two modalities, we employ maximum pooling to aggregate emotion-relevant information from each feature sequence:EQUATIONThen, s 𝑚𝑎𝑥 , m 𝑚𝑎𝑥 are considered the embeddings of speech and facial movements, respectively. To avoid misinterpreting speechinduced articulation for emotional cues, we introduce a classifier to predict the emotion style ê𝑠𝑡𝑦 from S 𝑒𝑚𝑜 1:𝑇 and compare it with the label of the emotional style of the ground truth e 𝑠𝑡 𝑦 . Contrastive Learning. We apply CLIP-style contrastive learning to finetune the module. Specifically Applications of the Joint Embeddings. The joint embedding space provides an effective method to measure the similarity of emotional saliency between speech and facial movements, thereby enabling the characterization of dynamic emotional intensity. Given S 𝑒𝑚𝑜 1:𝑇 and its embedding vector s 𝑚𝑎𝑥 , we calculate the similarity between s 𝑒𝑚𝑜 𝑡 and s 𝑚𝑎𝑥 , and obtain the corresponding emotional saliency sequence 𝐸 𝑠𝑖𝑚 1:𝑇 = (𝑒 𝑠𝑖𝑚 1 , . . . , 𝑒 𝑠𝑖𝑚 𝑇 ), 𝑒 𝑠𝑖𝑚 𝑡 ∈ R:EQUATIONAs shown in Figure 6 , frames with high similarity scores are more likely to accompany intense emotional expressions. These cues serve as significant emotional saliency indicators and will be leveraged within our system to direct the generator in synthesizing expressive facial animations. Then, the desired dynamic emotional intensity representations E 𝑖𝑛 We propose an emotion-guided feature fusion decoder, where emotion consists of two key components: emotional style embedding (controlled by the user) and dynamic emotional intensity representations (extracted by DEI). By effectively leveraging these emotional cues, the proposed decoder aims to map the speech to emotional 3D facial animation coefficients.Previous studies implement the decoder adhering to the structure of the transformer decoder [10, 28] . However, the transformer decoder requires autoregressive output, meaning that each frame's output is contingent upon the decoding of all speech features. Predicting facial movements for long-context speech would result in an undesirable increase in time complexity, especially in scenarios demanding swift inference. Given the inherent constraints of the transformer decoder, we employ an eight-layer TCN to construct our decoder and meticulously regulate the receptive field of each layer to manage the influence of each frame on its neighborhood.The one-hot emotional style embedding is first mapped to a 16dimensional vector E 𝑠𝑡 𝑦 and then padded to match the length of the speech feature S 1:𝑇 . Consequently, the speech feature S 1:𝑇 is concatenated with E 𝑠𝑡 𝑦 1:𝑇 and E 𝑖𝑛 1:𝑇 to get the fused feature F. Then, F will be fed into our decoder. In addition, we also used a similar feature fusion strategy in each temporal convolutional layer of TCN, the overall formula for feature fusion can be described as:EQUATIONFinally, a fully connected layer is employed to map the outputs of the emotion-guided feature fusion decoder to 185-dimensional control rig coefficients.We design a novel loss function comprising four distinct components: contrast loss, classification loss, position loss, and velocity loss. The overall function is given by:EQUATIONwhere we set the the trade-off parameters with the values as follows 𝜆 1 = 0.1, 𝜆 2 = 0.1, 𝜆 3 = 1.0, and 𝜆 4 = 4.0 by default in our experiments. In the following sections, we will provide a detailed explanation of each component. Contrast Loss. The contrast loss function is designed to optimize the alignment between speech and facial movements while learning the joint embedding space. Given a batch of samples with a batch size of N, let s 𝑚𝑎𝑥 𝑖 be the vector representation of speech i and m 𝑚𝑎𝑥 𝑗 be the vector representation of facial movements j. Consequently, we obtain the similarity score 𝑠 𝑖 𝑗 by computing the cosine similarity between these two vectors:EQUATIONSubsequently, the contrast loss function can be expressed as:EQUATION)where 𝜏 is the temperature parameter that adjusts the scaling of the similarity scores. Classification Loss. To ensure that the learned joint embedding space can sufficiently describe emotional intensity, we introduce a classification loss to constrain the learning process of our system. The classification loss is defined as:EQUATIONwhere M denotes the number of classifications, 𝑦 𝑠𝑡 𝑦 𝑖𝑐 is the observation function that determines whether the sample i carries the emotion label c, and 𝑝 𝑠𝑡 𝑦 𝑖𝑐 denotes the predicted probability that sample i belongs to class c. Position Loss. The position loss computes the distance between the predicted outputs and the ground truth of control rig sequences, which is described as:EQUATIONwhere T represents the length of each sample in the training set (i.e., the window size of the sliding window), c 𝑖 𝑡 and ĉ𝑖 𝑡 denote the ground truth and predicted values, respectively, for the t-th frame of sample i. The position loss serves to encourage the model to match the ground-truth performance. Velocity Loss. The velocity loss computes the distance between the differences of consecutive frames between predicted outputs and the ground truth of control rig sequences, inducing the generation with temporal stability. The formula is as follows:EQUATION) 
experiments 4.1 experimental settings: 
 Datasets. We train and validate DEITalk on our dataset, ETH-3D. ETH-3D is an audio-visual dataset consisting of five different styles of emotions (i.e., Neutral, Angry, Happy, Sad, and Surprised), and each emotional style corresponds to approximately 100 samples, with each sample lasting between 8 and 15 seconds. Specifically, the emotional intensity of each sample dynamically changes in accordance with the semantic salience of the corresponding transcribed text. We split the dataset with 70% of the samples included in the training set (ETH-3D-Train), 20% in the validation set (ETH-30-Val), and 10% in the testing set (ETH-3D-Test). For a broader evaluation, a small-scale, multilingual test dataset was created for generalization testing, where the speakers of the audio sequences are disjoint from those in ETH-3D.Network Architectures. In the dynamic emotional intensity modeling module, the TCN within the motion encoder consists of four 1D convolutional layers, each employing the ReLU activation function. The configuration includes {384, 384, 768, 768} filters for the respective layers, all with a filter size of 3, a stride of 1, and dilation and padding values set to {1, 2, 4, 8}, respectively. The classifier comprises two fully connected layers, yielding output channels of {64, 5}. In the emotion-guided feature fusion decoding module, the TCN within the decoder incorporates eight 1D convolutional layers with ReLU activation. Each layer is equipped with 512 filters, all set with a filter size of 3, a stride of 1, and dilation and padding values of {1, 2,Implementation Details. Our system generates facial parameters represented by a 185-dimensional control rig at a rate of 30 frames per second. For the overall training procedure, we employ Adam as optimizer, with a batch size of 32 and a learning rate set to 1e-4. Speech sequences are segmented using a sliding-window approach, with a stride of 5 frames and a window size of 128 frames. The TCN parameters in Wav2Vec2.0 are fixed, whereas the other components of the model are learnable. The value of 𝜏 in contrast loss is set to 0.07. DEITalk is implemented using PyTorch and takes around 10 hours (300 epochs) to train the whole network on ETH-3D with a single NVIDIA RTX3090 GPU.Baseline Methods. We compare DEITalk with two state-of-theart open-source methods, FaceFormer [10] and EmoTalk [28] , on ETH-3D-Test and wild audio, respectively, and render the predicted animations on the same MetaHuman avatar. Notably, FaceFormer requires a neutral 3D face mesh template during both the training and inference stages, with its output being the mesh vertices of the face. Since these elements are unavailable in our dataset, we made necessary adjustments to its network structure. This ensures that it can be trained on our dataset and facilitates a more equitable comparison with our proposed method.Lip movement and emotional expression are two crucial aspects in assessing the quality of facial animation. For instance, the lip vertex error (LVE) is used in MeshTalk [31] and FaceFormer [10] to measure the quality of lip synchronization, while the emotional vertex error (EVE) is used in EmoTalk [28] to reflect the quality of Lip Parameter Error. LPE measures the maximum error ℓ 2 between predicted values and ground truth for 86 specific control rig coefficients, which are responsible for controlling the movements of the lip and its surrounding areas.Emotional Parameter Error. Similar to LVE, EPE measures the maximum error ℓ 2 between predicted values and ground truth for 18 specific control rig coefficients governing the movements of the eyes, eyebrows, forehead, and surrounding areas. We report the LPE and EPE evaluation results for FaceFormer, EmoTalk, and DEITalk in Table 1 . Compared to the other two methods, DEITalk achieved a lower lip error and emotion expression error, indicating that it can produce more accurate lip movements and more realistic emotional expressions.Due to the many-to-many mapping nature of the audio-to-face task, solely relying on metrics for qualitative evaluation is insufficient. Human perceptual assessment is also needed. Consequently, we proceed to qualitatively evaluate our model from two perspectives.Lip Synchronization. We compared our model with FaceFormer and EmoTalk by feeding them identical audio inputs and generating corresponding facial animations. To avoid introducing emotionrelated interference in the lip synchronization evaluation, all audio inputs belong to the neutral emotion style. The comparison results (shown in our supplementary video) show that DEITalk exhibits Emotional Expression. We feed FaceFormer, EmoTalk, and DEITalk identical audio inputs containing various emotional styles and compare the emotional variation in the facial animations generated by them (detailed in our supplementary video). Figure 7 shows a comparison result of a specific emotional style, the animations generated by FaceFormer exhibit limited emotional differentiation due to the lack of optimization for emotional expression. EmoTalk and DEITalk exhibit comparable performance on the ETH-3D-Test, with DEITalk slightly ahead in the realism of emotional variations. In zero-shot settings, EmoTalk exhibits severely limited emotional expression capabilities (shown in Figure 8 ), often resulting in emotionless or even erroneous emotional output. On the contrary, DEITalk maintains a relatively superior and robust emotional expression performance. This is primarily attributed to the fact that DEITalk treats the emotional style as a user-controllable condition and dynamically analyzes the emotional intensity of the input audio. We strongly recommend that the reader watch our supplementary video, which offers more detailed comparisons to assess the motion quality of our approach.To comprehensively evaluate the performance of our proposed model, we conducted an extensive user study with a total of 136 participants. The evaluation process was divided into three subtasks: full-face comparison, lip-sync comparison (by covering the area above the nose), and emotion expression comparison (by covering the area below the nose). We selected 20 test cases (four for each Table 2 reports a detailed description of the user selection results. Our method achieved the highest positive evaluation results in all three sub-tasks, notably excelling in the emotion expression comparison task, where participants considered our approach significantly superior to others.To investigate the contributions of the key designs of our proposed method, including the dynamic positional encoding strategy, dynamic emotional intensity modeling module, and loss function, we conducted an ablation experiment using the evaluation metrics introduced in Section 4.2.As shown in Table 3 , the removal of the dynamic emotional intensity modeling module results in a significant increase in EPE, demonstrating its crucial role in the representation of emotional intensity. Meanwhile, abandoning the dynamic positional encoding strategy results in a slight increase in LPE and EPE, which indicates the effectiveness of DPE in enhancing emotional expression.From the perspective of the loss function, removing velocity loss leads to a clear decrease in performance, inducing noticeable shaking in the facial animation output. Removing the classification loss significantly increased the EPE, indicating that the dynamic emotional intensity modeling module could no longer effectively express emotional information. 
conclusion: 
 In this paper, we propose a novel speech-driven 3D facial animation framework (DEITalk) that achieves state-of-the-art performance in the generation of emotional 3D talking heads with aligned lip movements and realistic expressions. Unlike previous methods that regarded emotional intensity as a discrete label (e.g., one-hot encoding), we specifically design a dynamic emotional intensity modeling mechanism that allows the model to automatically analyze high-frequency emotional intensity information from speech features and provide reliable emotional saliency guidance for facial animation generation. To further enhance the emotional expression ability of DEITalk, we introduce a dynamic positional encoding strategy, an emotion-guided feature fusion decoder, and a four-way loss function, respectively. Moreover, DEITalk is a lightweight and artist-friendly model whose outputs are identity-free and can be directly utilized to manipulate facial animations of any MetaHuman avatar, making it seamlessly integrated into industrial pipelines.  
 
 Appendix:  


Decision: This paper introduces a novelty method for generating speech-driven 3D facial animations, focusing on dynamic emotional intensity modeling. After the authors' rebuttal, Reviewers found an agreement on considering the paper contribution valuable for publication.  So I tend towards accepting this paper as a poster.

Reviews: review: paper_summary: paper_summary: This paper introduces a novelty method for generating speech-driven 3D facial animations, focusing on dynamic emotional intensity modeling. The authors create a new dataset consisting of five emotional styles with a set of coefficients based on the MetaHuman character model. The DEITalk is designed to model emotional saliency variations in long-term audio contexts with a dynamic emotional intensity (DEI) modeling module and a dynamic positional encoding (DPE) strategy.
strength_weakness: strength_weakness: 1. This paper presents a new dataset and an end-to-end deep neural network, DEITalk, to synthesize 3D talking head animations with precise lip movements and rich stylistic expressions.
2, The paper is well-structured, starting with a clear identification of the limitations in existing speech-driven 3D facial animation methods, particularly their lack of emotional expression and the over-smoothing of emotional intensity. The authors effectively argue that incorporating dynamic emotional intensity modeling can significantly enhance the realism of 3D facial animations.
3. The experimental demos presented in the paper clearly demonstrate the effectiveness of emotion control. The DEITalk method produces animations with dynamic emotional expressions that vary in intensity over time, which is visually evident in the results. These animations are compared with state-of-the-art methods, and DEITalk consistently shows superior performance.
limitations: limitations: 1. It is unclear whether the ETH-3D dataset introduced in the paper will be made publicly available. Without open access, the dataset cannot be considered a significant contribution as it does not offer novel data processing procedures or insights into how different dataset characteristics affect model performance (e.g., scaling laws).
2, The pipeline for data collection using MetaHuman and LiveLink Face, which is used for animation generation, has already been validated in prior work such as “Transformer-S2A: Robust and Efficient Speech-to-Animation.” Therefore, this aspect of the paper does not present a novel contribution to the field.
3. The Dynamic Positional Encoding (DPE) introduced in the paper lacks references to FaceFormer’s Periodic Positional Encoding (PPE). The paper does not clarify the connection or distinctions between DPE and PPE, nor does it provide a justified motivation for introducing DPE over existing methods.
4. Section 4.5 does not provide an evaluation demonstrating that DPE outperforms the sinusoidal position encoding used in Transformers or FaceFormer’s PPE. Moreover, the demo video does not include an ablation study focusing on positional encoding.
5. The demo videos show noticeable emotional changes in the mouth corners and eye areas, but the cheek region appears to remain mostly static. This does not align with how human faces typically display emotions, suggesting a limitation in the model’s ability to generate comprehensive facial expressions.
6. Although the paper claims to learn continuous emotional intensity from speech, the experimental section does not detail how different intensities affect the animation. Emotion is still treated as a global variable, which is inconsistent with the paper’s motivation. To address this, the paper should include tests showing the impact of varying intensities for the same sentence on animation, such as using the MEAD dataset for evaluation.
paper_summary: paper_summary: This paper first collected a 3D talking head dataset comprising five emotional styles with a set of coefficients based on the MetaHuman character model. Next, the paper proposed an end-to-end deep neural network, DEITalk, which conditions on speech and emotional style labels to generate realistic facial animation with dynamic expressions. Extensive experimental results demonstrate that their method outperforms existing state-of-the-art methods.
strength_weakness: strength_weakness: - The paper presents an emotional 3D talking head dataset, which contributes to the development of the field of Speech-Driven 3D Facial Animation.
- The paper recognizes the shortcomings of the global emotion modeling using one-hot encoding and proposes a dynamic emotional intensity module to achieve fine-grained emotional expression.
limitations: limitations: - The dataset proposed in this paper is relatively small, which limits its practical applications to some extent.
- All experiments in this paper were conducted on the ETH-3D dataset, without comparisons on other public datasets (e.g., EmoTalk[1]). Therefore, the persuasiveness of the experimental results is limited. Moreover, sometimes better results may be attributed to the use of a superior dataset, especially compared to using more facial coefficients than the 52-dimensional blendshape coefficients.
- Recently, methods based on diffusion models, such as FaceDiffuser[2], have shown significant advantages in this field. The paper should include comparative experimental results to validate the advanced nature of the proposed model.
- In the speech-expression joint embedding space, the captured control rig (185,T) contains both emotional and semantic information. Therefore, is it reasonable to achieve fine-grained emotional alignment through this alignment method?
- From the visual effects of the demo video, one can observe the overall emotional changes, but it is difficult to discern the frame-level fine-grained emotional changes, which raises doubts about the effectiveness of the model.


[1] Peng Z, Wu H, Song Z, et al. Emotalk: Speech-driven emotional disentanglement for 3d face animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 20687-20697.

[2] Stan S, Haque K I, Yumak Z. Facediffuser: Speech-driven 3d facial animation synthesis using diffusion. In Proceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games. 2023: 1-11.
paper_summary: paper_summary: This paper presents DEITalk, a novel approach for speech-driven 3D facial animation. It emphasizes the importance of modeling emotional intensity and introduces the Dynamic Emotional Intensity (DEI) module to learn this from speech. The model is guided by both the DEI module and an emotional style label to generate emotive 3D facial animations. The authors collect a new 3D emotional talking head dataset, ETH-3D, to train and evaluate their model.
strength_weakness: strength_weakness: 1. This paper proposes a approach, DEITalk, for speechdriven 3D facial animation, which enables the synthesis of expressions with a specified emotional style while adapting to the dynamic emotional intensity in the speech.
2. They introduce a dynamic emotional intensity modeling module that employs a contrastive learning strategy to learn the correlation between speech and facial movements in emotional saliency.
3. They capture a 3D emotional talking head dataset, ETH- 3D, comprising five distinct emotional styles.
limitations: limitations: 1. The DEITalk shown in the supplementary video seems to work well for Chinese, but poor lip synchronization for other languages.
2. Will ETH-3D be made public? If not, it will contribute less to the community. What is the total duration of ETH-3D? The description in 4.1 seems to be a little short.
3. In Fig 3, the sliding windows currently used are generally not as large as the picture. For example, it is 0.2 seconds in wav2lip, and emotion does not change much.
4. On line 382, wav2vec 2.0 is not yet the state-of-the-art, you can try using hubert.
5. In Figure 6, the emotional-saliency curves appear to correlate with volume, while no significant change in emotional intensity is seen in the supplementary video.
6. There are few methods of comparison, and it is recommended to include evaluations for CodeTalker and SelfTalk.