EcoFace: Audio-Visual Emotional Co-Disentanglement Speech-Driven 3D Talking Face Generation
Authors: Jiajian Xie
Shengyu Zhang
Mengze Li
chengfei lv
Zhou Zhao
Fei Wu
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Talking face generation
Audio-visual emotional co-disentanglement
Speaker-specific
Primary area: generative models

Abstract: Speech-driven 3D facial animation has attracted significant attention due to its wide range of applications in animation production and virtual reality. Recent research has explored speech-emotion disentanglement to enhance facial expressions rather than manually assigning emotions. However, this approach face issues such as feature confusion, emotions weakening and mean-face. To address these issues, we present EcoFace, a framework that (1) proposes a novel collaboration objective to provide a explicit signal for emotion representation learning from the speaker's expressive movements and produced sounds, constructing an audio-visual joint and coordinated emotion space that is independent of speech content. (2) constructs a universal facial motion distribution space determined by speech features and implement speaker-specific generation. Extensive experiments show that our method achieves more generalized and emotionally realistic talking face generation compared to previous methods.

Published as a conference paper at ICLR 2025
AUDIO-VISURA EMOTIONAL Co- DISENTANGLEMENT SPEECH-DRIVEN 3D TALKING FACE GENERATION
ECOFACE:
Jiajian Xie Shengyu ; Mengze Li', Chengfei Lv²; Zhou Zhao'; Fei Wu²
!Zhejiang University AAlibab
{xiejiajianl sy-zhang, mengzeli, zhaozhou, f edw.cc.... chengfeiiv@@lliaaa-iiic. com
ABSTRACT
Speech-driven 3D facial animation has attracted significant attention due to its wide range of applications in animation production and virtual reality. Recent research has explored speech-cmotie disentanglement to enhance facial expres- sions rather than manually assigning emotions. However, this approach face issues such as feature confusion, emotions weakening and mean-face. To address these issues, present EcoFace, a framework that (I) proposes a novel collaboration objective to provide an explicit signal for emotion representation learning from the speaker's 's expressive movements and produced sounds, constructing an audio- visual joint and coordinated emotion space that is independent of speech content. (2) constructs universal facial motion distribution space determined by speech features and implement speaker-aware generation. Extensive experiments show that our method achieves more generalized and emotionally realistic talking face generation compared to previous methods
INTRODUCTION
Speech-driver 3D facial animation is an important and challenging problem that aims to generate lip synchronization and facial expressions based on arbitrary speech signals, which has several ap- plications such as games, movies and virtual reality. Over the past few years, research has advanced from mapping speech to lip movements Cudeiro al. (2019) to include details of facial movements Fan et al. (2022): Xing al. (2023). However, emotional information is often subtle compared to speech content information, leading the model to focus excessively on changes in lip movements but weakly emotional details. Recently, several studies have attempted to explicitly construct emotional features of speech, where a separate representation speech content and emotion allows the model to focus on learning specific types of features and reduce mutual interference between them.
Manually assigning emotions is emotion construction method used in early research, which constructs an independent expression space by encoding specified emotional features. MeshTalk (Richard et al. 2021) uses the video of the target emotion as the specified feature and EMOTE DDančček et al. 2023) uses emotion labels and intensity labels as the specified feature. These meth- ods allow for construction of an emotional space that is independent of the speech content, making the generation of emotional talking faces directional. However, this is difficult to apply in automatic generation scenarios due to the need for manual input. More recently, Emo (Peng et al. 2023b) employs disentanglement of speech content from emotion, allowing the model to extract content features and emotion features from speech without manual labeling. Specifically, it exchanges emotion embeddings of two speech segments with different contents and emotions during training, separating the emotion space from the content space. However, the lack of clear supervisory signals of content and emotion is prone the feature confusion problem, where the same driving result may occur for different emotional speech with the same content. Furthermore,
"Corresponding authors
Video samples and source code available at htttt:://ttooteeee .it.tg..t......... o

---

Published as a conference paper at ICLR 2025
due to the varying intensities of emotions and the different ways in which emotions are expressed in individual faces, the method suffers from emotion weakening and mean-face problems. To sum- marize, the research of speech emotion-contes disentanglement is under-explored and challenged with feature confusion due to the interaction of speech content and emotional information, emotion weakening due to differences in emotion intensity, and mean-face due differences in the speaker's style of emotional expression.
To address the above problems, we propose EcoFace, a framework that distills joint and coordinated audio-visual emotional features and enables personalized face generation. For the feature confusion and emotion weakening problem, we aim to provide explicit signals for the representation of emo- tional motions and emotion differentiation. Specifically, we design an audio-visual loss to supervise the discrepancy between emotional information captured by the emotional disentanglement encoder from facial motion and the audio stream, promoting synergy between audio-visual emotion features while mitigating interference from speech content. Additionally, we construct a contrastive-tripler loss to provide discriminatory signal for different types and intensities of emotion features, facil- itating the construction of a well-differentiated emotion space. As for the mean-face problem, we construct emotional motion generator (EMG), module that first maps speech features with different contents and emotions into a low-dimensional continuous distribution space representing universal facial movement patterns, and then the speaker-aware decoder generates stylized facial action results based on stylistic encoding. Using a low-dimensional distribution space as an input to the decoder not only maintains the fixed regularity of facial action generation, but also amplifies the stylistic information, enabling the capture and generation of different styles of action details. Moreover, as well-supervised signal facilitates feature disentanglement (Ding al. 2020), we pre-trained a lip- sync discrimination expert and constructed a visual emotion loss using emotional disentanglement encoder.
In summary, the main contributions of our work are as follows: (1) We propose an audio-visun visuall emotional co-disentanglement method that uses explicit supervisory signals to construct an audio- visual joint and coordinated emotion space independent of speech content. (2) We implemented mapping of emotional speech to a low-dimensional distribution space of universal facial movement and designed an emotional motion generator for stylized talking face generation. (3) Experiments show that our EcoFace outperforms other state-of-the-at baselines from the perspective of expres- sion quality and lip synchronization metrics.
2 RELATED WORK
Speech-Driven 3D Facial Animation Speech-driven 3D facial animation a task to generate realistic facial animations based on speech. Early research in this domain primarily focused on mapping-base approaches. including capturing the relationship between visual and face action units (FAUs) (Ekman & Friesen 1978) and building thoneme-visene mapping relationships (Taylor et al., 2012 Xu et al. 2013]] Edwards et al.) 2016). However, all these methods require lot of time and manpower to construct matching relationships and are difficult to generalize new persons.
In contrast, deep-learning-
approaches have emerged as a more scalable solution, allowing models to automatically learn motion patterns from data to generate 3D facial animations. VOCA (Cudeiro et al., (019) proposes a temporal convolutional neural network that takes speech and a silent 3D mesh template as input generate realistic animations. FaceFormer (Fan al.) 2022)
focuses on the long temporal sequences and successfully uses a transformer decoder ( (Vaswani
2017
obtain content information to generate sequential mesh sequences. CodeTalker (Xing al. introduces VQ-VAE (Van Den Oord et al. (017) to learn discrete motion priors to capture the speaker's movement characteristics and solve the overly smooth facial motions problem. SelfTalk (Peng et al. 2023a) further enhances generation quality through the introduction of consistency loss. All these approaches, however, neglect the interplay between speech-driven emotions and corresponding facial expressions.
2023)
Emotional 3D Face Animation Recent studies have underscored the critical role of emotion in creating realistic and expressive 3D facial animations by integrating emotional information. Specifi- cally, MeshTalk (Richard et al. (02T) extracts emotional action potential space from the target video to synthesize emotionally detailed actions such eyebrow movements. EMOTE (Danêček al.
2

---

Published as a conference paper at ICLR 2025
employs one-hot labels to control emotion, producing emotional facial motions through an emotion-content disentanglement loss. However, these methods require explicit control, lacking a direct connection to the emotion conveyed in the actual speech. Addressing this issue, Emo'Talk (Peng et al., (023b) disentangles speech emotion from content to generate emotional blend shapes based solely audio input. However, Emo'Talk's disentanglement process lacks a clear supervisory signal, leading to the problem of feature confusion. And it still needs to specify the intensity of the emotion explicitly to avoid emotion weakening. Unitalker (Fan et al., 2024), through multi-head architecture that effectively leverages multiple datasets with different annotations to improve the model's s generalization over emotion and synchronization. However, its mapping of speech emotion and content to the same coding space leads to the problem of weak and erroneous emotions.
2023)
In this paper, we focus on disentangling the emotion of speech from the content and reducing the confusion between speech emotion and content through the synergy of audio-visual emotional fea- tures. We also construct a low-dimensional facial action distribution space and implement style- guided 3D speaking face generation.
3
METHOD
Our method aims to achieve realistic and emotional 3D talking face generation driven solely by speech. Our approach employs the parametric FLAME model (Li et al., 2017) as the face represen- tation and uses the pre-trained speech encoder Hubert (Hsu et al., 2021) to extract speech content features (Sec. B.I. As shown in Fig. the overall framework consists of two main modules: (1) Speech and video emotional disentangiemest encoder (EDE), which use audio-visual co-contrastive learning to achieve the extraction of emotional features in both modalities (Sec. (32)). (2) Emo-
tional motion generator (EMG), which converts audio content features and emotion features into speaker-awar aware FLAME parameter sequences for facial imitation control (Sec. 3.3).
3.1 FORMULATION
As shown in Fig. a the input of EcoFace consists of the speech sequence A (a), (d)......)). a:), where t is the time of audio, each R and D is the sample rate, and speaking style vector E R", where n is the number of speaking styles. Our model then disentangles and extracts the emotion-related latent features E1.2T (e1, and content-related latent features C1.27
(C1, (CI...., ..... C2T) from the speech sequence AIa, where T is the number of frames, RI024 and E R 1024 and combines them with the style vector s to generates FLAME-based 3D facial animation represented by sequence of FLAME parameters Fir = (f1: fI), where f R25 is FLAME expression parameters VI, E R5 concatenated with jaw parameters 0:a00 E RR
f: V., 0/0w
3D Face Representation Compared predicting mesh vertices, we use the low-dimensional FLAME (Li et al.] 2017) parameter as an intermediate bridge for mesh driving, as the low- dimensional data representation not only has a faster computational speed but also simplifies the learning process. FLAME defines mesh with N = 5, 023 vertices and K = 4 joints, whose ge- ometry can be represented using parameters {B, zb, 0}. where B E 1000 is the shape parameter, ab E 1g50 the expression parameter, and E gak+3 is the head pose parameter. Given set of FLAME parameters, the 3D face mesh can be obtained with:
M(3 0, +b) = W (Tr(B, 0, ((3)) W)
where Tp outputs vertices by combining blend shapes, the standard skinning function W( T,J. JJ 0, W) rotates the vertices of T around joints J, and W performs linear smoothing. For facial animation, we predict the expression parameters v/b and the jaw components within the pose parameters 0.
Speech Content Representation Recently, self-supervised pre-trained speech models such as Wav2Vec2.0 (Baevski et al. 2020) and Hubert (Hsu et al., (021) have achieved good results in content representation. Based on this, we utilize Hubert as our chosen speech content encoder for facial animation generation, Haque & Yumak (2033) has proven it superior Wav2Vec2.0.
3

---

Published as a conference paper at ICLR 2025
Emotion: disentangle
Ebcoder
Emotion Fenture
Speaker-
pose
aware
Driving Audio
Person ID
Emotional Motion
shape
Hubert
Content feature
G enerator
FLAME
Gen Mesh
Pipline BcoFace
A Video festume
O Audio Jesture
Happy, Level
Happy, Level
GT Mosh
Encoder
EDE
EDE
EMG
campar oggn
ED Eaudiol
Woopand
f4, o
Lemm
Hubert
Peshure Space
Emotion Specee
EDEytuual
Leo-contrastiin
Leriptet
Decoder
Person
Lyyne
Mouth landmark
Angry, Level
Happy, Level
Gen Mesh
(b) The structure of audio-visual emotional co disentanglement
(c) The structure of emotional motion generator
Figure 1: Overview of the proposed EcoFace. Subfigure (a) depicts the disentanglement of emo- tional and content features from audio, generating speaker-aware FLAME and rendering expressive meshes. Subfigure (b) illustrates how features corresponding to different emotions move away from each other the feature space (left), while features representing different emotion intensities re- main distinct within the same emotional space, and specch-vides pair features encouraged to be as similar as possible. Subfigure (c) shows the structure of the emotional motion generator. Dashed arrows indicate processes performed only during training, and only the dashed rectangle part is used during inference.
3.2 AUDIO-VISUAL EMOTIONAL CO-DISENTANGLEMENT
Compared with content information, speech-emotional information often represented by subtle audio features, such as the level of intonation, speed of speech and volume. Therefore, encoding all information into the same specialized space would introduce significant uncertainty into the network, and the extraction of emotional information is disrupted by the content information. We employ the audio-visual loss described below to facilitate synergistic representations of emotional motions in speech and video, while contrastive-triplet loss is applied to provide clear separation signals for establishing an emotional space independent of content.
Emotional Disentanglement Encoder To match the speech content feature dimensions, we adopt Hubert's 's network structure our speech emotional disentanglement encoder. For the visual emo- tional disentanglement encoder, since emotion a gradual and continuous process, even in cases where there is an emotion mismatch (such as at the beginning of speech), it should still be encoded into the same emotion feature space to reduce emotional inconsistency in the generated results. Therefore, similar to 'Shi al. (2022), we use adapted ResNet-18 extract visual emotion fea- tures at the time series level. Denoting visual feature extractor as Evisuat, given a series of rendered 3D face images PLT,, this process be defined as:
Fmisual (P.T) + (VI...., Ver), 1021
Disentangled feature space construction The overview the audio-visual emotion co- disentanglement process illustrated the left side of Fig. [b) through the of audio-visuall co- contrastive loss. Firstly, to spatially distance the features different emotions as much as possible, inspired by Khosla et al. (2020), constructed emotion-supervised contrastive loss using ground
4

---

Published as a conference paper at ICLR 2025
truth emotion labels. Specifically, for set of N randomly sampled senal--motoo--ftturreee
or
pairs, {æt, 3k k==1.... the corresponding batch used for training con- sists of 2N pairs, {ãt, Dr 1.....2. where 22 is the same as 24 (k L...N) and f22 ys- Let i E / = {1 {1...2N) } be the index of an arbitrary augmented sample, and let j (i) be the index of other augmented sample originating from the same source sample. The loss takes the following form:
Ccontrastive
Σ EET
log
exp (24 2p/T)
1 27
2T
(4)
P()
exp (zi =a/T)
(EA(1)
where is the average of o over time, since emotional features at different times under the same data should belong to the same emotion space. The symbol denotes the inner dot product, r e R+ is scalar temperature parameter, A (1) = T (i) and P (i) {p E A (1) 5p up=00 ū.} is the set of indices of all positives in the batch distinct from and |P (i) its cardinality. With the supervision of this loss function, features from the same category are closer together than those from different categories.
Additionally, found that speech with different emotions corresponds different visual repre- sentations in the periods where speech is not present, which can introduce noise into the emotional disentanglement of speech. Furthermore, mentioned earlier, the emotions conveyed in speech subtle and can easily be influenced by the content, leading to feature confusion. contrast, visual emotional features are quite pronounced (e.g., the downward pressure of the corners of the mouth, eye-widening, etc.). Inspired by Wang et al. (2023); Zhang et al. (2020), we construct audio-visual loss follows, with z" the audio emotion feature and x is the visual emotion feature. This loss aims to make the emotional features of speech and visual in the same sample as similar as possi- ble, effectively sharing visual information with the speech and reducing the interference speech content in the disentanglement process:
2T Σ
Laudio Laudio-visual visual
2112.
(5)
2T
1=1
In summary, the audio-visual co-contrastive loss is as follows:
L Cco-contrastive
= Lcontrastive + Laudio-visucl
(6)
Single emotion space construction As shown on the right side of Fig. [b). a distance threshold is necessary to prevent the issue where Lcontrastive can overly compress the distance between similar features, resulting features with the same emotion but different intensities being too close each other. Similar to Schroff et al. (2015), we construct the emotional triplet loss that maintains a certain level of differentiation between features of different emotional levels within the same emotion type by employing the distance threshold, while also ensuring the clustering effect of features of the same emotional level. The loss is defined follows:
Ciriplet Σ11 211 1 22'11 1 +
Given a batch with N samples, z1 is the anchor feature, z is the positive feature with the same emotion level label as the anchor, =ạ" is the negative feature with the different emotion level label as the anchor and a is a margin that is enforced between positive and negative pairs.
3.3 EXPRESSIVE MOTION GENERATOR
To better capture the low-dimensional distribution space of facial motions corresponding to emo- tional speech and to achieve expressive and stylized 3D head motion generation, we introduce a variational autoencoder (VAE) [Kingmal 2013) to perform a generative and expressive audio-to- FLAME- parammtt transformation, namely the expressive motion generator, as shown in Fig. Tc).
Facial motions distribution space construction We first map the ground truth FLAME sequence to a low-dimensional distribution space. To better extract features from the input sequences and con- struct long-term temporal relationships in the output samples, we design the encoder and decoder as
5

---

Published as a conference paper at ICLR 2025
multilayer dilated convolutional network, where the dilation factor of the convolutional layers is gradually increased, allowing the sensory field to grow exponentially with depth. The encoder com- presses the sequence of parameters of frames fIrT into T/q latent distributions (n1:7//, /tll///,1////// g1:T/4) using the speech content features 2f:27 and speech emotion features fi:27 conditions, where the stride size the first one-dimensional convolution in the encoder, smoothing the FLAME parameters to reduce instability between frames:
Enc(fi: Ri.2T, 21.27) (p
(8)
Sampling </.r// from No,, 1), we can get the final facial motion latent sequence:
21.7/g 21/T/9 x O1TT/ + P1.T/e-
(9)
Since the Gaussian prior for vanilla VAE (Kingma| 2013) is sampled independently for each time- indexed data point, the lack of correlation can introduce instability into the sequence generation task. For this reason, we follow Ye et al. (2023) and use flow-based prior. For the encoder output latent sequence zq. the prior module, which is composed of ID-convolution coupling layer, mul- tilayer dilated convolutional network and a channel-wise flip operation, outputs the time-associated latent sequence Zp conditioned the speech content features and the speech emotion features. By calculating the KI. divergence of by and Zp, we ensure that the distributions of both are similar, allowing us achieve a mapping through the speech features to the distribution of facial motions. This approach enables the production of realistic and stable results during the inference phase using the input speech features.
Speaker-aware motion decoder We introduce identity embedding to guide the decoder in gen- crating actions that capture the speaker's speaking style, such as variations how much different speakers open their mouths in response different emotions. The identity embedding takes per- sonal ID as input and generates a personalized feature s E R 1024 consistent with the audio feature dimensions. Since for diw-isenniionll distribution space, the decoder can easily sense the nuances of the input features, we simply add the personalized features to the speech features and use them conditions, combining them with the motion latent sequences input to the decoder generate the FLAME sequences. This process can be described as follows:
Dec(21:T/4- G. <1.27: 8) - fur
(10)
Training Process We use the Monte-Carlo ELBO loss (Ren et al. (222)) to train the EMG model. Besides, we independently train sync-expert D Dyyne that measures the possibility that the input audio content features and FLAME mouth region landmarks are in-sync. For emotion accuracy, we use the emotional disentanglement encoder trained in Sec. 3.2 extract the emotion features of the rendered ground truth images sequence and predicted images sequence and compute their similarity. To summarize, the training loss of EMG is as follows:
LEMGl0,0, w,, = (e (o c,)), + (as c, ee (zlc,))
(11)
- -Fo (log D (s) Ee- (elo) (loge),
where o, 0, denote the model parameters of the encoder,decodee prior and the visual emotion encoder, respectively. c, e, denotes the condition features of EMG. D is the discriminator of sync- net. The ground truth and predicted FLAME parameters are represented by f and f. respectively. The sequence of images rendered by f denoted by
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Data Preparation. To train our EDE, we use an emotional talking face video dataset, RAVDESS
(Livingstone & Russo 2018), which contains 1440 video clips of different actors speaking with 8 emotion categories. Due to RAVDESS's s limited utterances, five hours of videos from the HDTF /Zhan et al. 2021), high-fidelity talking face video dataset includes over 300 subjects and 10k

---

Published as a conference paper at ICLR 2025
Table I: Quantitative evaluation results. Best performance in bold, and the second best underlined. For better visualization, we scale up the LVE of 10 mmm.
RAVDESS
HDTP
VOCASET
MEAD
Methods
VE-FID
LVB
LSE-D 1
LSE-CT
LVE
LSE-D 1 LSE-C1
LVE
LSE-D L LSE-C
VE-FID
LVE L LSB-D J LSE-C1
FazeFomeer
86.75
5.66
9.559
0.926
3.89
11.394
0.727
4.27
10.999
0.662
73.02
8.93
9.726
0.641
CodeTalker
82.71
11.755
10.810
0.667
73.77
8.25
10.07%
Emote
34.01
10.452
0.884
4.07
11.407
0.736
11.174
19.36
4.91
10.049
EmuTalk
51.98
3.18
10.058
0.908
3.56
11.117
0.802
3.89
10.874
0.693
58.74
7.87
9.875
0.562
UniTalker
59.39
5.42
9.915
0.928
4.11
10.563
0.815
3.91
10.812
0.713
54,11
7488
9.584
0.689
GT
10.136
11.535
0.824
10.964
0.813
10.037
Ours
21.57
2.19
9.616
1,010
2.61
10.253
0.823
3.86
10.757
0.743
32.44
5.21
9.113
0.709
different sentences, were used along with the RAVDESS dataset to train our EMG, A random se- lection of 80% of these datasets was used for training, 10% for validation, and 10% for testing. In addition, to fairly compare the models' lip-syne accuracy as well as emotion generalization, the VOCASET (Cudeiro et al., 2019) and MEAD (Wang al., 2020) datasets will be used for the eval- uation. We first processed the video in the original datasets by converting the frame rate to 25 fps and the speech sample rate to 16kHz. Then, EMOCA (Daněček et al. 2022) was used to obtain the FLAME parameters sequence.
Compared Baselines. We compare model with five methods: FaceFormer (Fan et al. 2022,
Code'Talker (Xing et al., 2023), Emote (Danéček et al., 2023). EmoTTalk (Peng et al.) (023b) and UniTalker (Fan et al. (024). For a fair comparison, RAVDESS and HDTF datasets, since these models were trained on the unemotional dataset. As both UniTalker and TmoTalk employed the RAVDESS dataset for training purposes and EMOTE utilizes the MEAD dataset for emotion capture, we utilize their pre-trained weights. All methods require conditions on training speaker identity during inference. Therefore, for unseen subjects in the test dataset, we obtain their predictions by conditioning on all training identities.
retrain FaceFormer and Code on the
Implementation Details. We employ the Adam Optimizer across all modules. The EDE trained for 10,000 iterations, with the batch size set to 30. This training takes about I hour, using learning rate of 5 x 10-5, Furthermore, we use 30,000 iterations with a batch size of 50 and learning rate of 5 x 10-5, which took about 20 hours to train our EMG. All experiments are performed on a single NVIDIA RTX 3090 GPU.
4.2 QUANTITATIVE EVALUATION
Evaluation Metrics. We evaluate our method with baselines across two factors: 1) Facial expres- sion quality. Previous work used EVE (emotional vertex error) (Peng et al. 2023b), which measures the maximum le error of the vertex coordinate displacement in the forehead and eye region. How- ever, emotional expression is coordinated process involving multiple facial muscles, including the corners of the mouth, the nose, and other facial features. In light of this, we propose the VE-FID (video emotion FID), an adaptation of FID (Heusel al.] 2017) that uses a video emotion feature extractor from a frame attention network (Meng et al., 2019). replacing the inception network to focus on emotion properties. 2) Audio-Lip synchronization. We first compute the LVE (lip vertex error) that is used in previous work (Richard et al.) 2021). This metric computes the maximum & error among all lip vertices in the test set and averages & error across all frames. We then employed LSE-D (Lip Synchronization Error Distance) and LSE-C (Lip Synchronization Error Confidence) from Synenet (Prajwal et al. 2020) to evaluate the lip movement accuracy and smoothness of the rendered faces sequence. In all the tables, indicates "the smaller the better", and f indicates "the larger the better".
Evaluation on Lip sync. As shown in Table I our method achieves the highest performance in LSE-D and ranks first or second in LSE-C across all datasets. This indicates that it has strong gen- eralization and accurate lip-synchronization capabilities. is noteworthy that although Faceformer and Codetalker demonstrated superior lip- op-syncchronizttonn performance when trained on VOCASET, a decline in tip-synchronization accuracy was observed when emotion data was incorporated into the training process. This outcome further substantiates the hypothesis that our model is capable of ef- fectively separating amotion-specifie details from the speech content, thereby ensuring the precision and accuracy of the lip movements.
Evaluation on Emotion. Table [ shows our method significantly outperforms VE-FID in the RAVDESS dataset, suggesting that the emotional expressiveness of our method closely resembles that of real human expressions. Although EMOTE, which utilizes ground truth emotion labels to
7

---

Published as a conference paper at ICLR 2025
Adecmrses
Joney
Phighesk/
rbefore
/doyy (ngry)
Jay/ (Hapes)
Ahe Gurprise)
Groural Trith
FaueFoure Grouxd
26
Cookfhllle
CochTallxx
Emonle
Emtils
2
EnwiTaik
EnuTel
25
Uhitallee
UniTalox
25
25
Chors
Ouas
(a) Comparsion lip sync on HDTF
(b) Comparsion of emotion on RAVDESS
Figure 2: Comparison of generated key frame results. We show the ground truth frames used for comparison and mark the results with red arrows for out-of-sync and poor emotional quality, while zooming in the emotional details of the eyes in the pink box.
generate expressions, achieves the best VE- FID on the MEAD dataset, our method is only second to EMOTE in this metric by gap of 13, while EmoTalk, which uses the same training data as has larger gap of 39, which demonstrates our method's generaliza- tion ability emotion generation. Further- more, we assess the stability of the model's emotion generation
Table Results of emotion generation stability.
VE-FID
FIDJ
Methods
HDTF
VOCASET
Emotional
Neutral
FaceFormer
61.12
50.05
40.07
46.08
CodeTalker
59.53
54.96
43.32
53.54
Emote
37.03
35.72
38.11
34.42
EmoTTalk
27.17
38.22
38.33
33.73
UniTalker
37.04
27.59
39.4
32.779
the HDTF and VO- CASET datasets using VE-FID and compute FID metrics on the emotional and non-emotional datasets evaluate the realism of the model's gen- cration. Table 2 demonstrates that our model attains the optimal results on both VE-FID and FID metrics, indicating that our model is capable of stable and reliable facial emotional action generation.
Ours
26.43
23.40
28.54
31.81
4.3 QUALITATIVE EVALUATION
Visual Comparison In Fig. 2 we compare our method with the SOTA method on the RAVDESS and HDTF test sets. As shown subfigure (a), while most of the methods are able to generate natu- ral lip movements without emotional interference, they are sometimes inconsistent with the ground truth. For example, when pronouncing /kr/ 'decrease", the lips should pout. However, all meth- ods except our method and Unitalker produce the /i:s/ pronunciation. Additionally, EMOTE shows crooked mouth when pronouncing /'h/. As depicted in subfigure (b), the addition emotional infor- mation results in varying degrees of incorrect lip movements across different models. For example, the lips should be pursed when pronouncing /b/, but Codetalker appeared to pout, while Emotalk seems to pronounce /ai/. Furthermore, methods other than EMOTE often yield incorrect inex- pressive expressions that do not align with the ground truth. For example, when expressing angry, Faceformer loses emotional details around the eyes, and both Emotalk and Unitalker exhibit happy
8

---

Published as a conference paper at ICLR 2025
Dogs are sitting by the door
embedding
EDE
BMG
Surprised
Kids are talking by the dour
Degie
Neutral
Video emotion embosking
Hapgy
Dogs are sitting by the door
Sal
Surprised
BMG
(a) Use different speech or video as emotional feabures
(b) TSNE visualization of feature space
Figure 3: The effect of emotion embedding. Subfigure (a) shows our audio/video emotion-driven result. Subfigure (b) shows a t-sne visualization the distribution of our disentanglement feature space.
emotional expressions. Although EMOTE uses emotion labels to generate emotional expressions, it frequently produces unnatural expressions, such stiff eyebrows, which deviate from the ground truth. In contrast, our model not only maintains correct lip movements but also enables detail con- trol for different emotions. For example, in the cases of anger, happiness, and surprise, our model generates distinct representations of the details around the eyes, highlighting the impact of EDE on generating emotional expressions.
Effeet of Emotion Embedding Fig. b) demonstrates the clustering effect of our speech and video emotion embeddings, where different colors represent different emotion classes, and the cen- troids of different intensities within the same emotion are highlighted through stroking and bolding. can be seen that our model effectively captures a meaningful emotion space. Additionally, to evaluate the model's effectiivness in disentangling and generating emotional expressions, ducted an experiment: we selected neutral speech to extract content features, which were then fed into the EMG module along with the emotional features of speech and video samples containing different emotions and speech contents. As shown in Fig. (]a), this exchange effectively mod- video while maintaining precise
ifies the emotional expression match the reference speech lip-synchromization with the original speech. This shows that our EDE effectively constructs an audio-visual consistent emotionally disentangled feature space, reducing the interference of speech content with emotional information.
Table 3: User study results.
User study To evaluate the proposed model more comprehensively, we followed EmoTalk's comprehensive user questionnaire of 120 multiple-choice questions with 20 sen- tences selected as test cases from RAVDESS and VOCASET test datasets, and used the FLAME template to compare and analyze each of the models in terms of full-face comparison, lip-synchronizationn comparison and emotion expression comparison. As shown in Table 3 model received the most positive feedback from participants, scoring the highest on all three metrics by vote of 30.90%, 29.40% and 41 24%, respectively, compared the other five models. is worth noting that in terms of emotional expression, our model has an advantage over the other methods. Overall, the majority of participants considered our method superior to others.
Methods
full-face
lip syne emotion expression
FacelFormer
18.12%
13.32%
6.91%
CodeTalker
15.53%
11.67%
5.32%
Emote
10.13%
13.31%
20.89%
EmoTalk
17.17%
12.76%
12.01% %
UniTalker
20.14%
19.54%
13.63%
Ours
30.90%
29.40 %
41.24%
Ablation Studies We conducted an ablation study to assess the contributions of different compo- nents of our model. As can be seen in Table 4 there was modest increase in emotion expression eтror after removing Liriplet, with weakening of the emotion as in Fig. 4a), indicating its effec- tiveness emotion intensity differentiation. Similarly, removing Lemo (the last term of Equation [T]

---

Published as a conference paper at ICLR 2025
EcoFace
EcoFace
w/o Leripett
GT
w/o bvisual-auee
(a) emotion weakening GT
(b) averaging emotional performance
EcoPace
w/o Lsynn W Lmnsh
(c) Mouth motion instability problem
Figure 4: Ablation results while removing relevant modules.
caused a significant increase in emotion expres- sion error, underscoring the importance of vi- sual supervision in emotion learning and ex- pression. Additionally, both emotion accuracy and lip synchronization declined after removing Lea. highlighting the EDE effectiveness in re- w/o Lemo
Table 4: Ablation study for our components.
Methods
VE-FID,
IVE.
LSF-D,
LSE-CT
w/o Lu
28.11
3.18
9.812
0.921
Ltriptes
26.69
3.23
9.633
1.008
w/o Lsyne
22.01
2.22
10.068
0.946
30.88
2.51
9.603
1.021
ducing the interference of speech content well as lip movements on emotional features. We
Ours
21.57
2.19
9.616
1.010
also observed that without Lve. the generated results exhibited emotion averaging. In Fig. ((b), the left side shows generation results at different speech times. Without Lee. the results maintain single expression over time, while EcoFace display variations in mouth corners and eye details. On the right, the eye movement sequences show that EcoFace replicates the widening and squinting movements similar to the ground truth, whereas no change occurs without Lea. This demonstrates that Lvee effectively transfers visual emotion details to speech emotion features. Finally, replacing Leyne (the second-last term Equation [T] with Limes led to not only decreased lip synchroniza- tion but also unstable lip movements, illustrated in Fig, (c).
DISCUSSION AND CONCLUSION
In this work, propose audio-visual emotional co-disentanglement and speaker-aware archi- tecture for speech-driven emotional 3D facial animation. We construct an audio-visual joint and coordinated emotion space that is independent of speech content by introducing emotional motion and emotionally differentiated supervisory signals during training. At the same time, we construct low-dimensional distribution space for facial actions to improve the decoder's sensitivity to style factors and achieve personalized generation. Extensive experiments show that our method outper- forms previous methods terms of emotional quality and lip synchronization. However, a key challenge in our model is the non-interpretability of the emotion representations, we cannot deter- mine which disentangled emotion features correspond to specific facial changes, such as the degree of mouth opening or whether the eyes are wide open or squinting. One of the future works is to address this problem by exploring how facial musele movements, such as changes in facial action units (FAUs), correspond to these emotion representations, to improve the precision and control of emotional expressions. Additionally, the ability to infer emotions from speech content presents important research opportunities for enhancing the accuracy of emotion generation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by 2030 National Science and Technology Major Project (2022ZD01 National Natural Science Foundation of China under Grant (No. 62222211). National Natural Sci- ence Foundation of China (No. 62402429), Key Research and Development Program of Zhejiang Province (No. 2025C01026, 2024C03270), ZJU Kunpeng&Ascents Center of Excellence, Zhejiang University Education Foundation Qizhen Scholar Foundation and Ningbo Yongjiang Talent Intro- duction Programme (2023A-397-G), In addition, this work supported by Alibaba Group through Alibaba Research Intern Program and Alibaba Innovation Research (AIR) Program.
REFERENCES
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning speech representations. Advances in neural information processing systems, 33: 13:12449- 12449-12460, 2020.
Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael Black. Capture, learning, and synthesis of 3d speaking styles. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10101-10111, 2019.
Radek Daněček, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 20311 10312220 2022.
Radek Daněček, Kiran Chhatre, Shashank Tripathi, Yandong Wen, Michael Black, and Timo Bolkart. Emotional speech-driven animation with content-emotion disentanglement. In SIG- GRAPH Asia 2023 Conference Papers, pp. 1-13, 2023.
Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang. Max Welling, and Zhuowen Tu. Guided variational autoencoder for disentanglement learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 7920-7929, 2020.
Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. Jali: animator-centric viseme model for expressive lip synchronization. ACM Transactions on graphics (TOG), 35(,):1-11, 2016.
Paul Ekman and Wallace Friesen. Facial action coding system. Environmental Psychology & Nonverbal Behavior, 1978.
Xiangyu Fan, Jiaqi Li, Zhiqian Lin, Weiye Xiao, and Lei Yang. Unitalker: Scaling up audio-driven 3d facial animation through unified model. arXiv preprint wwXiv:2408000762 2024.
Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech- driven 3d facial animation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18770-18788 2022.
Kazi Injamamul Haque and Zerrin Yumak. Facexhubert: Text-less speech-driven (x) pressive 3d facial animation synthesis using self- supervseee speech representation learning. In Proceedings of the 25th International Conference on Multimodal Interaction, pp. 282-291, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460 2021.
Elizabeth A Kensinger. Remembering emotional experiences: The contribution of valence and arousal. Reviews the Neurosciences, 15(4):241-2) 2004.

---

Published as a conference paper at ICLR 2025
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems, 33:18611-18673, 2020.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 131.:1111 2013.
Dimitrios Kollias and Stefanos Zafeiriou. Affect analysis in-the-wild: Valence-arousal, expressions, action units and unified framework. arXiv preprint arXiv:2103.1579) 2021.
Tianye Li, Timo Bolkart, Michael Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6): 2017.
Steven R Livingstone and Frank Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)ec016339,, 2018
Debin Meng, Xiaojiang Peng, Kai Wang. and Yu Qiao. frame attention networks for facial expres- sion recognition videos. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 3866886665 3870. IEEE, 2019. URL https: /tt.:tt////ggt.bbb wtt.shub..ooocccooooommmm wwwp://pen--ebb///nnnototo
Ziqiao Peng, Yihao Luo, Yue Shi, Hao Xu, Xiangyu Zhu, Hongyan Liu, Jun He, and Zhaoxin Fan. Selftalk: self-supervised commutative training diagram to comprehend 3d talking faces. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 5292-5301, 2023a.
Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun Hc, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech-driven emotional disentanglement for 3d face animation. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 20687-20697, 2023b.
KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip syne expert is all you need for speech to lip generation in the wild. Proceedings of the 28th ACM international conference on multimedia, pp. 484-492, 2020.
Yi Ren, Jinglin Liu, and Zhou Zhao. Portaspeech: Portable and high-quality generative text-to- speech. Advances in Neural Information Processing Systems, 34: 14:1333-11397 2021.
Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh. Meshtalk: 3d face animation from speech using cross-modality disentanglement. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1173 2021.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 815-823, 2015.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio- visual speech representation by masked multimodal cluster prediction. arXiv preprint 1/Xiv:2201.02 2022.
Sarah I. Taylor, Moshe Mahler, Barry-John Theobald, and lain Matthews. Dynamic units of visual speech. In Proceedings of the 11th ACM SIGGRAPH//Erographics conference on Computer Animation, pp. 275-284, 2012.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang. Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking- face generation. In European Conference on Computer Vision, pp. 700-717. Springer, 2020.
Zehan Wang, Yang Zhao, Haifeng Huang, Jiageng Liu, Aoxiong Yin, Li Tang, Linjun Li. Yonggi Wang, Ziang Zhang, and Zhou Zhao. Connecting multi-modal contrastive representations. Ad- vances in Neural Information P Processing Systems, 36:22099-22111 2023.
12

---

Published as a conference paper at ICLR 2025
Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong.
Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12780-12790, 2023.
Yuyu Xu, Andrew W Feng, Stacy Marsella, and Ari Shapiro. A practical and configurable lip sync method for games. In Proceedings of Motion on Games, pp. 131-140. 2013.
Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. arXiv preprint arXiv:2301.134400 2023.
Shengyu Zhang, Tan Jiang, Tan Wang, Kun Kuang, Zhou Zhao, Jianke Zhu, Jin Yu, Hongxia Yang, and Fei Wu. Devibert: Learning deconfounded visio-linguistic representations. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 1373-4382, 2020.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face gen- eration with high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3661-3670, 2021.
13

---

Published as a conference paper at ICLR 2025
250
(50, 1024))
250
(50,1024)
Transformer
Linear Interpolation
X50
$25
(25,1024)
CNN Encoder
Res.Net
(16000,1)
2F
(28,1,128,123)
(a) Structure of audio encoder
(b) Structure of video encoder
Figure 5: Structure of the emotional disentanglement encoder.
A DETAILS OF MODELS
A.1 EMOTIONAL DISENTANGLEMENT ENCODER
The Emotional Disentanglement Encoder (EDE) comprises an audio encoder and a video encoder. The architecture of the audio encoder, as depicted in Fig. [[a), consists of convolutional encoder followed by a transformer module. For a one-second speech segment, sampled at 16 kHz, the input to the audio encoder is matrix of dimensions (16000, 1). This is processed to yield feature matrix of size (50, 1024). The video encoder, shown in Fig. (b). employs modified ResNet- 18 architecture. Specifically, the first convolutional layer in the ResNet- is replaced with 3D convolutional layer with a kernel size of 5 x X 7. The resulting visual feature tensor is subsequently flattened into one-dimensional vector via 2D average pooling layer. For one-second video sequence, the input to the video encoder is a matrix of dimensions (25, 1, 128, 128), where 25 represents the frames per second (fps) and each frame is grayscale image of resolution 128 by 128 pixels. After processing by the residual neural network. a feature matrix of size (25, 1024) is produced, containing feature vectors for each frame. To align the output dimensions with those of the audio encoder, linear interpolation is applied, resulting in final feature matrix of size (50, 1024).
A.2 EMOTIONAL MOTION GENERATOR
Our emotional motion generator consists of an encoder, decoder, and flow prior model. The encoder. as shown Fig. ([ja)s composed of a ID-convolution followed by ReLU activation and layer normalization, and condition- WaveNet. The decoder, shown in Fig. b(b), consists of a condition- NeveeNt and a ID transposed convolution followed by ReLU and layer normal- ization. The prior model, shown in Fig. (ee). is normalizing flow, which is composed of ID-convolution coupling layer and a channel-wise flip operation. Condition- WayNer consists of multiple layers that composed of a dilated-connollttin and ID-convolution, in which the condi- tional features are computed by the ID-convolution and summed with the input features computed by the cilated-comvolution to achieve conditional control. Audio emotional features and Hubert fea- tures are utilized as the audio condition of these three modules, while person ID is utilized as the person condition of the decoder.
A.3 SYNC-EXPERT
Our sync-expert inputs window of Ti consecutive mesh landmark frames and an audio feature clip of size T, D, where Ti and Tı are the lengths of the video and audio clip respectively, and D the dimension of Hubert features. The sync-expert the input audio and landmarks are synchronized. It consists of a landmark encoder and audio encoder, as shown Fig. 7 both of which are comprised of a stack of ID-convolutions followed by batch normalization and ReLU. We cosine-simin rosine-iimilarity with binary cross-entropy loss train the sxnc-expert. Specifically, we compute cosine-simee sisiia-simmlarity for the landmark embedding and audio embedding to represent the probability that the input audio-landmark pair is synchronized. The
trained to discriminate whether
14

---

Published as a conference paper at ICLR 2025
FLAME parameter sequence
μ.
bubert
Condition- WavNet
TransposedConv +ReLU Norm
ED audioo
Conv1D +ReLU
P Person id hubert audooo
+Layer Norm
Condition- Wav Net
FLAME parameter sequence
(a) Encoder
Decoder
fuature
2p
kc Laayrr
Conv1D
FLIP
tanh
hubert
CDn+1D+ Condition- WavNet
EDE audio
condition
CorviD ID
Dilated Conv
z0,(0,1)
Input feature
(c) Flow Prior
(d) Condition WavNee
Figure 6: The structure of encoder, decoder, flow prior and condition-wavner in emotional motion generator.
Pos/Neg?
cosine similarit
Rel.u
ReLu
IxN
Conv +BN
ConvID
hubert clip
mesh landmark clip
Figure 7: The structure sync-expert.
training loss of sync-expert be represented as:
a max |al22 114122e) c)
Logne = CE
(12)
15

---

Published as a conference paper at ICLR 2025
Facr, umdd
Figure 8: t-sne reault of the emotion space (w/o triplet loss).
Figure 9: t-sne reault of the emotion space.
B VISUALIZATION OF EMOTIONAL INTENSITY DIFFERENTIATION
B. THE MOTIVATION FOR COMBINING CONTRASTIVE LOSS AND TRIPLET LOSS TO DISTINGUISH EMOTIONAL INTENSITY
The Valunce-Arousal model Kensinger 2004 Kollias & Zafciriou 202I) a widely utilized framework for emo- tion classification. As shown in IO] the VA model suggests that emotions can be represented as combina- tions of valence and arousal, with arousal being par- ticularly key in distinguishing between different emo- tional intensities. This implies that intensity assessment is inherently tied to the emotional type assessment (via arousal), where emotions of the same type share similar valence but can differ subtly in arousal. Therefore, our approach first allows the model to learn separate latent spaces for each emotion, effectively distinguishing the arousal and valence ranges for different emotions via the contrastive loss. Afterward, the triplet loss works within each emotion's 's latent space to further differentiate emo- tions based on their intensity by focusing the subtle differences in arousal. This two-step process ensures that our model can effectively discriminate between both emotional types and their varying intensities.
Figure 10: The Valence-Arousal model.
16

---

Published as a conference paper at ICLR 2025
B.2 T-SNE VISUALIZATION OF EMOTIONAL INTENSITY
As shown in Fig. 8 we use only contrast loss to construct the emotion space, and since arousal reflects both emotion category features and emotion intensity features, and emotion intensity fea- tures are relatively homogeneous, the model ignores intensity features, and there is confusion in the distribution of emotion intensities within each separate emotion space. As shown in Fig. [9] after combining the triplet loss, there is a distinction between emotion intensity distributions within the same emotion space.
C EMOTIONAL EXCHANGE EXAMPLES
C.I EXCHANGE OF ONE EMOTION
We select news speech clip (neutral) to extract the speech content features, and select the surprised (level 2) and happy (level I) speech of two different characters to extract the speech emotion features, and then input them into EMG along with the content features respectively for generation. The results are shown in Fig. 12] While ensuring consistent lip synchronization, our model can generate different expression details, such upturned corners of the mouth and wide eyes.
C.2 EXCHANGE EMOTION IN MORE CHALLENGING EXPERIMENT
In order to verify whether the model will learn emotional features from speech content, we selected happy and sad emotional speech to extract speech content features, and selected related speech or video with very different emotions to extract emotional features, respectively. The generated results are shown Fig. 13 and Fig. [4 can be seen that our model better achieves decoupled learning of speech content and emotion.
C.3 EXCHANGE OF Two EMOTIONS
We select a news speech segment (neutral) to extract speech content features, and select happy (level 2) and sad (level 1) speech of two different characters to extract speech emotion features and concatenate them together, and then input them together with content features into our EMG for generation. The results are shown Fig. 15] While ensuring consistent lip synchronization, our model also implements smooth emotional changes, such as gradual change in the corners of the mouth from upward downward.
D SPEAKER-AWARE EFFECTS
speaker
The motivation behind designing speaker- aware generation stems from our observation that different speakers exhibit distinct speech styles and emotional expression details. If the model is trained to fit data from all speakers without considering these differences, it can lead to issues such as lip-syncing inconsisten- cies (e.g., pursed lips, twisted lips) and reduc- tion in emotional expressiveness. To demon- strate the impact of this design choice, first trained our model without considering speaker- aware conditions and evaluated it on the test sets of the RAVDESS and HDTF datasets. The results, as shown Table [5] indicate a slight weakening in emotional performance, but more notably, there is significant reduction in lip-sync accuracy. As shown in Fig. []] the crooked, abnormally pursed lips that occur without consideration of the particular speaker can be effectively avoided by consideration of the particular speaker's 's style.
ExoFace
Figure 11:
The Speaker- aware effects.
17

---

Published as a conference paper at ICLR 2025
Table 5: Quantitative evaluation results speaker-aware module.
RAVDESS
HDTF
Methods
VE-FID
LVE
LSE-D
LSE-C
LVE
LSE-D 4 LSE-C1
w/o speaker-aware
25.67
3.53
10.552
0.801
3.47
11.891
0.711
Ours
21.57
2.19
9.616
1.010
2.61
10.253
0.823
covid
job
compass
to
at
GT (neutral)
Surprise (Ievel2)
Happy (level1)
Figure 12: Generation results for exchanging whole speech emotional features.
Kids are talking by the door
ED ED Eandio
EMG
Angry Dogs are sitting by the door
Hubert
t
Happy
Kids are talking by door
25
DE
EMG
Sad
Figure 13: Generation results for exchanging emotion from positive to negative.
18

---

Published as a conference paper at ICLR 2025
Dogs are sitting by the door
ED andio
Happy Kids are talking by the door
Hubert
t
Sad
Dogs are sitting by the door
DE
EMG
Surprised ?
Figure 14: Generation results exchanging emotion from negative to positive.
thousand
this
cooperation
cup
GT (neutral)
Happy(level2) sad((eeell)
Figure 15: Generation results for exchanging speech emotional features.
19


Reviews:
[{"review_id": "CEPNA3q5mX", "reviewer": "Reviewer_r7xK", "summary": "The paper proposes EcoFace, a framework for generating 3D talking faces using speech signals. The framework first constructs an audiovisual emotion space that is independent of the speech content. Using a Variational Autoencoder the framework then generates FLAME parameters (a low-dimensional representation of facial movements). The model’s encoder is conditioned on both speech and emotion features to create a latent representation that captures expressive nuances. Finally, a decoder that is conditioned on the speaker is used for the facial animation generation. Experimental results show better performance in both emotional expressiveness and lip-sync accuracy when compared with state-of-the-art methods.", "questions": "1. You mention in the introduction that your model can discriminate the signal of different types and intensities of emotion features. How does your model interpret varying intensities of emotion and how does the contrastive-triplet loss contribute to this? \n\n2. Sec. 4.1: To evaluate the methodology on the test set for unseen subjects you condition on all training identities. Why use all identities rather than a subset?\n\n3. Although you retrained FaceFormer and CodeTalker on RAVDESS and HDTF, you did not retrain EMOTE which seems to provide better results than your approach on MEAD that the model was trained on. For fair comparison could you perform this experiment by retraining EMOTE on the RAVDESS too?\n\n4. Table 3: Can you show the results on each dataset separately and for each emotion? Since your model was trained on RAVDESS it may contain some bias towards that dataset when compared with EMOTE.\n\n5. Sec. 4.3,”Effect of emotion embeddings”: Have you tried to extract content features from an emotional speech signals? How would the model perform in this more challenging experiment?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Novel approach for emotion disentanglement using speech and visual information.\n2. Comprehensive evaluation through quantitative/qualitative metrics, user studies, and ablation experiments.", "weaknesses": "1. Some methodological choices are unclear, and additional evaluation details are needed (see questions).\n2. Speaker-specific modeling. Generation can only be performed for speakers the model has seen during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Nd6LwFBLH", "reviewer": "Reviewer_AbM6", "summary": "EcoFace addresses issues in 3D facial animation: feature confusion (lack of clear signals for content and emotion), emotion weakening (difficulty in controlling emotion intensity), and the mean-face problem (limited control over individual speaker’s style of expression). The authors tackle these issues by introducing explicit signals for emotional motion representation and intensity control using audio-visual loss and a contrastive triplet loss to distinguish emotion intensities. They also generate speaker-specific, stylized facial animations with a lip-sync discriminator.", "questions": "1. L219: What kinds of augmentations were applied to the video frames and audio samples?\n2. L225: The latent representations are averaged over the sequence. How is it ensured that continuous emotional information along the sequence is not lost? Could the authors provide more details on the contrastive loss in Equation 4, and clarify if other methods besides averaging were considered, such as mapping to a lower dimension with a learnable layer?\n3. fig 1a: The method name is missing in the caption.\n4. fig 1b: The triangle and circle shapes representing features lack labels indicating what they refer to.\n5. fig 4 and Ablation Studies (pages 9,10): \"Ecotalk\" is used instead of \"EcoFace.\"\n6. What was the rationale for training the sync expert independently? How are the FLAME region landmarks obtained—are they input as rendered crops or as 3D vertices?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The work presents following strengths:\n1. EcoFace introduces an audio-visual emotion disentanglement mechanism to supervise the discrepancy between emotional information captured by facial motion and the audio stream, effectively capturing information from both audio and video. \n2. EcoFace controls within-emotion intensity by using an emotional triplet loss.", "weaknesses": "1. The GT videos on the webpage show severe issues, such as unnatural mouth closure and a jerky appearance, which don’t match the [original implementation](https://download.is.tue.mpg.de/emoca/assets/emoca_v2_comparison.mp4). Since EcoFace is trained on meshes obtained using EMOCA, the authors should clarify how their model produces improved animations compared to the GT animations on which it was trained, as shown in their demo videos.\n2. The EMOTE comparison videos on the webpage don’t match the exemplar videos from other methods, such as [FaceTalk](https://youtu.be/7Jf0kawrA3Q?si=ZvxQT3FD27Eh-RDj), [3DiFACE](https://youtu.be/Mep5pAU3TPc?si=Pe93jkT_eC_pBjPZ), [EMOTE](https://download.is.tue.mpg.de/emote/EMOTE_SupMat_video.mp4) . Could the authors explain the observed differences?\n3. Triplet loss in eq 7 operates within a single emotion space. Since it focuses on a single emotion, how does EcoFace ensure that emotion intensities for high arousal, high valence (e.g., surprise) are disentangled from those of high arousal, low valence (e.g., anger or fear)?\n4. Since EcoFace claims to generate speaker-specific animations, it would help to provide more details on the number of identities in the training set and any animation results for unseen subjects for better evaluation.\n5. Training details are unclear. With a batch size of 30, how are typical forward passes structured to use 2N pairs for computing contrastive loss in Eq. 4, and how is triplet loss computed per emotion in Eq. 7?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nNTJRmWStS", "reviewer": "Reviewer_kKfU", "summary": "This paper addresses the challenge of separating emotional information from speech content in facial animation, which often leads to feature confusion and emotion weakening. This work, EcoFace, introduces audio-visual co-disentanglement and a speaker-specific motion generator to address these issues. By using an audio-visual loss for emotion consistency and a contrastive-triplet loss for distinct emotional space, EcoFace creates a low-dimensional motion distribution that captures speaker-specific styles for personalized animation.", "questions": "1. What is the demographic information in the user study?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a novel framework to distanglment emotion and speech content in the face synthesis field.\n2. This work is well-written and well-organized.\n3. The results shows the efficiency of the proposed framework.", "weaknesses": "Major Concerns:\n\n1. LSE-C Range Consistency\n   There appears to be a discrepancy in the LSE-C values reported in this manuscript compared to those commonly cited in the community. For instance, DINet [https://arxiv.org/pdf/2303.03988] lists an HDTF LSE-C value (as GT) of 8.9931, while the value reported in this work is 0.824. Clarification regarding this disparity is needed, as it may suggest a difference in evaluation methodology or algorithm implementation. \n\n2. Definition Clarity for L_emo and L_sync in Ablation Study.\n   The explanation of L_emo and L_sync in Table 4 is missing, which decreases in interpreting the results and understanding the contributions of each component.\n\n3. Validity and Interpretation of VE-FID Metric\n   Introducing VE-FID as a new metric to measure emotion expression is a great try. However, some results in Table 1 and Table 4 raise questions. For example, the comparison between EcoFace (21.57) and EmoTalk (51.98) shows a large difference, even when EcoFace lacks L_emo (30.88). Does this suggest that EcoFace may outperform EmoTalk on emotion-related metrics even without the emotion-related modules or guidance?\n\n4. Lack of Results for Speaker-Specific Generation\n   One of key contributions is implementing speaker-specific generation; however, the absence of results supporting this feature weakens its impact. Providing empirical evidence or case studies to demonstrate speaker-specific generation’s effectiveness would strengthen the claim.\n\nMinor Issues:\n\n1. On Page 3, Lines 160-161, please use “Wav2Vec 2.0” instead of “Wav2Vec2”.\n\n2. In Figure 1(a) on Page 4, the method name is currently missing.\n\n3. On Page 10, in Figure 4(a) and Figure 4(c), Line 518, the label/term should be corrected from “EcoTalk” to “EcoFace.”\n\n4. Demographic Information in User Study", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "baPAyA08Gc", "reviewer": "Reviewer_puML", "summary": "This paper describes a framework for disentangling facial expressions of emotion from facial motions related specifically to speech.  The goal is to later re-synthesize expressive talking faces driven by speech.  Furthermore, a triplet loss is included to ensure that different expressions can be separated in the representation, and that the differences in the magnitude of same expression can be disambiguated.  To account for differences in the way that different people display facial expressions, the generator is conditioned on a learned identity embedding..", "questions": "How is the level of emotion in the training data assigned?\n\nOn line 135 — is an element of s \\in R^{n} the probability of that expression being present?\n\nIn Equation (7) how is alpha set?\n\nIn the paragraph following Equation (7), should z_l} be z_{i} to match the equation?\n\nIn Equation (11), does each term contribute equally to the loss?  There is no weighting to account for things like differences in scale?\n\nOn line 477 — you mentioned the margin that you beat the baselines by being 30.9%, 29.4%, and 41.24%.  This is not a margin though is it?  Are these not the scores that your approach attained?  The margin would be the difference.\n\n*Suggestions*\nChange references to. “speaker-specific” to “speaker-aware”.  Speaker-specific typically means that you train a different model for each individual speaker.  \n\nOn line 135 you mention “where each a \\in R^{D} has D sampled audio.  I think you mean that each a \\in R^{D} is a D-dimensional feature vector.  I am not sure what ”D sampled audio“ means.\n\nIn Section 3.1 it would help to specify that the speech features are latent features and not typical speech features, e.g., from a Mel-filter bank.\n\nIn Section 3.1 it was not clear that for T frames of audio, why there are 2T frames of emotion-related and 2T frames of content-related features.  Maybe explain this when these are introduced.\n\nThe equations should flow as part of the sentences.  Throughout the paper you end a sentence before providing the equation.\n\nLine 146 — replace low-latitude with low-dimensional.\n\nOn line 232, you refer to “the voiceless phase”.  Voiceless has specific meaning in speech.  To be clear here you should avoid using this terms and refer to “periods where speech is not present”.  (For reference, voiceless speech is speech produced when the vocal chords are apart.).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem being tackled is important and challenging.  We are highly sensitive to discrepancies in generated facial motion that accompanies speech.  This work tackles the problem by disentangling facial motion related to emotion from facial motion related to speech.  Furthermore, accounting specifically for the degree of expression is effective.\n\nThe work uses open data to aid re-producibility.  Furthermore, code will be made available.\n\nThe approach is effective.  The performance against the baselines is good.  The demo videos are impressive.  Talking head generators often cannot generalize to things like singing (because the sustained gestures look unnatural) but the approach here does an excellent job.  Furthermore, the differences in the articulation for different languages is very well captured.  For example, the lip-rounding in articulation of French speech is very impressive.\n\nThe ablations show the importance of the design of the components of the system.\n\nI appreciate the inclusion and attention to detail in Section 4.3.", "weaknesses": "A limitation of the objective metrics, such as LVE, is that they do not account for type of error.  A larger LVE in the articulation of, say, /k/ might be insignificant, but a large error in the articulation of, say, /b/ is highly problematic.  I realize these are standard metrics, but I still see these as problematic for this reason.\n\nSee the questions/suggestions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)