Emotional Face-to-Speech
Authors: Jiaxin Ye
Boyuan Cao
Hongming Shan
Venue: ICML 2025 poster (2025)
Keywords: Generative Model
Discrete Diffusion Model
Speech Generation
Text-to-Speech
Primary area: applications->language_speech_and_dialog
TL;DR: We propose a novel RVQ-based discrete diffusion framework DEmoFace for a new task (Emotional Face-to- Speech, eF2S), which is the first attempt to customize consistent vocal style including timbre and emotional prosody solely from the face.

Abstract: How much can we infer about an emotional voice solely from an expressive face? This intriguing question holds great potential for applications such as virtual character dubbing and aiding individuals with expressive language disorders. Existing face-to-speech methods offer great promise in capturing identity characteristics but struggle to generate diverse vocal styles with emotional expression. In this paper, we explore a new task, termed *emotional face-to-speech*, aiming to synthesize emotional speech directly from expressive facial cues. To that end, we introduce  **DEmoFace**, a novel generative framework that leverages a discrete diffusion transformer (DiT) with curriculum learning, built upon a multi-level neural audio codec. Specifically, we propose multimodal DiT blocks to dynamically align text and speech while tailoring vocal styles based on facial emotion and identity. To enhance training efficiency and generation quality, we further introduce a coarse-to-fine curriculum learning algorithm for multi-level token processing. In addition, we develop an enhanced predictor-free guidance to handle diverse conditioning scenarios, enabling multi-conditional generation and disentangling complex attributes effectively. Extensive experimental results demonstrate that DEmoFace generates more natural and consistent speech compared to baselines, even surpassing speech-driven methods. Demos of DEmoFace are shown at our project https://demoface.github.io.

Emotional Face-to-Speech
Jiaxin Ye Boyuan Cao
Hongming Shan
Abstract
1. Introduction
When we encounter a person's face on platforms like Insta- gram or Facebook without hearing their voice, our minds instinctively generate auditory expectations based visual cues. These expectations are shaped by our experiences and cultures, influencing how we perceive individuals to sound based on their external appearance, such as age, gender, na- tionality, or emotion (Paris et al., 2017; Taitelbaum- -weed & Fostick, 2016). These preconceived notions drive us to form judgments about others' voices even before they speak.
How much can infer about an emotional voice solely from expressive face? This intriguing question holds great potential for applications such as virtual character dubbing and aiding indi- viduals with expressive language disorders. Ex- isting face-to-speech methods offer great promise in capturing identity characteristics but struggle to generate diverse vocal styles with emotional expression. In this paper, explore new task, termed emotional fuce-to-speech, aiming to syn- thesize emotional speech directly from expressive facial cues. To that end, we introduce DEmo- Face, a novel generative framework that leverages discrete diffusion transformer (DiT) with cur- riculum learning, built upon a multi-level neural audio codec. Specifically, we propose multimodal DiT blocks to dynamically align text and speech while tailoring vocal styles based on facial emo- tion and identity. To enhance training efficiency and generation quality, we further introduce a coarse-to-fine curriculum learning algorithm for multi-level token processing. In addition, we develop enhanced predictor-free guidance to handle diverse conditioning scenarios, enabling multi-conditions. generation and disentangling complex attributes effectively. Extensive experi- mental results demonstrate that DEmoFace gen- erates more natural and consistent speech com- pared to baselines, even surpassing speech-driven methods. Demos of DamoFace are shown our project https://demoffccegitttut
In recent years, face-guided Text-to-Speech (TTS) (Goto et al., 2020; Kang et al., 2023; Jang et al., 2024), also known as Face-to-Speech or F2S, has attracted growing interest with diverse applications, such as virtual character dub- bing and assistance for individuals with expressive language disorders. The goal of the F2S is to create voices that are consistent with the guided face. However, users increasingly expect generated speech that not only replicates speakers' identities but also conveys rich emotional prosody, enhanc- ing their experience in human-machine interactions (Wen et al., 2024; Cong et al., 2024a; Ye et al., 2023b). This ex- pectation is beyond the scope of existing F2S tasks, lacking explicit guidance to produce desired emotional speech.
Considering that facial expressions are the most direct indi- cators of emotion, we propose an extension the F2S task grounded in visual cues, termed emotional Face-to-Speech (eF2S). An example of the proposed eF2S task is illustrated in Fig. (bb). Unlike the conventional F2S task, which con- verts text to speech guided solely by identity embeddings extracted from a reference face, our eF2S task further decou- ples identity and emotion from the facial input, producing speech that preserves the speaker's identity while enriching it with the emotional expression derived from the reference face. While the new eF2S task may initially appear to only require the generated speech to convey emotions, it raises several novel challenges. On the task level, traditional F2S tasks are insufficient for eF2S, as they focus on converting text into speech that reflects facial characteristics without considering emotional states. On the method level, previous expressive speech generation methods focus generating speech tied to either specific identity or a specific emotion, but customizing both simultaneously remains a significant caalenge-ppariiculary in the absence of speech cues.
Institute of Science and Technology for Brain- Inspired Intelli- gence. MOE Frontiers Center for Brain Science, Key Laboratory of Computational Neuroscience and Brain- Inspierd Intelligence, and State Key Laboratory of Brain Function and Disorders, Fudan University, Shanghai, China. Correspondence to: Hongming Shan <hmshan @ edu.e...
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Emotional Face-to-Speech
Prompt
Hello World
Hello World
Speech Output
Speech Output
Prompt
Spce-to-Speech
Face Prompt
Emotional Face-ta- Sp-eehh
"Neutral"
"Surprisad"
"Angry"
"Surprised"
Inconsistent
Consistent
(a) F2S struggles to model consistent speech
(b) Our cF2S explores consistent synthesis from rich visual cucs
Figure I: Tasks comparison. (a) Conventional Face-to-Speech (F2S). (b) The introduced Emotional Face-to-Speech (eF2S). Given text and face prompts, the model is expected to generate speech that aligns with both the facial identity and emotional expression. Our eF2S offers novel perspective for generating consistent speech without relying any vocal cues.
To address these challenges, we propose a novel discrete Diffusion framework for Emotional Face-to-speech, called DEmoFace, which is the first attempt to generate consis- tent auditory expectations (i.e. identity and emotions) di- rectly from visual cues. Specifically, mitigate the one-to- many issues inherent in continuous speech features (Xue et al., 2023), we begin by discretizing the speech gener- ation process utilizing neural audio codec with Residual Vector Quantization (RVQ). Considering the low-to-high frequency distributions across different RVQ levels, we then introduce discrete diffusion model with a coarse-to-fine curriculum learning strategy, to enhance both training ef- ficiency and diversity of generated speech. Building this, we propose multimodal DiT (MM-DiT) for the re- verse diffusion process, which dynamically aligns speech and text prompts, and customizes face-style linguistic ex- pressions. Furthermore, for challenging multi-conditional generation of eF2S, we develop enhanced predictor-fre free guidance (EPFG) on the discrete diffusion model, boosting efficient response the global condition while facilitat- ing the decoupling of multiple local conditions. Compre- hensive experiments demonstrate that DEmoFace achieves diverse and high-quality speech synthesis, outperforming the state-of-te oatte-----h--att the-art models in naturalness and consistency exceeding even speech-guided guided methods. The contributions of this paper are summarized below:
2. Related Work
2.1. Residual Vector Quantization for TTS
Neural audio codecs (Zeghidour al., 2022: Zhang et al., 2024) enable discrete speech modeling by reconstructing high-qualin audio at low bitrates. Residual Vector Quantiza- tion (RVQ) (Vasuki & Vanathi, 2006; Zheng et al., 2024) is a standard technique for codecs that quantizes audio frames by multiple hierarchical layers of quantizers, with each sub- sequent layer working on the residual of the previous stage. Recent models (Kharitonov et al., 2023; Shen al., 2024) rely on codecs with the RVQ to synthesize speech, and show promising performance on naturalness. For example, VALL-E (Wang et al., 2023) employs Encodec (Défossez et al., 2023) to transform the speech into a sequence of discrete tokens, then uses an auto-regressive (AR) model to predict tokens based on phonemes and speaker prompts. However, existing methods manner, leading to unstable and inefficient sampling. We propose a discrete diffusion framework DEmoFace to re- construct tokens from the RVQ codec, achieving faster and more diverse sampling by parallel iterative refinement.
mainly based on the AR
2.2. Face-driven TTS
Face-driven TTS (F2S) aims to synthesize speech based on visual information about the speaker (Plüster et al., 2021; Lee et al., 2024: Goto et al., 2020; Kang et al., 2023; Ye Shan, 2025). Previous F2S methods focus on how to learn visual representation from speech supervision. For example, Goto et al. (Goto et al., 2020) propose supervised gen- eralized end-to-end loss to minimize the distance between visual embedding and vocal speaker embedding, However, existing methods ignore the rich emotional cues inherent in the face, which often generate over-smoothing speech lacking diverse emotional naturalness. A Although Kang et al. (Kang al., 2023) additionally introduce speech prosody codes enhance the naturalness. They still depend on the speech prompt to achieve natural generation, which does not satisfy the requirements of eF2S. In contrast, our DEmo-
We introduce an extension to F2S task. named Emotional Face-to-Speech (eF2S), which is the first attempt to cus- tomize a consistent vocal style solely from the face.
We propose a novel discrete diffusion framework for speech generation, which incorporates multimodal DiT and RVQ-based curriculum learning, achieving high- fidelity generation and efficient training.
We devise enhanced predictor-free guidance to boost sampling quality in colliiccodiiional scenarios of eF2S.
Extensive experimental results demonstrate that DEmo- Face can generate more consistent, natural speech with enhanced emotions compared previous methods.
2

---

Emotional Face-to-Speech
Face only utilizes visual cues to form the emotional auditory expectations without relying any vocal features.
for each dimension with the same token-level diffusion ma- trix 0 (f)) RR where (t)) is the noise schedule and Qlok is designed to diffuse towards an absorb- ing state [MASK] Then the forward equation is formulated as P(r x = exp (o()) (at, (at,,r) =h)), where transition probability matrix P(a =b)) := p(ri(zo), and cumulative noise (() So (8))ss. There are two probabilities in the Pilu: 1 e ( for replacing the current tokens with [MASK], e -+()) for keeping unchanged.
2.3. Emotional TTS
Emotional TTS aims to enhance synthesized speech with emotional expressiveness (Li et al., 2024; Guo et al., 2023b; Chen et al., 2022). Existing methods can be divided into two categories based on how to integrate emotion informa- tion into TTS systems. For emotion label conditioning (Ye et al., 2023a; Wen et al., 2022: Ye et al., 2022), EmoD- iff (Guo et al., 2023b) introduces a diffusion model with soft emotion labels as a classifier guidance. In contrast, V2C-Net (Chen et al., 2022) employs emotion and speaker embeddings from reference face and speech individually for speech customization. However, previous methods do not explore how to learn both speaker identity and emotion from the face image. We offer a novel perspective of the relationship between auditory expectations and visual cues for TTS without relying on any vocal cues.
Reverse denoising process. As the diffusion matrix Qi is known, the reverse process can be given by a reverse tran- sition rate matrix Q: (Sun et al., 2023; Kelly, 2011), where
Qt (z), and z/-ar * zt. or (z) 2²) = Σ Expr. Q1(z,a)). The reverse equation is formulated as follows:
p(a) or)
p(z()
p(=f-avl=)) d -1( At) (2)
where we can estimate the ratio p( At)
(which is known
3. Preliminary: Discrete Diffusion Models
p(z))
as the concrete score (Lou et al., 2024; Meng et al., 2022) to measure the transition probability or closeness from a state xf at time I to a state .' at time / - Al) of Q by a score
Continuous Diffusion Models (CDM) (Blattmann et al., 2023; Li et al., 2023; Ruan et al., 2023) have achieved state-of-the-as results in generative modeling, but face chal- lenges speech generation due to high-dimensional speech features and excessive diffusion steps, frustrating practical application. The fundamental solution lies in compressing the speech feature space, such as discrete space.
network so so(ri, 1) ~ p(ai_a))
p(z)) | So that the reverse matrix is parameterized to model the reverse process yo (=)))))=)))) (i.e. parameterize the concrete score).
Training objective. Denoising score entropy (DSE) (Lou et al., 2024) is introduced to train the score network sg:
Recently, Discrete Diffusion Models (DDMs) have shown promise in language modeling (Meng et al., 2022; Lou et al., 2024) and speech generation (Yang et al., 2023; Wu et al., 2024). We emphasize that DDM has yet to be explored in multi-conditional speech generation with high-quality audio compression. In this paper, to our knowledge, we take the first attempt to generate RVQ-based speech tokens with DDM. Below, we outline the forward and reverse processes of the DDM, along with its training objective.
Lo Ex pp((
Σ Q (1) [so (c()()
(3)
Cafat log 80 (at, 1) o; + N(Cotot) di,
where the concrete score Catat
p p(=](=)) and a normaliz-
ing constant function N(c) == clog c that ensures loss non-negative. During sampling, we can replace the concrete score with the trained score network on Equation (2).
Forward diffusion process. Given a sequence of tokens x _.1 nd from a state space of length d like d' The continuous-time discrete Markov chain at time is characterized by the diffusion matrix Q. E R xn" (i.e. transition rate matrix), as follows:
4. Methodology
4.1. Task Formulation for eF2S
p(a/ ))) = ببيد +
((())) (1)
Given a triplet of multimodal-driven conditions {Ci, Cemo: Gext), which correspond to reference identity, emotion, and text, respectively, the eF2S task aims to syn- thesize speech based on the c. More precisely, the syn- thesized speech content aligns with the text condition Cext while its voice identity and emotional attributes correspond to the identity condition Cjd and emotion condition Cemo- respectively- both extracted from the input face.
where x/j denotes i-th element of Tt, Qe Ars ./) is the (#t+ .:) element of Qt, denoting the transition rate from state æt to state zi+ at time f, and 8 is Kronecker delta. Since the exponential size of Q. existing works (Lou et al., 2024; Ou et al., 2024) propose to assume dimensional inde- pendence, conducting a oineeiimalll diffusion process
3

---

Emotional Face-to-Speech
Speech Waveform
Face Conditioner
Codec Decoder
Scale
||||II
Spench
GEZE
Groza Mereal
MLP
ArcFace
Euler Sampler
Scale,
19
FaceNet
Taker Length
Layer
omo
IVV3-
doore
RVO-1 WoO--Score
AVDH dowe
Artss-Attenton
Molt-HHadd
Text Conditioner
Norm
Hilla
SpeechTS
Multi-level Linear Heads
Apst Tuet
Cond Cumst
Scale
Speech Tokenization & Diffusion
MM-DiT Block
Mutti-Head Sulf-Attention
Codecc Encoderr
Embed
Arargge
Embed
Binee
AD
Scale, Shift
MLP
time
Layer Norm
Embed
Elenent wide Addition
Embedd
Embedding Layer
text
Tokon Embedding
Frezes Weight
Concetenat
Masked Token xf
Times
Condition Safe
Figure 2: Overall framework of DEmoFace. The MM-DiT inputs masked token x 2, time t, and condition set to synthesize speech, consisting of N blocks for conditioning and 12 linear heads to predict concrete scores. During training, we propose a curriculum learning that first inputs lew-level tokens and refines them by adding high-level tokens progressively. During sampling, an Euler sampler with our EPFG refines the tokens, while a codec decoder reconstructs the waveform.
4.2. Overview of DEmoFace
RVQ speech tokenization. We utilize the recent RVQ- based codec (Wang et al., 2024) as the tokenizer, which achieves hierarchical modeling of diverse information across different RVQ layers. Given single-channel speech signal, the tokenizer compresses it to the output tokens "iie {1...., Coode 12xxdu, where is the i-th RVQ level of to- ken, dusk is the length of the token sequence, respectively. The number of RVQ layers 12 with a codebook size Goode = 1, 024 in each layer.
Fig. illustrates the overview of DEmoFace. The MM- DiT comprises N blocks for conditional information injec- tion and 12 linear heads for concrete score prediction. The masked tokens 2, -1-1--12 obtained via speech tokenization and forward diffusion, with face and text conditioners form- ing the condition set Meanwhile, identity and emotion conditions with time are injected through adaptive layer normalization (AdaLN) (Peebles & Xie, 2023), and the text condition is injected with cross-attention. During train- ing, we propose a curriculum learning algorithm, which first inputs low-level tokens ar." "1-'" and refines them by adding high-level token x²' progressively. During sampling, we utilize an Euler sampler with our EPFG to iteratively refine the generated tokens, while a codec decoder recon- structs the waveform. Notably, when ground truth speech provided during training, the reference features Cgcle are extracted from the GE2E (Wan et al., 2018) to guide identity customization. During inference, aligned face encoder to extract the Gu instead of Cgc2e-
RVQ token diffusion process. Given the hicrarchical struc- ture of RVQ tokens, following the previous diffusion pro- cess (Lou al., 2024), we randomly corrupt each level token 2 timestep t. Specifically, tokens 2 from the codec encoder according to the cur- riculum training stage, where denotes the max level for the current input. We then conduct the diffusion process as defined in Equation (I) for x where I <
first extract input
use the cross-modal
Conditioners. For face conditioner, as presented Fig. 2, we build identity encoder and emotion encoder to learn identity embedding Cid and emotion embedding Cemo, re- spectively. Specifically, we first employ composite iden- tity embedding by introducing two face recognition models ArcFace (Deng et al., 2022) and FaceNet (Schroff et al., 2015), then utilize a multilayer perceptron (MLP) for trans- formation and shape alignment. To precisely model the high-fidelity vocal style associated with the face, we ex- tract the speech speaker embedding GzeZe from the speaker
Next, we detail the key components in DEmoFace.
4.3. Conditional Concrete Score Modeling
For the concrete score 89 modeling, first define the to- kenization and forward diffusion processes, followed by description of the conditioners and architecture, and con- clude with the modulation of the concrete score using EPFG.

---

Emotional Face-to-Speech
recognition model GE2E (Wan et al., 2018), and make Cu aligned with the Cyez across modalities using cosine sim- ilarity, LI, and L2 losses, detailed in Sec. 4.4. For the emotion embedding, we employ a strong facial expression recognition model Poster2 (Mao et al., 2023). Since the continuous emotion embedding of backbone is insufficient for decoupling identity information (Liu et al., 2024), we leverage the predicted label and learnable embedding layer to learn identity-agnone agnostic embedding Cemo-
Instead of sampling from 8g (a, 1, we can utilize tem- perature sampling (Kingma & Dhariwal, 2018; Mehta et al., 2023) for more controllable generated outputs by introduc- ing s6()) (x, c) = s0 f)2 Wy denotes the guidance scale. However, this compositional guidance lacks interactions among local conditions and struggles to guide sampling with a global consistent direc- tion. Inspired by the formulation of the HBM, the score can also be formulated as (z).;, ~ where Z is the normalizing constant, and fa is the energy function. Hence, we can associate compositional and joint conditions by summing up the energy functions, and finally obtain the modulated score by multiplying both scores:
s2 I by (u,
where
k==
(a( efo(e)
efa(e)
Pe (a c. cTuteT
For text conditioner, we introduce a text encoder to learn text embedding Cext. Specifically, raw text is preprocessed into an International Phonetic Alphabet (IPA) phone se- quence using& a standard IPA phonemizer. Next, embed- ding is extracted from a pre-trained text-speech encoder SpeechTS (Ao et al., 2022), and is then subsequently pro- jected into the hidden state via an MLP.
K
(w) (x, t) =sa (2, t)2 8g
(z.t.wla 2 (2.t);
20 (2,t,c)s
(5)
(2,t),
Joint
Multimodal DIT. We propose the Multimodal DiT (MM- DIT), which differs from DiT (Peebles & Xie, 2023) in three aspects. (1) Input, the masked speech tokens x;;-0 at timestep / are fed to embedding layers and subsequently averaged to serve as the input. (2) Conditioning, to cus- tomize face-style speech generation, we concatenate Ga and Cemo along with the timestep embedding. is passed through an MLP to inject the global face-style condition. The MLP aims to regress the scale and shift parameters 7, 3i, a2, 72. fo2 for the AdaLN. Additionally, to learn face-style linguistic expressions, we apply cross-attention with rotary position embeddings (Su et al., 2024) enabling dynamic alignment with text Ceet- (3) Output, we incorpo- rate 12 linear heads including a combination of AdaLN and linear layer to predict concrete scores for each RVQ level.
Compositional
where c = {Caa, Cemo: Gua),, we controls the scale of guid- ance strength for the joint injection of all conditions, while wi for S ls S K is assigned to each independent attribute. Please refer to Appendix C for detailed derivation.
4.4. Curriculum-based Training and Inference
Training. Curriculum learning aims to progressively train the model from simple to hard tasks, with the key challenge of identifying samples varying in difficulty. Previous studies show that neural networks prioritize low-frequency informa- tion first (Rahaman et al., 2019). Fig. 5(a) shows different frequency distributions across RVQ levels, with low-level features exhibiting fww-frequency patterns. Therefore, we reveal curriculum learning for RVQ-based tokens from the frequency perspective. As shown in Fig. 2, we gradually introduce higher-level tokens x; every 3 epochs, starting from previous low-level (i.e. low-frequency) tokens x; - to high-level tokens ..! facilitating effective training.
Enhanced predictor-free guidance. Several guidance tricks can boost sampling quality for the conditional gen- eration, such as predictor-free guidance (PFG) (Nisonoff et al., 2024; Ho & Salimans, 2021). However, given K conditions e = fci,...,,c cx) ). these guidance methods are not readily amenable to nulti-conditional scenarios (Liu et al., 2022). From the perspective of Energy-Base -Baed Models (EBMs), we propose an Enhanced PFG (EPFG) enhancing the efficient response to global condition while facilitating the decoupling of local conditions.
Furthermore, the training procedure contains two stages: concrete score prediction and identity feature alignment. In the concrete score prediction, the training objective is the multi-level DSE loss based on Equation (3) with the sum across current ri RVQ levels as Licore For conducting EPFG in Equa-
E Loseex",, tion (5), we randomly set Ø with 10% probability for each condition, and enforce all conditions set to Ø for 10% samples. For the cross-modal alignment, we use co- sine similarity, LI, and L2 losses to align the visual iden- tity vectors with Cgeze- With these compositional losses, the training objective for the face encoder is as Latign - --cos(Cia- Cgete) + +L1 L1(ca,, Cgeze) + ++1((ce) Cca, Cgoze). Notably, to avoid information degradation with reacher-studen dis- tillation, we directly train MM-DiT with ground truth Cge2e and use the aligned identity embedding eu during inference.
Specifically, to simplify the notation, we define 2f, -i- At as x, P. The key of conditional sampling process is to estimate the concrete score &o (z,t,c); ~ p(0) linked p(a)
to the transition probability. Using Bayes rule, we can obtain compositional concrete score &o (x, 1, c)2 from p(z,-1 = i x x, c) based on Equation (2):
K
so to(o,t,ck)a
8o (I,t,c) c)22 = so 2)
II
(4)
so tz
5

---

Emotional Face-to-Speech
Methods
Audio
Visual
EmoSimt
SpkSimt
RMSE4
MCDJ
WER(%))
Ground Truth
1.0000
1.0000
0.00
0.0000
10.82
Acoustic-guided Speech Generation
EmoSpeech (Diatlova & Shutov, 2023)
/
0.7667
0.5677
114.70
7.1328
29.59
FastSpeech2 (Ren et al., 2021)
/
/
0.7010
0.5217
115.97
7.3461
29.49
V2C-Net (Cong et al., 2023)
V
0.6788
0.5773
115.55
6.8901
29.54
HPM (Chen et al., 2022)
/
0.6817
0,4404
97.19
7.7614
77.31
StyleDubber (Cong et al., 2024b)
0.6742
0,4753
103.59
7.4497
43.14
DEMOFACE" (Ours)
0.7921
0.7990
94.68
6.5505
19.72
Visual-guided Speech Generation Face- TTe-TTS (Lee et al., 2023)
X
0.5230
0.1968
118.96
8.4649
17.47
DEmoFace (Ours)
X
0.6965
0.6679
101.18
6.8601
20.78
Table I: Speech quantitative results. The Audio and Visual indicate whether specific modality conditions are used for speech generation guidance. 1 (L) means the higher (lower) value hetter. We bold the best-performing method. Notably, the denotes that we use the speech condition Cgelo, rather than the face condition Gj to guide identity conditioning.
Inference. During inference, we introduce a frame-level du- ration predictor to estimate speech durations, initializing the input length dua- Then the reverse process is executed with Euler sampling (Lou et al., 2024) and EPFG with 96 steps. The details of the duration predictor refer to Appendix E.1.
Cepstral Distortion (MCD) (Chen al., 2022) to assess discrepancies between generated and target speech. Addi- tionally, the Word Error Rate (WER) (Wang et al., 2018; Radford et al., 2023) is used to gauge intelligibility. For the expressiveness, we calculate cosine similarity metrics based on emotion embeddings (Ma et al., 2024) and d-vectors to assess emotion similarity (EmoSim) and speaker identity similarity (SpkSim), as well the Root Mean Square Error (RMSE) for FO (Hayashi et al., 2017).
5. Experimental Results
5.1. Experimental Setups
Datasets. All our models are pre-trained on three datasets with pairs of face video and speech: RAVDESS (Living- stone & Russo, 2018), MEAD (Wang et al., 2020; Gan et al., 2023), and MELD-FAIR (Carneiro al., 2023). For data pre-processing, we first resample the audio to 16 kHz, and apply a speech separation model SepFormer (Subakan et al., 2021) to enhance voice. Then, we introduce Whis- per (Radford al., 2023) filter non-aligned text-speech pairs. Then, all models are trained combination of all three datasets. The RAVDESS and MEAD of the combined one are randomly segmented into training, validation, and test sets without any speaker overlap. For the MELD-FAIR, we follow the original splits. Additionally, these datasets lack sufficient semantic units in real-world environments, making challenging to train TTS model. We incorporate 10-hour subset from LRS3 (Afouras al., 2018) for pre- training, allowing the model be comparable Face-TTS trained 400 hours of LRS3. Finally, the combined dataset comprises 31.33 hours of audio recordings and 26,767 ut- terances across 7 basic emotions (i.e. angry, disgust, fear, happy, neutral, sad, and surprised) and 953 speakers.
Implementation details. We implement DEmoFace based on DiT architecture (Peebles & Xie, 2023). We use a log- linear noise schedule (() (Lou et al., 2024) where the ex- pectation of the number of masked tokens is linear with t. During training, we use the AdamW optimizer (Loshchilov & Hutter, 2019) with learning rate of le-4, batch size 32. The total number iterations is 300k. During inference, we use the Euler sampler to conduct the reverse process with 96 steps. We the joint guidance scale wy 1.9, and compositional scales wi w2 1.0, w3 1.6. Please refer to Appendix E for more implementation details.
5.2. Quantitative Evaluation
For quantitative evaluation, we compare DEmoFace with previous state-of-the-art (SOTA) methods, categorized into two paradigms based on the type of guidance. The acoustic- guided methods customize identity using acoustic prompts, and drive emotion generation from either visual or acoustic cues, while the risual-guided methods aim to customize both identity and emotion only from visual conditions.
Evaluation metrics. For eF2S, we evaluate the genera- tion performance based on naturalness (i.e. speech quality) and expressiveness. For the naturalness, we employ Mel
Objective evaluation. As shown in Tab. 1, compared with Face-TTS, we achieve 17.35% and 47.11% 11% improvements in terms of EmoSim and SpkSim, reflecting the great abil-
6

---

Emotional Face-to-Speech
Ground Truth
(b) EmoSphech
Stbee
Face Prompt
Angry'
WM
MA
Face Prompt
with Fear
with 'Srppisee
Figure 3: Speech qualitative results. The red rectangles highlight key regions with acoustic differences or over-smoothing issues, and the red dotted circle shows similar FO contours with ground truth. Zoom in for more details.
ity to maintain voice-identity while enhancing consistency. For prosody modeling. we can estimate more precise FO contour with relative 14.95% gains, exhibiting more natural speech expressiveness. The MCD improved by relative 18. 18.96% indicating minimal acoustic difference with the target speech. While Face-TTS achieves a better WER by utilizing over 10 times the data, DEmoFace significantly improves naturalness and consistency with fewer data.
Methods
MOS t
MOS,a f
MOS f
EmoSpeech
2.30=0.19
2.78+0.13 2.93+0.09
Face-TTS
2.28=0.09
2.67
2.75++0.09
DEmoFace (Ours)
8.17-0.18
0.10
3.26
Table 2: Subjective evaluation on speech naturalness, and similarity of identity (i.e. timbre) and emotion, compared with acoustic- and visual-guided methods.
Notably, we observe that the visual-guided DEmoFace even outperforms the cooustic-guided methods, which are the most efficient for speech generation by leveraging isomor- phic features. It demonstrates that DEmoFace bridges the cross-modal gap using only heterogeneous face features. Furthermore, we introduce the croustic-guides DEMO- FACE* replacing face condition Ga with speech condition Cgc2o, which gains greater improvements than other acoustic- guided methods in all metries by a large margin.
Notably, it achieves a 38% relative improvement in MOS confirming its superiority in generating high-quality speech. Furthermore, compared to the acoustic-guided EmoSpeech, DEmoFace yields higher MOS and MOS,a scores, re- flecting its ability to generate speech with greater emotional and identity consistency.
5.3. Qualitative Results
Subjective evaluation. We further conduct the subjective evaluation with 50 participants, to compare DEmoFace with SOTA methods (Diatlova & Shutov, 2023; Lee et al., 2023). Specifically, we introduce three metrics based on five mean opinion scores (MOS), each rated on scale from I to 5 in 0.5 increments. Among them, MOS evaluates speech naturalness and generation quality, while MOS,s, and MOSemo assess the perceived similarity of timbre and emotional prosody, respectively. We randomly generate 15 utterances from the test set for subjective evaluation.
Qualitative comparisons. As shown in Fig. 3, from mel- spectrogram in the first row, we observe severe temporal in- consistencies and over-smoothing issues in Face-TTS, Emo- Speech, and V2C-Net, which result in misaligned speech durations and a significant degradation in generation qual- ity. Furthermore, the FO curves in the third row indicate that these baselines produce pitch contours that deviate markedly from the ground truth (GT), reflecting mismatches in prosodic features such pitch height, emotion, and in- tonational patterns. In contrast, our DEmoFace produces speeches that are more closely aligned with the GT in both spectrogram structure and FO dynamics, highlighting the benefits of our enhanced multi-condition conditional generation and dynamic synchronization capabilities.
The scoring results of the user study, summarized with 95% confidence intervals in Tab. 2, demonstrate that DEmoFace has clear advantage over SOTA methods all metrics.

---

Emotional Face-to-Speech
DEmoFace (Ours)
Ground Truth
DEmoFace (Ours)
DEmoFace wo Ceno
Face-TTS
DEmoFace wa Cit
Face-TTS
(a) Emotion Distribution Visualization
(b) Identity Distribution Visualization
Figure 4: t-SNE visualization based emotion2vec (i.e. speech emotion features) and d-vector (i.e. speech identity features) representations. Each point represents speech sample, and colors indicate different emotions or identities. Points in close distance indicate higher similarity, while overlapping points suggest potential confusion. We compare our DEmoFace with real speech, an ablation without emotional or identity conditioning, and Face-TTS baseline. Results show that we achieve better clustering than other methods with great ability in emotional and identity generation.
Visualization of speaker embeddings. To explore the di- versity of emotional expressions and speaker identities in the generated speech, Maaten & Hinton, 2008) to visualize the distribution of emo- tion2vec features and d-vector features extracted from the synthesized speech. As shown in Fig. 4, ablating the emo- tion condition Cemo or speaker condition Ca leads to wrong emotion or speaker clustering. The feature overlap implies that the model may struggle to generate speech with clearly distinguishable identity and affective characteristics, validat- ing the effectiveness of our proposed emotional and identity conditioning. Furthermore, Face-TTS exhibits noticeable wrong mixing among different emotions and speakers, sug- gesting that it fails to perceive speaker timbre and emotional prosody effectively. In contrast, our DEmoFace effectively clusters the speech embeddings from the same speaker and emotion, while maintaining stronger speaker- and emotion- discriminative properties, indicating the ability to synthesize diverse vocal styles and faithful speeches from visual cues.
Vars
EmoSim
SpkSimt
RMSE
MCDJ
WER,
(a)
0.69
0.41
109.62
8.25
22.35
utilize the t-SNE technique (Van der
(b)
0.65
104.92
7.29
21.35
(c)
0.67
0.66
104.41
7.24
27.52
(d)
0.69
0.58
115.30
8.33
39.06
(c) 0.67
0.63
106.67
7.31
40.13
Ours
0.70
0.67
101.18
6.86
20.78
Table 3:
Ablation studies. "Vars" refers to the ablation variants, with (a) to (e) indicating the removal of identity condition Ca. emotion condition Cemo. curriculum learning, identity feature alignment, and EPFG, respectively.
Ablation curriculum learning. To demonstrate the ef- fectiveness of curriculum learning, we input all RVQ-level tokens during the whole training process as the variant (c) in Tab. 3. Based on the fact that RVQ Codec preserves semantic information in low-level tokens while retaining acoustic details in high-level tokens (Nishimura al., 2024), Fig. shows: (1) we achieve better WER and EmoSim than baseline during early training, as prioritized low-level leam- ing effectively captures low-level information; (2) SpkSim initially lags due to unseen high-level tokens but improves as they are progressively introduced. The results highlight the effectiveness of curriculum learning.
5.4. Ablation Studies
Ablation on multimodal conditioning. As the variant (a)
in Tab. 3, the results demonstrate that incorporating Gaa sig- nificantly enhances speaker similarity with 26% SpkSim improvement and speech naturalness with 17% MCD rel- ative improvement, while maintaining stable EmoSim and WER scores. Similarly, the inclusion of Camo leads to im- proved emotional prosody in EmoSim and RMSE without compromising intelligibility. These results indicate that DE- moFace can successfully decouple different conditioning signals from others through our EPFG algorithm.
Ablation on identity alignment. Due to the heterogeneous differences between speaker features extracted from vision and speech, achieving accurate cross-modal identity cus- tomization remains challenging. As shown in Tab. (d), removing identity alignment leads to a significant 9% drop in SpkSim, indicating clear misalignment between real
8

---

Emotional Face-to-Speech
[n] RVQ-Level Feature Distribution
(b) Word Error
(c) Emotion Similarity
Similarity
Oan
Dutu
335
Veldditio Murtic
2.2
0.5
032
Prequency
Reletive Training Cast
Radedive Training Cost
Relatiee Tralning Cast
Figure 5: Ablation study on curriculum learning. (a) Feature distribution across RVQ levels, with low-level features showing low-frequee wwwfrrequecyy patterns. (b)-(d) For the baseline without curriculum learning, we vary the number of training epochs compared with three metrics on the validation set. The effect is evident for WER and EmoSim while slight on SpkSim.
Grid
Serach
Bo Colputational Efficiency
performance with efficiency, we utilize 96 steps in this paper. Furthermore, we evaluate inference Real- Time Factor (RTF) to measure the time (in seconds) required to synthesize one second of speech. Specifically, we utilize a mini-batch size of I and 32 with the Number of Function Evaluations (NFE) fixed at 32. When using a batch size of 1, our method achieves an RTF of 0.49, demonstrating the potential for real-time speech synthesis approximately twice as fast as real-time. With batch size of 32, the RTF further improves to 0313, highlighting the efficiency of DEmoFace.
10
Cowkk
200.11
Figure 6: (a) Parameters grid search for the EPFG, with axes as two-parameter combinations, colors as normalized performance. (b) Effect on sampling steps.
6. Conclusion
timbre and visual identity embedding. Moreover, the pres- ence of noise in visual features further disrupts the genera- tion process, resulting in degraded performance across all evaluation metrics.
We propose DEmoFace, the first RVQ-based discrete dif- fusion framework for eF2S with high diversity and quality, serving as foundation for future research in multimodal personalized TTS systems. Specifically, the MM-DiT dy- namically aligns speech and text prompts, and customizes face-style linguistic expressions. Meanwhile, we introduce a coarse-to- coarse-to-fine fine curriculum learning strategy to enhance train- ing efficiency. Moreover, to boost efficient response to the global condition while facilitating the decoupling of multi- ple local conditions, we develop an enhanced predictor-frt free guidance to achieve multi-conditional generation of eF2S. Extensive experimental results show that DEmoFace out- performs existing methods and effectively synthesizes the speech solely from the visual cues.
Ablation on EPFG. The EPFG enables effective gener- alization across combinations of multiple conditions, even those unseen during pre-training. From Tab. (e), we ob- serve that EmoSim, SpkSim, and WER metrics have de- graded when all conditions are treated as unified one, showing the incorporation of EPFG can significantly en- hance multi-conditional generation quality. Furthermore, conduct grid search for all parameters of the EPFG on the validation set. As shown in Fig. 6(a), axes denote two parameter combinations (i.e. (wo E [1.0, 2.0], w1 E [1.0, 1.4)), (w1 E [1.0, 1.4], W2 E [1.0, 2.0)), the color of the grid indicates normalized performance score, and the red rectangle marking the final parameter combination we select. We observe performance degradation (i.e. the light area) with complex entanglement, as the unconditional score dominates with low guidance scales across conditions, fur- ther demonstrating the effectiveness of the EPFG.
Limitation. Despite achieving advanced performance, we struggle to precisely reconstruct true voice due to visual-voice biases within the dataset, tending to produce average- sounding speech. Furthermore, the limited size of the dataset poses significant challenge to achieving accurate pronunciation. We will incorporate larger datasets to improve the diversity and intelligibility of the model.
Ablation on inference efficiency. To explore the effective- ness of sampling steps, we first normalize each metric to [0, 1] and obtain the average performance. As shown in Fig. 6(b), the performance improves with more steps, satu- rating at 32 steps. It demonstrates that DEmoFace achieves acceptable generation quality with just 32 steps. To balance
A Acknowledgment
This work was supported in part by STI2030-Major Projects (No. 2021ZD0200204), National Natural Science Founda- tion of China (No. 62471148), and Shanghai Center for Brain Science and Brain-inspired Technology.

---

Emotional Face-to-Speech
Impact Statement
Cong, G., Qi, Y., Li, L.., Beheshti, A., Zhang, Z., den Hengel, A., Yang, M., Yan, C., and Huang. Q. Style- Dubber: Towards multi-scale style learning for movie dubbing, In Findings Proc. Annu. Meeting Assoc. Com- pul. Linguisties, pp. 6767-6779, 2024h.
Given the privacy of identity information in face and speech, ensuring compliance with ethical standards and privacy rules essential safeguard individual rights and prevent poten- tial misuse. We stress the necessity of consent agreements for using published models, to ensure responsible applica- tion while respecting individual privacy and rights.
Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Trans. Mach. Learn. Res., 2023, 2023.
References
Deng, J., Guo, J., Yang. J., Xue, N., Kotsia, I., and Zafeiriou, S. Areface: Additive angular margin loss for deep face recognition. IEEE Trans. Pattern Anal. Mach. Intell., 44 (10):592--979,, 2022.
Afouras, T., Chung. J. S., and Zisserman, A. LRS3-TED: a large-scale dataset for visual speech recognition. CoRR, 1091009990 2018.
Ao, J., Wang, R., Zhou, L., Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q.. Zhang, Y. Wei, Z., Qian, Y., Li, J., and Wei, F. SpeechTS: Unified-modal encoder-decoder pre-training for spoken language processing. In Proc. Annu. Meeting Assoc. Comput. Linguistics, pp. 5723- 5738. Association for Computational Linguistics, 2022.
Diatlova, D. and Shutov, V. EmoSpeech: Guiding Fast- Speech2 towards emotional text to speech. In ISCA Speech Synthesis Worksh., pp. 106-112, 2023.
Gan, Y., Yang, Z., Yue, X., Sun, L.,, and Yang, Y. Efficient emotional adaptation for audio-driven talking- gen- eration. Int. Conf. Comput. Vis., pp. 22577-22588 2023.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in discrete state-spaces. In Adv. Neural Inform. Process. Syst., pp. 17981-17993, 2021.
Geng, C., Han, T., Jiang, P., Zhang, H., Chen, J., Hauberg, S., and Li, Improving adversarial energy-basee model via diffusion process. In Int. Conf. on Mach. Learn., 2024.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Puttern Recog,, pp. 22563- 22575, 2023.
Goto, S., Onishi, K., Saito, Y., Tachibana, K., and Mori, K. Face2Speech: Towards multi-speaker text-to-speech synthesis using an embedding vector predicted from face image. In Annu. Conf. Int. Speech Commun. Assoc.,
pp. 1321-1325, 2020.
Carneiro, H. C. C., Weber, C., and Wermter, S. Whose emotion matters? speaking activity localisation without prior knowledge. Neurocomputing, 145:12 2023.
Guo, Q., Ma, C., Jiang, Y., Yuan, Z.,, Yu, Y., and Luo, P. EGC: Image generation and classification via diffu- sion energy-based model. In Int. Conf. Comput. Vis., PP. 22895-22905, 2023a.
Carreira, J. and Zisserman, A. Quo vadis, action recogni- tion? A new model and the kinetics dataset. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 7724-4733 2017. Chen, Q. Tan, M., Qi, Y., Zhou, J., Li, Y., and Wu, Q. V2C: Visual voice cloning. In IEEE Conf. Comput. Vis. Pattern Recog,, pp. 21210-21219, 2022.
Guo, Y., Du, C., Chen, X., and Yu, K. Emodiff: Inten- sity controllable emotional text-to-speech with soft-label guidance. In IEEE Conf. Acoust. Speech Signal Process., pp. 1-5, 2023b.
Hayashi, T., Tamamori, A., Kobayashi, K., Takeda, K., and Toda, T. An investigation of multi-speaker training for wavenet vocoder. In IEEE Autom. Speech Recognit. Understanding Worksh., pp. 712-718, 2017.
Chung, J. S., Nagrani, A., and Zisserman, A. VoxCeleb2:
Deep speaker recognition. In Yegnanarayana, B. (ed.), Annu. Conf. Int. Speech Commun. Assoc., pp. 1086-1090, 2018.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. Adv. Neural Inform. Process. Syst. Worksh, pp. 1-14, 2021.
Cong, G., Li, Qi Y., Zha, Z., Wu, Q. Wang, W., Jiang, B., Yang, M., and Huang, Q. Learning dub movies via hierarchical prosody models. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 14687-14697, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. Ady. Neural Inform. Process. Syst., 2020.
Cong, G., Pan, J., Li, L., Qi Y., Peng, Y., van den Hengel, A., Yang, J., and Huang, Q. Emodubber: Towards high quality and emotion controllable movie dubbing. CoRR, 2412.08988, 2024a.
Ito, K. and Johnson, L. The Ij speech dataset. https:// // weithiton com Lt///JJJJppe MWJ-Speech-Daee wat.echtttttttttttaaaaaaa 2017.
10

---

Emotional Face-to-Speech
Jang, Y., Kim, J., Ahn, J., Kwak, D., Yang, H., Ju, Y., Kim, I,, Kim, B., and Chung, S. Faces that speak: Jointly synthesising talking face and speech from text. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 8818-8828, 2024.
Livingstone, S. R. and Russo, A. The ryerson audio-visual database of emotional speech and song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in north american english. PLOS ONE, 13(5):e0166391, 2018.
Kang, M., Han, W., and Yang, E. Pace-stylespeech: Im- proved face-to-voice latent mapping for natural zero- shot speech synthesis from a face image. CoRR, abs/2311.0552 2023.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. In Int. Conf. Learn. Represent, 2019.
Lou, A., Meng, C.,, and Ermon, S. Discrete diffusion mod- eling by estimating the ratios of the data distribution. In Int. Conf. on Mach. Learn., 2024.
Kelly, F. P. Reversibility and stochastic networks. Cam- bridge University Press, 2011.
Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Gir- gin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., and Zeghidour, N. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Trans. Assoc. Comput. Linguistics, 11:1703-1718, 2023.
Ma. Z., Zheng, Z., Ye, J., Li, J., Gao, Z., Zhang. S., and Chen, X. emotion2vec: Self-supervised pre- vre-training for speech emotion representation. In Findings Proc. Annu. Meeting Assoc. Comput. Linguistics, pp. 15747-15760. Association for Computational I Linguisties, 2024.
Kim, T. and Bengio, Y. Deep directed generative mod- els with energy-based probability estimation. CoRR, htss/106603439,, 2016.
Mao, J., Xu, R., Yin, X., Chang, Y., Nie, B., and Huang, A. POSTER V2: A simpler and stronger facial expression recognition network. CoRR, abs/2301.12149, 2023.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Int. Conf. Learn. Represent., 2015.
Mehta, S., Kirkland, A., Lameris, H., Beskow, Székely, É., and Henter, G. E. Overflow: Putting flows top of neural transducers for better TTS. In Annu. Conf. Int. Speech Commun. Assoc pp. 4279-44--844 4283, 2023.
Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible Ixl convolutions. In Adv. Neural Inform. Process. Syst., pp. 10236-10245, 2018.
Meng, C., Choi, K., Song. J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data. In Adv. Neural Inform. Process. Syst., 2022.
Lee, J., Chung, J. S., and Chung, S. Imaginary voice: Face- styled diffusion model for text-to-speech. In IEEE Conf. Acoust. Speech Signal Process., pp. 1-5, 2023.
Nishimura, Y., Hirose, T., Ohi, M., Nakayama, H., and Inoue, N. HALL-E: hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis. CoRR, abs/2410.04336 2024.
Lee, J., Oh, Y., Hwang, I,, and Lee, K. Hear your face: Face-based voice conversion with FO estimation. CoRR, abs/2408.0990 09802, 2024.
Li, X., Cheng, Z., He, J., Peng, X., and Hauptmann, A. G. MM-TTS: A unified framework for multimodal, prompt- induced emotional text-to-speech synthesis. CoRR, abs/2404. 183988 2024.
Nisonoff, H., Xiong, J., Allenspach, S., and Listgarten, J. Unlocking guidance for discrete state-space diffusion and flow models. CoRR, 0as/2406.01177) 2024.
Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z.,, and Li, C. Your absorbing discrete diffusion secretly mod- els the conditional distributions of clean data. CoRR, abs/2406.03736, 2024.
Li, Y. A., Han, C., Raghavan, S., Mischler, G., and Mesgarani, N. StyleTTS 2: Towards human-level text- to-speech through style diffusion and adversarial training with large speech language models. Adv. Neural Inform. Process. Syst., 2023.
Paris, T., Kim, J., and Davis, C. Visual form predictions facilitate auditory processing at the nl. Neuroscience, 343:157-164, 2017.
Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable diffu- sion models. In Eur. Conf. Comput. Vis., volume 13677, pp. 423 439, 2022.
Peebles, W. and Xie, Scalable diffusion models with transformers. Int. Conf. Comput. Vis., pp. 4172-4182, 2023.
Liu, R., Ma, B., Zhang, W., Hu, Z., Fan, C., Lv, T., Ding, Y., and Cheng, X. Towards simultancous and granular expressin control in personalized face gener- ation. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 21 2114-2123, 2024.
Plüster, B., Weber, C., Qu, L., and Wermter, S. Hearing faces: Target speaker text-to-speech synthesis from a face. In IEEE Autom. Speech Recognit. Understanding Worksh., pp. 757-764, 2021.

---

Emotional Face-to-Speech
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In Int. Conf. on Mach. Learn., volume 202, pp. 2-492228518, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Attention is all you need. In Adv. Neural Inform. Process. Syst., pp. 5998- 6008, 2017.
Rahaman, N., Baratin, A., Arpit, D., Draxler, F. Lin, M., Hamprecht, F. A., Bengio, Y., and Courville, A. C.C. On the spectral bias of neural networks. In Int. Conf. on Mach. Learn., volume 97, pp. 5301-5310, 2019.
Wan, L., Wang. Q., Papir, A., and López-Moreno, I. Gener- alized end-to-end loss for speaker verification. In IEEE Conf. Acoust. Speech Signal Process., pp. 4879-4883, 2018.
Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T. FastSpeech 2: Fast and high-quality end-to-end text to speech. In Ins. Conf. Learn. Represent. 2021.
Wang. C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, I.., Zhao, S., and Wei, F. Neural codec language models are zero-shot text to speech synthesizers. CoRR, abs/2301.02) 111111,,,11,,,,11111111111111111 2023.
Ruan, L., Ma, Y., Yang, H., He, H., Liu, B., Fu, J., Yuan, N. J., Jin, Q., and Guo, B. MM-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In JEEE Conf. Comput. Vis. Pattern Recog., pp. 10219-10228, 2023.
Wang, K., Wu, Q., Song, L., Yang, Z., Wu, W., Qian, C.,, He, R., Qiao, Y., and Loy, C. C. MEAD: A large-scale audio- visual dataset for emotional talking- facc generation. In Eur. Conf. Comput. Vis., volume 12366 of Lecture Notes in Computer Science, pp. 700-717, 2020.
Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 815-823, 2015.
Wang. Y., Stanton, D., Zhang. Y., Skerry-Ryan R. J., Batten- berg, E.,, Shor, J., Xiao, Y., Jia, Y., Ren, F., and Saurous, R. A. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In Ins. Conf. on Mach. Learn., volume 80, pp. 5167-5176, 2018.
Shen, K., Ju, Z., Tan, X., Liu, E., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. NaturalSpeech 2: Latent diffu- sion models are natural and zero-shot speech and singing synthesizers. In Int. Conf. Learn. Represent.. 2024.
Wang, Y., Zhan, H., Liu, I.., Zeng, R., Guo, H., Zheng, J., Zhang, Q., Zhang, S., and Wu, Z. MaskGCT: Zero-shot text-to-speech with masked generative codec transformer. CoRR, abs/2409.007550 2024.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In Int. Conf. Learn. Represent., 2021.
Wen, X., Ye. J., Luo, Y., Xu, Y., Wang. X., Wu, C., and Liu, K. Ctl-mtnet: A novel capsnet and transfer Marning-based mixed task net for single-corpus and cross-corpus speech emotion recognition. In Int. Joint Conf. Artif. Intell., pp. 2305-2311, 2022.
Su. J., Ahmed, M. H. M., Lu, Y.. Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 1880637 2024.
Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., and Zhong, J. Attention is all you need in speech separation. In IEEE Conf. Acoust. Speech Signal Process., pp. 21-25, 2021.
Wen, X., Liu, K., Luo, Y., Ye, J., and Chen, L. Twacapsnet: a capsule network with two-way attention mechanism for speech emotion recognition. Soft Comput., 28(15-16): 8701-8713, 2024.
Sun, H., Yu, L.. Dai, B., Schuurmans, D., and Dai, H. Score- based continuous-time discrete diffusion models. In Int. Conf. Learn. Represent., 2023.
Wu, Z., Li, Q., Liu, S., and Yang, Q. DCTTS: discrete dif- fusion model with contrastive learning for text-to-speech generation. In IEEE Conf. Acoust. Speech Signal Process., pp. 11336-1 1 1340. IEEE, 2024.
Haitelbaum-Sweall R. and Fostick, L.. Auditory and visual information in speech perception: A developmental per- spective. Clinical linguistics phonetics, 30(7):531-544 2016.
Xue, R., Liu, Y., He, L., Tan, X., Liu, L., Lin, E., and Zhao, S. Foundationtts: Text-to-speech for ASR customization with generative language model. CoRR, abs/2303.02939 2023.
Van der Maaten, L. and Hinton, G. Visualizing data using t-SNE. Mach. Learn. Res., 9(11), 2008
Yang, D.,, Yu, J.. Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text- to-sound generation. IEEE ACM Trans. Audio Speech Lang. Process, 31:1720-1733,, 2023.
Vasuki, A. and Vanathi, P. A review of vector quantization techniques. IEEE Potentials, 25(4):39-477 2006.
12

---

Emotional Face-to-Speech
Ye, and Shan, H. Shushing! let's imagine authentic speech from the silent video. CoRR, abs/2503.14928, 2025.
Ye, J., Wen, X., Wang, X., Xu, Y., Luo, Y., Wu, C., Chen, L., and Liu, K. Gm-tcnet: Gated multi-scale temporal convolutional network using emotion causality for speech emotion recognition. Speech Commun., 145:21-35, 2022.
Ye, J., Wei, Y., Wen, X., Ma, C., Huang, 7., Liu, K., and Shan, H. Emo-dna: Emotion decoupling and alignment learning for cross-corpus speech emotion recognition. In ACM Int. Conf. Multimedia, pp. 5956-5965, 2023a.
Ye, J., Wen, X., Wei, Y., Xu, Y., Liu, K., and Shan, H. Tem- poral modcling matters: novel temporal emotional mod- eling approach for speech emotion recognition. In JEEE Conf. Acoust. Speech Signal Process., pp. 1-5, 2023b.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. SoundStream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process., 30:495-5005 507,55077220 2022.
Zhang. X., Zhang, D., Li, S., Zhou, Y., and Qiu, Speech- Tokenizer: Unified speech tokenizer for speech language models. In Int. Conf. Pattern Recog., 2024.
Zheng, Y., Tu, W., Xiao, L.,, and Xu, X. Srcodec: Split- residual vector quantization for neural speech codec. IEFE Conf. Acoust. Speech Signal Process., pp. 451-455, 2024.
13

---

Emotional Sace-to-Speech
Appendix
This appendix provides the following extra contents:
Appendix A shows detailed notations and definitions;
Appendix B provides preliminary of the discrete diffusion model;
Appendix presents a detailed derivation of enhanced predictor free guidance;
Appendix D includes the statics of datasets used in this paper;
Appendix E supplements the experimental details of our DEmoFace and each baseline method;
Appendix F contains extra experimental results; and
Appendix G incorporates the details of subjective evaluation.
A. Detailed Notations and Definitions
Tab. A-I provides the notations and definitions of variables used in the paper.
Notation
Definition
Vector variables representing sequence of tokens.
on x², Rr,
2
Scalar variables representing token or state in discrete diffusion process. State space {1...., n of token sequence length and token dimension n. Diffusion forward matrix (reatrannition rate matrix) at t time.
xd
Q
Q:
Diffusion reverse matrix at time.
lok
Token-level transition rate matrix filled with absorbing state [MASK].
2
Kronecker delta function of two variables x, f. which is I if the variables equal, and 0 otherwise.
p
Probability distribution of the forward diffusion process characterized by Qt-
go
Probability distribution of the reverse diffusion process characterized by score model
P.o
Transition probability matrix from time 0 to time t.
Score network to estimate the ratio (i.e. concrete score) p(=i_ae) p(a))
Concrete score p(d ima) p(a, (Pa)
Ca
N(c)
Normalizing constant function of denoising score entropy loss.
Ti
i-th level of RVQ tokens, where 1 < 12. r denotes the current max level during our curriculum training. Codebook size.
Gende
dook
Length of the token sequence.
Condition set including Cia, Cemo; Gtext
01, 01,71,80 91, 81
Scale and shift parameters for the adaptive layer normalization.
Table A-1: Detailed Notations and Definitions
B. Preliminary: Discrete Diffusion Models
Continuous Diffusion Models (CDM) have been one of the most prominent and active areas in generative model- ing (Blattmann al., 2023; Li al., 2023; Ruan al., 2023). which has shown state-of-the-arr performance in various fields. However, for speech generation, the high-dimensional speech features and excessive diffusion steps lead high resource usage and inefficient inference, frustrating practical application. The fundamental way lies in compressing the speech feature space, such discrete space.
In recent years, Discrete diffusion models (DDM) have shown promise in language modeling (Austin et al., 2021; Meng et al., 2022; Lou et al., 2024; Ou et al., 2024), which are characterized by a forward and reverse process like continuous diffusion models (Song et al., 2021: Ho et al., 2020). Nevertheless, DDM has yet to be explored speech generation. In
14

---

Emotional Face-to-Speech
this study, we introduce the DDM for speech token generation. Below, outline the forward and reverse processes, along with the training objective.
Forward diffusion process. Given sequence of tokens ......." xd from state space of length like X = {1, (1,..... n) The continuous-time time discrete Markov chain at time is characterized by the diffusion matrix Qt E IR" xx" (i.e. transition rate matrix), follows:
p(a1+)al=+)) = Ouitann + (()) o(D)),
(A-1)
where æt denotes i-th element of wt, Qx 2t) is the (e =f) element Qz, denoting the transition rate from state ay state ri= at time and 8 is the Kronecker delta. Since the exponential size of Q2, existing works (Lou et al., 2024; Ou al., 2024) propose to assume dimensional independence, conducting dimension with the same token-level diffusion matrix Q! a(t) E R where (tt) is the noise schedule and Qiek is designed to diffuse towards an absorbing state [MASK] in this study. Then the forward equation is formulated as follows:
mie-dimensional diffusion process for each
P( =x) = exp (a(t) (1), mb))
(A-2)
where transition probability matrix P( == )(>م)=))) and cumulative noise at)) So (s)ds. There are two probabilities in the Prjo: 1 - e ( for replacing the current tokens with [MASK], e +(t) for keeping it unchanged, where the diffusion transition rate matrix Qtok is defined as:
0
1
0
-1
0
1
Qlik
(A-3)
-1
1 0
Therefore, we can parallel sample the corrupted sequence æ, directly from Do in one step. During the inference, we start from 2r filled with [MASK] tokens and iteratively sample new set of tokens Φ;-1 from (2;--11;;).
Reverse denoising process. As the transition rate matrix Q is known, the reverse process can be given by reverse p(ut and (x), and x/ At # o/.
transition rate matrix Q (Sun et al., 2023; Kelly, 2011). where Q ( xf)
p(+;)
or (z x5) - EzS,, (1) The reverse equation is formulated follows:
p(af-s(lat) = b +
(A-4)
p(a) ad (which is known as the concrete score (Lou et al., 2024; Meng et al., 2022) to p(xz)
where we can estimate the ratio measure the transition probability or closeness from a state of at time to state i' at time t At) of Q, by score network
se(2), f):;
pla] At)
p(x]) - So that the reverse matrix is parameterized to model the reverse process qo (
(i.e. parameterize the concrete score).
Training objective. Denoising score entropy (DSE) (Lou et al., 2024) is introduced to train the score network se:
(" Exe (ea)
Σ
Q Qi (+) sso (r):1) Cafet log 80 (1t)1) + N(capot) di,
(A-5)
2.f0.
p ( and a normalizing constant function N(c) == eloge - that ensures loss (
where the concrete score Catat non-negative. After training, we can replace the concrete score with the trained score network on Equation (A-4), conducting the sampling process.
15

---

Emotional Face-to-Speech
C. Derivation of Enhanced Predictor-free Guidance
For the discrete diffusion model, given the random variable value x; - = ... ...af from a state space of length d with the absorbing state [MASK] at time t, the unconditional probability distribution p(a/- laj) = ÓuÍ_ + Q1 Ar +)) +
of (t) during sampling Equation (A-4) shown. For the conditional probability distribution p( c), the key is to obtain the conditional transition rate matrix Q1 (x(:--2x::/1))
Firstly, following (Nisonoff' et al., 2024) to simplify the notation, we define ot as H2 and utilize the properties of the Kronecker
delta ô (i.e. the function is if the variables are equal, and 0 otherwise) to derive another form of the unconditional probability distribution p(a = 2 Z = z):
p( At 2|22 z) = -da-- - + = o(At)
= = + 85
61- (1, A + o(A))
(A-6)
òg (I, + +(1 (1- (2, + o(At).
Then, we utilize the formulation in Equation (A-6) and the Bayes rule to build the conditional probability distribution
p(21-21= = 2 xx= = x, c). combining predictive distribution p(c]ar) and unconditional distribution p(z = f.,re = x)) as:
p(21-11= = -//,z = xr c) = p(c/2=- At = 3, = x) x)p((x = 5 xt = x)
p(c|2x =x)
p(c/ = 2t x2 = ((((pp)((///)) (It-
= =6/2== = =x)
Σ p(c((1a 21 = x², = x)p( = =x'lare x))
(A-7)
[ō (1 (1+ (1,, + +(1 +11- 65. ( + o(A()]
plelue- -
Σ -:rii-::sr'....
Mela; A 2. [d (1+ Q: a) + +11- óg 1Q x) At + o(Ar))
where we use Equation (A-6) replace (Z x) in A-7 and define po(0, = p( 2, = x). We further simplify the formulation:
Pr (a.w)
[5 (1 + Qr (1, A1) + (1 8g Q1((, Q1(3,
P(I1-A1= = 2/2, x,
c)
Pe (1r0))
Σx x'' pele,.)
Pe [=
[62.. (1 (1+ Q (z, x) At) (1 QQ0
8z (1 (r( + polz,z)
PelG.a) (1 8z ) (0, AL (At))
(A-8)
(1 (2) + Ex'te Deld'
x At + ((A)
Pola,a)
60.2 () + Pol=,.)
Pola,a) (1 80 x
At At
+ f(At, x.
where f is function of At, and as At- + 0 we can use Taylor expansion of [+ R 1 1- ((A,,,,)) + o(AP):
(o() x))
p(z = 2/2 = xz c) = x (1 + (1(2
(1 82,z (2, x) x) [(A))]
Po(x, x)
pe(2",x) Q1 x)) At + +(A))
[1 Q((2,))
()(1,
(A-9)
x'
1-
Pe(x', x))
+ (1 - 6z.0) Pe( x)
(3' x)Ai
(,(t, + o(A))
(e(x,
Pe( x)
From the expression in Equation (A-6) and property of the Q (i.e. Q1(1,)) + Ex Q: xx) = 0) (Lou et al., 2024), we can derive our conditional transition rate matrix:
(2( = pc(2,z) x)
(=(x, (2,
(A-10)
16

---

Emotional Face-to-Speech
where can be deduced that the matrix also satisfies the same property Qe(2, + + Qcl2', 0. Therefore, rewrite Equation (A-9)
p -a/a, -I òz, (1
(1 (122
(A-11)
= + Q2 x
+ d(At).
Furthermore, to achieve predictor-free guidance (Ho & Salimans, 2021; Nisonoff et al., 2024), we use the Bayes rule to relive the dependence on any predictor/Classifier:
(((xx p(c| At 2, x))
Q (z)
p(c|5:--1 At = 81
plat- A: = 8/0;= = x. c.pppppeps Car =
pl At = x/., =
p(zí-A1 x 2/22- x, c)p(c Q.
p(#i-s: = f//x, = x)
(A-12)
p(u = 8/02 = x. p(ac-At = 2/ = x)
p1 = 1//2 v. c) p( = 2/122= = (ē,;z))
sa(x, 1) (.)
so 1)z
where utilize the concrete score so (z, t, c) in Equation (2) estimate the ratio like p(s--ar= Similar to previous methods, we can introduce guidance scale (i.e. guidance strength) as:
Qi ælc) ~ $6 (z, t, C)2 Q.(2,z)= 5g (x, 2)
5g (x, t, c). S (x,, t)
(sa(z, t) (i, x)) = 5% (x, t, c)2 Qu (2,
(A-13)
5q w4 (x, f)
where Qlak is the fixed diffusion transition rate matrix in Equation (A-3). Since {ci..... ct} contains & independent conditions, we can rewrite Equation (A-12) into multi-conditional form as:
p -,
(fc., cx) = Q, x))
II 1=1
p F4
I,
p(=1- = p( 2/2;
Q.(a, x))
II
((
x/ = I, p( -
i=1
(A-14)
(I,t,c))
Q)(2,z) x))
II
$g (x, t)a
Sg (x,t, Ci) (x, t)
sp (2 t)
Q @ (f, X).
Energy- Energy-Based Models (EBMs) (Guo et al., 2023a; Geng et al., 2024; Liu et al., 2022) are class of generative models and also known as non-normalized probabilistic models. Given speech token sequence æ and a learnable neural network fo. the probability distribution of EBM can be formulated as:
eo((
Po (x)
(A-15)
Z
where Z = e is normalizing constant, and fo is the energy function. Inspired by the formulation of the EBM, the score also be formulated as $z(z)² == Ps (=(a) e /Z
co where Zt, i Æı-Ai, Z is the normalizing constant, and fo is the energy function. As we typically define the energy function as sum of multiple terms (Kim Bengio, 2016), we can associate each term with the joint and compositional ones, and the final probability distribution expressed as a product of both. Hence, we can obtain the modulated score šg (I, t) by multiplying the compositional score and joint score (i.e. up the energy functions):
Po(x) /
(11) 89
(z, t)2 =so( t)
II
Wi (z.t.c)a (2,0))
(x)
(A-16)
(z,1),
Rm
Joint
Compositional
17

---

Emotional Face-to-Speech
where e (Cas, Cemo: h wo controls the scale of guidance strength for the joint injection of all conditions, while we for 1 S i & is assigned each independent attribute (i.e. identity, emotion, and semantics with k 3). Therefore, can rewrite Equation (A-11) as:
p( 8/5 x, c) Ōá mlu+Oo
o(At) 2e δg + 58 (z, Qi (2.z)
(A-17)
D. Datasets
D.1. Dataset Statistics
All our models are pre-trained on three datasets with pairs of face video and speech: RAVDESS (Livingstone & Russo, 2018), MEAD (Wang et al., 2020; Gan et al., 2023), and MELD-FAIR (Carneiro et al., 2023). The RAVDESS contains 1,440 English utterances voiced by 12 male and 12 female actors with eight different emotions. The MEAD is a talking-face video corpus featuring 60 actors and actresses talking with eight different emotions at three different intensity levels. The MELD-FAIR introduces a novel pre-pree proccesessing pipeline to noisy alignment issues of the MEAD (Wang et al., 2020) consisting of text-audio-video pairs extracted from the Friends TV series. Then, for the training, train all our models using a combination of all three datasets. The RAVDESS and MEAD of the combined one randomly segmented into training, validation, and test sets without any speaker overlap. In contrast, we follow the original splits of the MELD-FAID dataset with speaker overlap. Additionally, these datasets lack sufficient semantic units in real-world environments, making challenging to train a TTS model. We incorporate a 10-hour subset from LRS3 (Afouras et al., 2018) for pre-training, allowing the model to be comparable to Face-TTS trained on 400 hours of LRS3. Finally, the combined dataset comprises 31.33 hours of audio recordings and 26,767 utterances across 7 basic emotions (i.e. angry, disgust, fear, happy, neutral, sad, and surprised) and 953 speakers.
D.2. Data Preprocessing Details
For data rre-proeessing, considering the presence of non-primary speakers and background noise such as audience interactions in the recordings, we first resample the audio to a single-channel 16-bit at 16 kHz format, then apply Sep- Former (Subakan al., 2021), state-of-the-atr model in speech separation, isolate the primary speaker's 's audio and reduce noise from other voices. Then, we introduce an automatic speech recognition model Whisper (Radford et al., 2023) to filter non-aligned text-speech pairs (i.e. WER higher than 10%).
E. Model Details
E.1. Implementation Details of DEmoFace
Table A-2 shows more details about our DEmoFace. Firstly, for our multimodal diffusion transformer, it contains 12 MM-DIT blocks, with channel numbers 768, attention heads 12 for each block. We train the model using the AdamW optimizer (Loshchilov & Hutter, 2019) with B1 0.9, 32 0.999, a learning rate of le-4, batch size 32, and a 24GB NVIDIA RTX 4090 GPU. The total number of iterations is 300k. For a fair comparison, we do not perform any pre-training or fine-tuning on the test set. During inference, we use the Euler sampler with 96 steps following (Lou et al., 2024).
Secondly, we train our identity encoder achiving face-speech alignment on a 24GB NVIDIA 4090 GPU, with total batch size of 12 samples. We use the AdamW optimizer (Loshchilov & Hutter, 2019) with 91 0.9, ,2 = 0.999, 10e-9. takes 80k steps for training until convergence.
Lastly, we design frame-level duration predictor to predict the target speech duration during inference, which obtains the total duration of the target speech through summing up the phoneme-level inputs. We directly estimate the total target speech duration instead of the phoneme-level durations. The duration predictor has three convolution layers and a MLP architecture to predict duration from the frozen SpeechTS (Ao et al., 2022) encoder. The predictor is trained using the Adam optimizer with 0.9 and 0.999. The initial learning rate is set to lc-4 with a learning rate decay of 0.999. We use a total batch size of 32 and train the model with NVIDIA 4090 GPUs at least 100k steps.
18

---

Emotional Face-to-Speech
Model
Configuration
Parameter
In / Out Channels
1/1
Number of Transformer Blocks
12
Hidden Channel
768
Attention Heads
12
Multimodal Diffusion Transformer
e Identity Embedding Dimension
256
Emotion Embedding Dimension
128
cees Text Embedding Dimension
768
Activate Function
SiLU
Dropout
0.1
Input
Waveform
Sampling Rate
24kHz
Hopsize
480
Number of RVQ Blocks
12
Speech Codec
Codebook size
1024
Coodbook Dimension
8
Decoder Hidden Dimension
512
Decoder Kernel Size
12
Number of Decoder Blocks
30
ArcFace-Net Output Dimension
512
FaceNet Output Dimension
512
Identity Encoder
MLP Channels
(512, 512, 256, 256)
Activate Function
GeLU
Input
SpeechT5 Text Embedding
Conv Channel
256
Duration Predictor
Conv Kernel
5
MLP Channels
(256, 1)
Activate Function
ReLU
Table A-2: Implementation details about our DEmoFace.
E,2. Implementation Details of Baselines
EmoSperch. Accroding to EmoSpeech (Diatlova & Shutov, 2023) official code', we reproduce training process on our pre-training dataset. EmoSpeech introduces conditioning mechanism that captures the relationship between speech intonation and the emotional intensity assigned to each token in the sequence. Then train EmoSpeech following the original setting of its paper. We use the Adam optimizer (Kingma & Ba, 2015) with 81 = 0.5, 32 = 0.9, 10-9 and follow the same learning rate schedule in vanilla transformer (Vaswani et al., 2017). It takes 300k steps with batch size 64 for training until convergence in single GPU. In the inference process, the output mel-spectrograms of the EmoSPeech are also transformed into speech samples using the pre-trained vocoder?.
FastSpeech 2. Since FastSpeech 2 (Ren et al., 2021) is not open source and emotion-awareness, reproduce its method on our pre-training dataset based on the code and its emotion-award version V2C-Net (Cong et al., 2023). To model the
in FastSpeech 2, following previous methods (Chen et al., 2022; Cong et al., 2023), we utilize emotion embeddings from an emotion encoder I3D (Carreira & Zisserman, 2017) and speaker embeddings extracted via a generalized end-to-end speaker verification model (Wan et al., 2018) as additional inputs. These embeddings are projected and added to hidden embeddings before the variance adaptor. Then we train FastSpeech following the original setting of its paper. We use the Adam optimizer (Kingma & Ba, 2015) with 9 0.9, B2 0.98, 10-9 and follow the same
"htpps//gttuu..com// http::/www....o./ogn/na/aaasehha
19

---

Emotional Face-to-Speech
learning rate schedule in vanilla transformer (Vaswani al., 2017). It takes 300k steps with batch size 48 for training until convergence. In the inference process, the output mel-spectrograms of the FastSpeech 2 are also transformed into speech samples using the pre-trained vocoder.
V2C-Net. The V2C-Net (Chen al., 2022) is not open source, so we reproduce its method based its original paper and project'. To exploit the emotion from the reference video, it utilizes emotion encoder I3D (Carreira & Zisserman, 2017) to calculate the emotion embedding and proposes speaker encoder comprising 3 LSTM layers and a linear layer to explore the voice characteristics of different speakers. Then we train V2C-Net on our pre-training dataset according to the setup outlined in the original paper. The Adam optimizer (Kingma & Ba, 2015) is employed with hyperparameters set to 31 = 0.9, and B2 = 0.98. The learning rate schedule followed the approach used in the vanilla transformer (Vaswani et al., 2017). takes 300k steps with a batch size of 48. During inference, the generated mel- see-sppetrogram is converted into speech using the pre-trained vocoder.
HPM. According to HPM (Cong et al., 2023) official code', we reproduce training process on our pre-training dataset. utilizes an emotion face-alignment network (EmoFAN) (?) to capture the valence and arousal information from facial expressions and also utilizes an emotion encoder I3D (Carreira & Zisserman, 2017) to calculate the emotion embedding. For training,
Adam (Kingma & Ba, 2015) with learning rate 10 5,- 81 0.9, 32 0.98, 10-9 to optimize the HPM. It takes 500k steps with batch size 16. During inference, the generated mel-sppettrogram is converted into speech using the pre-trained vocoder.
StyleDubber. According StyleDubber (Cong al., 2024b) official code", we reproduce training process on dataset CMC-TED. StyleDubber introduces the cross-attention to enhance the relevance between textual phonemes of the script and the reference audio as well as visual emotion. For training, we use Adam (Kingma & Ba, 2015) with learning rate 0.00625, 31 0.9, 82 = 0.98, c = 10-9 to optimize the model. It takes 300k steps with batch size 64, During inference, the generated mel-spectrogram is converted into speech using the pre-trained vocoder.
Face-TTS. We use the official-releseed pre-trained model? of the Face-TTS (Lee et al., 2023), which is pre-trained on multiple large-scale TTS datasets (such as LRS3 (Afouras et al., 2018). VoxCeleb2 (Chung et al., 2018), and LJSpeech (Ito & Johnson, 2017), etc.). Following its original inference pipeline, the input face image is resized into 224 x244 pixels and embeds onto 312-dimensional vector. The output speech is decoded from their released vocoder in 1 16kHz sampling rate.
F. Additional Results
We conduct extra experiments under our acoustic-guide version DEmoFace ", as shown in Fig. A-1, from mel-spectrograms in the second row, the other baselines show severe over-smoothing issues, resulting quality degradation. Furthermore, from the FO curve the second row, the other baselines exhibit distinct FO contours showing different pitch, emotion, and intonation with the GT. Our results are closer to the GT with those acoustic-guided methods.
For More audio samples please refer to our supplementary material.
"hepss//ittuh.com/Glllaaa http:///hhttt..momoaaaa//a/aaaaa
20

---

Emotional Face-to-Speech
Ground Truth
(o) Ours
[b] Face-TTS
(c) EmoSpeech
(d) V2C-Net
(e) StyleDubber
Speech Prompt
M for Puhiog
Speech Prompt
with Happy
Speech Prompt
with Angry
Figure A-1: Speech qualitative results on acoustic-guided version DEmoFace x The red rectangles highlight key regions with acoustic differences or over-smoothing issues, and the red dotted circle shows similar FO contours with ground truth.
G. User Evaluation
We further conduct the subjective evaluation with 50 participants, to compare our DEmoFace with SOTA methods (Diatlova & Shutov, 2023; Lee et al., 2023). Specifically, we introduce three metrics based on five mean opinion scores (MOS), each rated on scale from to 5 in 0.5 increments. Among them, MOS,at evaluates speech naturalness and generation quality, while MOS and MOSemo assess the perceived similarity of timbre and emotional prosody, respectively. We randomly generate 15 utterances from the test set for subjective evaluation.
We conduct the subjective evaluation with 15 participants, to compare our DEmoFace with SOTA methods. Specifically, introduce five mean opinion scores (MOS) with rating scores from 1 to 5 in 0.5 increments, including MOSmat> MOS for speech naturalness (i.e. quality) and consistency (i.e. emotion and speaker similarity). We randomly generate 10 samples from the test set. Here, we give definitions of both MOS scores on Tables A-3.
Level
Value
MOSmat
Description
MOSa Description
MOSemo Description
Excellent
5
Natural, fluent, and clear speech with no rhythm semantic is- sues. High intelligibility and quality.
Timbre strongly reflects the speaker identity inferred from the facial cues, including gender and age.
Emotion expression closely matches the expression cues in the image.
Good
Mostly clear speech with minor rhythm issues. Acceptable over- all quality.
Timbre generally aligns with the imagined identity, with slight de- viations age gender Timbre partially corresponds to the visual identity cues but with evident inconsistencies. Timbre poorly aligns with the inferred identity, showing signif- icant mismatch.
Emotion is largely consistent with the facial expression, with slight differences.
Fair
Noticeable unnaturalness er- rors; speech requires effort to un- derstand. Moderate quality. Disfluent speech with abnormal rhythm and unclear articulation. Low intelligibility and quality. Unintelligible, highly disfluent, and unclear speech. Unaccept- able quality.
Emotion partially matches facial cues but with evident inconsis- tencies.
Bad
2
Emotion poorly aligned with fa- cial expression.
Poor
1
Timbre shows no meaningful re- lation the imagined speaker identity.
Emotion completely inconsis- tent with visual cues.
Table A-3: MOS MOS.. and MOS descriptions.
21


Reviews:
[{"review_id": "KXkSkcd6oX", "reviewer": "Reviewer_mfqr", "summary": "The paper introduces Emotional Face-to-Speech (eF2S), a novel task that synthesizes emotional speech solely from expressive facial cues. The authors propose DEmoFace, a generative framework leveraging a discrete diffusion transformer (DiT) with curriculum learning, integrated with a multi-level neural audio codec. Key contributions include a multimodal DiT block, a coarse-to-fine curriculum learning strategy, and an enhanced predictor-free guidance mechanism. Experimental results demonstrate improved naturalness and emotional consistency compared to baselines, even surpassing speech-driven methods.\n\n## update after rebuttal\n\nThanks for the clarification. I maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive feedback and constructive suggestions, and try our best to address your concerns as follows.\n\n\n\n**Q1: Facial expression ambiguity** \n\nThank you for your insightful comment. In this paper, we leverage a pre-trained facial expression recognition model to generate one-hot emotion labels for learning emotional embeddings, serving as a plug-and-play component. This allows for the integration of stronger models, such as micro-expression recognition, to enhance expression precision and reduce expression ambiguity. We will incorporate this discussion in the future version. \n\n\n\n**Q2: Limited data size and multilingual extension** \n\nThank you for your insightful comment. To address concerns about model generalization, our submitted paper ensures that test set speakers are unseen for RAVEDSS and MEAD, demonstrating strong generalizability. In addition, per Reviewer kkEx's suggestion, we have supplemented an out-of-domain evaluation on LRS2 by randomly sampling 1,500 utterances, with results provided in the Q2 response to Reviewer kkEx. Experimental results show that DEmoFace achieves better generalizability and outperforms the Face-TTS even with unseen speakers and semantic content. \n\nExpanding to unseen languages is an interesting challenge that requires multilingual datasets and handling linguistic and phonetic differences. However, this is beyond the scope of our eF2S task, which focuses on generating speech aligned with facial identity and emotional expression. In the future, we plan to explore multilingual speech generation using International Phonetic Alphabet (IPA) embeddings.\n\n\n\n**Q3: Training computational efficiency with standard TTS model**\n\nThank you for your valuable suggestion. Following DiTAR [1], which evaluates the FLOPs of standard Non-AutoRegressive (NAR) TTS models, we compare DEmoFace with both Continuous (Cont.) and Discrete (Disc.) NAR methods at Number of Function Evaluations (NFE = 32). As shown in the following table, DEmoFace achieves comparable training efficiency to other NAR TTS models. In the future, we will further optimize its efficiency and extend DEmoFace from cross-modal face-to-speech to standard TTS to explore its broader impact. \n\n| Type      | Methods                 | Params | TFLOPS |\n| --------- | ----------------------- | ------ | ------ |\n| Cont. NAR | E2-TTS (NFE=32)         | 0.3B   | ~56.5  |\n| Cont. NAR | F5-TTS (NFE=32)         | 0.3B   | ~37.4  |\n| Disc. NAR | MaskGCT (NFE=50)        | 1.1B   | ~116.7 |\n| Disc. NAR | NaturalSpeech 3 (NFE=1) | 0.5B   | ~8.9   |\n| Disc. NAR | **DEmoFace** (NFE=32)   | 0.2B   | ~12.9  |\n\n[1] DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation\n\n\n\n**Q4: Comparison with acoustic-driven methods**\n\nIn Table 1 of the main paper, the key difference between acoustic-guided and visual-guided methods lies in the modality used to model timbre and emotional prosody, as reflected in SpkSim and EmoSim. However, both approaches rely on textual modality to guide semantic content generation, as indicated by WER. Our acoustic-guided DEmoFace\\* outperforms the visual-guided DEmoFace due to a smaller distribution shift from ground-truth (GT) speech. This is because DEmoFace\\* is trained and tested with speech modality, while DEmoFace is trained with GT speech but tested with vision modality. \n\n\n\n**Q5: Superiority of EPFG**\n\nThe key difference between EPFG and vanilla PFG in discrete diffusion lies in multi-condition disentanglement versus aggregation. As stated in the main text (lines 401 and 432), EPFG significantly enhances multi-conditional generation quality, mitigating semantic confusion caused by aggregation. We will clarify this more explicitly in the future version.", "claims_and_evidence": "The claim that eF2S generates emotional speech purely from facial expressions is well-supported by experiments.\nThe introduction of a discrete diffusion model and curriculum learning is novel and backed by strong empirical evidence.\nHowever, the claim that DEmoFace surpasses speech-driven models needs further justification—some results (e.g., WER) still favor speech-guided methods.", "methods_and_evaluation_criteria": "The proposed method is well-structured, with clear pipeline descriptions.\nThe use of curriculum learning to gradually introduce high-level tokens is innovative.\nThe evaluation metrics (WER, MCD, EmoSim, SpkSim) are appropriate.", "theoretical_claims": "The paper extends diffusion models to multimodal emotional speech generation, which is a promising direction. The enhanced predictor-free guidance (EPFG) mechanism is theoretically well-motivated, but the justification for its superiority over standard PFG needs further clarity.", "experimental_designs_or_analyses": "The dataset selection is reasonable, covering RAVDESS, MEAD, MELD-FAIR, and a subset of LRS3.\nThe comparison with state-of-the-art models (e.g., Face-TTS, EmoSpeech) is thorough.", "relation_to_broader_scientific_literature": "The paper builds upon previous face-driven TTS, emotional TTS, and discrete diffusion models.", "essential_references_not_discussed": "The references are sufficient.", "other_strengths_and_weaknesses": "Strengths:\n\n•\tNovel problem formulation (eF2S).\n\n•\tStrong empirical results with extensive benchmarks.\n\n•\tOpen-source potential for future research.", "other_comments_or_suggestions": "Add ablation studies on dataset quality and robustness.", "questions_for_authors": "1. How does the model handle ambiguous facial expressions (e.g., neutral vs. mild happiness)?\n\n2. How would DEmoFace perform on out-of-domain data (e.g., unseen speakers, languages)?\n\n3. What are the training costs and computational efficiency compared to standard TTS models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tjkeHmb2T9", "reviewer": "Reviewer_kkEx", "summary": "The paper introduces a task named Emotional Face-to-Speech (eF2S), which aims to synthesize emotional speech directly from expressive facial cues. The proposed DEmoFace leverages a discrete diffusion transformer with curriculum learning to achieve the SOTA eF2S performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive comments and efforts in reviewing our manuscript. We try our best to address your questions as follows.\n\n**Q1: Limited novelty**\n\nThank you for the opportunity to clarify the distinctions from previous methods. Although DEmoFace builds on existing discrete diffusion models (DDMs), our work makes key contributions. \n\n1. While DDMs have shown great promise in text generation, their potential for speech generation—particularly with RVQ codec tokens—remains underexplored. \n2. Existing DDMs struggle with effective guidance for multi-conditional generation. Our EPFG addresses this challenge by providing more efficient guidance, supported by strong empirical evidence, while also introducing theoretical insights that enhance the understanding of multi-conditional generation in DDMs. \n3. DEmoFace is a unified framework for both acoustic- and visual-guided speech generation, with extensive results demonstrating its efficiency.\n\n**Q2: Limited data size**\n\nTo address concerns about model generalization, we randomly sample 1,500 utterances from the LRS2 benchmark to evaluate its performance on real-world out-of-domain data.  Results demonstrate that DEmoFace consistently achieves high-quality generation and outperforms the Face-TTS, even for unseen speakers and content. \n\n| Methods      | EmoSim$\\uparrow$ | SpkSim$\\uparrow$ | RMSE$\\downarrow$ | MCD$\\downarrow$ | WER$\\downarrow$ |\n| ------------ | ---------------- | ---------------- | ---------------- | --------------- | --------------- |\n| Face-TTS     | 0.64             | 0.13             | 104.39           | 14.29           | **16.60**       |\n| **DEmoFace** | **0.75**         | **0.64**         | **96.50**        | **12.75**       | 20.26           |\n\nFurthermore, as suggested by Reviewer 7YBQ, we have conducted dataset-wise experiments on LRS2, performing full fine-tuning of the base model for 20 epochs on four datasets to capture dataset-specific knowledge. The results are as follows: \n\n| Methods          | $\\text{Num}_\\text{utterance}$ | $\\text{Num}_\\text{word}$ | $\\text{Num}_\\text{speaker}$ | EmoSim$\\uparrow$ | SpkSim$\\uparrow$ | RMSE$\\downarrow$ | MCD$\\downarrow$ | WER$\\downarrow$ |\n| ---------------- | ----------------------------- | ------------------------ | --------------------------- | ---------------- | ---------------- | ---------------- | --------------- | --------------- |\n| Finetune-RAVDESS | 1,140                         | 25                       | 19                          | 0.57             | 0.56             | 112.68           | 13.74           | 92.05           |\n| Finetune-MELD    | 2,150                         | 2,996                    | 143                         | 0.72             | 0.53             | 105.82           | 13.08           | 38.16           |\n| Finetune-MEAD    | 8,876                         | 6,504                    | 36                          | 0.70             | 0.50             | 102.47           | 13.06           | 33.64           |\n| Finetune-LRS3    | 14,601                        | 15,401                   | 719                         | 0.72             | 0.63             | 101.58           | 12.86           | 20.73           |\n| **DEmoFace**     | 26,767                        | 15,545                   | 917                         | **0.75**         | **0.64**         | **96.50**        | **12.75**       | **20.26**       |\n\nWe have four key findings: 1) limited semantic content (low utterance and word counts) leads to higher WER; 2) limited speaker diversity (a small number of speakers) negatively affects SpkSim; 3) emotion-rich but small datasets like RAVDESS may not accurately reflect real-world distributions, as acted emotions tend to be exaggerated; and 4) mixed-dataset training improves generalization across all aspects on out-of-domain real-world data.\n\nIn future, we will expand the dataset to enhance diversity and real-world applicability.\n\n**Q3: Clarification on efficiency analysis**\n\nWe measure latency on 4090 GPU with a mini-batch size of 1 and 32 utterances, and function evaluation numbers (NFE) of 32 and 64. Latency is averaged over the test set utterances, and we report the Real-Time Factor (RTF), which indicates the time (in seconds) required to synthesize one second waveform. The results show that our method with NFE=32 and batch size=1 has the potential to build a real-time TTS system, being 2 times faster than real-time. In future, we will optimize inference efficiency by developing accelerated sampling techniques.\n\n|                                  | RTF$\\downarrow$ |\n| -------------------------------- | --------------- |\n| VoiceCraft (batch size=1, NFE=1) | 1.92            |\n| ChatTTS (batch size=1, NFE=1)    | 0.30            |\n| Ours (batch size=1, NFE=32)      | 0.49            |\n| Ours (batch size=1, NFE=64)      | 0.96            |\n| Ours (batch size=32, NFE=32)     | 0.13            |\n| Ours (batch size=32, NFE=64)     | 0.23            |", "claims_and_evidence": "The claims made in the paper are generally well-supported by clear and convincing evidence. The authors provide extensive experimental results, including both quantitative and qualitative evaluations.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the task. One potential limitation is the dataset size and diversity. While the authors use a combination of RAVDESS, MEAD, and MELD-FAIR, these datasets are relatively small and may not fully capture the variability in real-world scenarios.", "theoretical_claims": "It seems the theoretical claims are correct in this paper.", "experimental_designs_or_analyses": "I have checked the experimental designs. It seems the authors have provided comprehensive experiments in the main text and appendix.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Weaknesses:\n1. Limited Novelty and Efficiency:\nThe paper builds on existing techniques (e.g., discrete diffusion models, curriculum learning) without introducing fundamentally new algorithms. While the combination of these methods is creative, the lack of novel theoretical or algorithmic contributions limits the paper's originality. Additionally, the authors do not discuss the efficiency of the proposed framework, such as computational cost or inference speed, which is critical for real-world applications.\n2. Limited Dataset Size and Diversity:\nThe experiments are conducted on relatively small datasets (RAVDESS, MEAD, and MELD-FAIR), which primarily focus on English speakers and Western facial expressions. This limits the generalizability of the results and raises concerns about the model's performance in more diverse cultural and linguistic contexts. The lack of evaluation on larger or more varied datasets hinders the paper's ability to demonstrate the framework's robustness and applicability to real-world scenarios.", "other_comments_or_suggestions": "None", "questions_for_authors": "Please see the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p2ZzB67CqX", "reviewer": "Reviewer_7YBQ", "summary": "This paper argues that extracting and applying emotional expressions as well as identities when generating speech based on face prompt input is effective in resolving face-speech mismatch.\n\nTo this end, we propose an Emotional Face-to-Speech (eF2S) method that goes beyond the existing Face-to-Speech (F2S) method and applies emotions extracted from faces to the generated speech.\nThe proposed framework for eF2S (DEmoFace) generates speech by directly integrating both identity and emotional expressions from face input.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments and suggestions, motivating us to rethink a more comprehensive experiment evaluation. We try our best to address your questions as follows.\n\n\n\n**Q1: Emotion ambiguity between text and face**\n\nThank you for your insightful comment. Determining the dominant and subordinate modalities is crucial for resolving ambiguities in multimodal emotion generation. However, in our eF2S task, we focus on inferring timbre and emotional prosody solely from visual cues, with facial expressions serving as the dominant modality rather than textual cues. In the future, we will explore integrating face, text, and speech for more consistent emotion generation by dynamically adjusting modality impacts through cross-modal shift estimation.\n\n\n\n**Q2: Dataset-wise experiment for out-of-domain robustness**\n\nWe appreciate your insight for a more thorough evaluation of our method. To address your concern, we have supplemented dataset-wise experiments on the out-of-domain LRS2 benchmark. Specifically, due to the limited dataset size, we conducted full fine-tuning of the base model for 20 epochs on each of these four datasets to learn dataset-specific knowledge and tested them on LRS2. Please refer to the Q2 response for Reviewer kkEx for quantitative results, and we will include this evaluation and discussion in the future version.\n\n\n\n**Q3: Clarification on t-SNE experiment**\n\nTo further compare our method with real speech data and ablation methods, we have added the requested visualizations, which can be found in Figures 1 and 2 in the anonymous link https://anonymous.4open.science/r/demoface. The results demonstrate that our method exhibits a clustering distribution similar to that of real speech.\n\n\n\n**Q4: Extra ablation study on emotion conditioning**\n\nThank you for helping improve the clarity of our paper. We have conducted an ablation study on the $\\boldsymbol{c} _\\text{emo}$  condition. The results below show that $\\boldsymbol{c} _\\text{emo}$   enhances speech naturalness and expressiveness without significantly affecting SpkSim or WER, confirming that DEmoFace effectively decouples different conditions. The associated t-SNE visualizations can be found in the anonymous link provided above. \n\n| Methods                        | EmoSim$\\uparrow$ | SpkSim$\\uparrow$ | RMSE$\\downarrow$ | MCD$\\downarrow$ | WER$\\downarrow$ |\n| ------------------------------ | ---------------- | ---------------- | ---------------- | --------------- | --------------- |\n| wo $\\boldsymbol{c} _\\text{emo}$ | 0.64             | 0.65             | 104.92           | 7.29            | 21.35           |\n| **DEmoFace**                   | **0.70**         | **0.67**         | **101.18**       | **6.86**        | **20.78**       |\n\n\n\n**Q5: Limited subjective evaluation**\n\nFollowing the setup of Face-TTS [1] using 17 evaluators, we initially conducted similar evaluations, and we acknowledge that a larger number of evaluators ensures more reliable results. Per your suggestions, we have expanded the evaluation to 50 evaluators with 15 samples each, and we also introduce new MOS metrics for both timbre （MOS$ _\\text{id}$） and prosody （MOS$ _\\text{emo}$） evaluation as Reviewer 4KNc suggested. The new evaluation results with 95% confidence intervals are as follows:\n\n| Methods   | MOS$ _\\text{nat}\\uparrow$ | MOS$ _\\text{id}\\uparrow$ | MOS$ _\\text{emo}\\uparrow$ |\n| --------- | ------------------------ | ----------------------- | ------------------------ |\n| EmoSpeech | 2.30±0.19                | 2.93±0.09               | 2.78±0.13                |\n| Face-TTS  | 2.28±0.09                | 2.67±0.12               | 2.75±0.09                |\n| **DEmoFace**  | **3.17**±0.18                | **3.20**±0.11               | **3.26**±0.12                |\n\nThe results show that DEmoFace outperforms EmoSpeech and Face-TTS in naturalness, identity timbre, and emotional prosody consistency. We will revise our manuscript accordingly. \n\n\\[1\\] Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech\n\n \n\n**Q6: Clarification on existing method limitations**\n\nThe limitations of existing methods can be categorized into two aspects: 1) **task-level**, where they cannot jointly model speaker identity and emotion solely from visual cues, and 2) **method-level**, which involves issues like one-to-many mapping with limited diversity or inefficiencies in continuous NAR or discrete AR frameworks. Building on these insights, we propose a novel discrete diffusion framework for a novel task—emotional face-to-speech generation.\n\n\n\n**Q7: References to be discussed**\n\nThank you for your suggestion. We will certainly incorporate the relevant discussion and citations [2-3] in the future version. \n\n\\[2\\]: Deep facial expression recognition: A survey \\\n\\[3\\]: Emotion Recognition and Generation: A Comprehensive Review of Face, Speech, and Text Modalities", "claims_and_evidence": "I agree that generating speech synchronized with emotion is essential for natural speech synthesis. \nHowever, is it necessary to extract emotion from a single video frame? \nFor generating speech that aligns with emotional intent, it is reasonable to assume that text-based emotion should have a more significant influence than facial emotion.\n\nThe Emotion of mismatch between face and text can lead to inconsistencies in speech generation. \nFor example, if the image shows a smiling face, but the text expresses anger (e.g., “I am angry”), this would represent a perceptual conflicts between the generated speech expression and the text expression. In this paper does not provide a detailed discussion on how to handle such conflicts—specifically, which modality (face or text) should take precedence in determining the speech’s emotion.", "methods_and_evaluation_criteria": "In this paper, the authors did not provide a comparative evaluation across the datasets used in training. While it employs multiple datasets (RAVDESS, MEAD, MELD-FAIR, and LRS3), there is no analysis on how the model performs differently across these datasets.\n\nSpecifically, it is unclear whether emotion-rich datasets like RAVDESS lead to better emotion modeling compared to conversational datasets like MELD-FAIR. Additionally, there is no discussion on whether training on one dataset generalizes well to others.", "theoretical_claims": "Appendix B outlines the preliminaries of the discrete diffusion model, offering relevant definitions and concepts.\n\nAppendix C presents the full derivation of the Enhanced Predictor-Free Guidance (EPFG) equations (Equations 4-5).\n\nHowever, the explicit mathematical derivations for Equations (1-3) are not detailed in the supplementary materials.", "experimental_designs_or_analyses": "1. Lack of experiments about emotion condition:\nThe paper does not experimentally verify which factor—text emotion or face emotion—plays a more dominant role in generating speech emotion when they conflict.\n\n2. Inconsistency in the interpretation of Figure 4 (t-SNE visualization):\nThe paper claims that emotion regulation leads to more natural speech synthesis, but Figure 4 (t-SNE visualization) primarily shows a distribution based on gender differences.\n\n3. Lack of verification of face emotion to speech synthesis:\nThere is no evaluation of performance when generating speech using only identity $c_{id}$ and text $c_{text}$, without the emotion condition $c_{emo}$.", "relation_to_broader_scientific_literature": "This paper builds on prior work in Face-to-Speech (F2S) and expressive TTS by focusing on visual emotion conditioning, which has been underexplored in previous works.\n\nUnlike previous TTS that primarily focused on identity-based speech generation, this work explicitly decouples identity and emotion to synthesize the speech that consistent emotion between speech and face.", "essential_references_not_discussed": "In this paper, the generated speech is incorporated a emotion of face by extracting emotional features. \n\nHowever, it lacks a discussion on how Facial Emotion Recognition has evolved and its limitations in speech emotion modeling.", "other_strengths_and_weaknesses": "- Strengths:\n\nIn this paper is proposed a novel speech generation framework, incorporating emotion conditioning from facial features, which has been insufficiently explored in previous works.\n\n- Weaknesses:\n\nThe lack of dataset-specific experiments makes it difficult to evaluate whether this model generalizes well.\n\nThe t-SNE visualization (Fig. 4) only shows gender-based clustering, which does not directly validate the effectiveness of emotional conditioning.\n\nThere is no ablation study for removing the emotion condition, which is critical to justify the importance of facial emotion in speech generation.", "other_comments_or_suggestions": "In the introduction section, when explaining the limitations of existing methods compared to the authors’ approach, the explanation is difficult to follow. (L26~45)", "questions_for_authors": "1. How does the model resolve conflicts between text emotion and face emotion when they are contradictory?\n\n2. Why does not measure dataset-wise comparison performance? \n\n3. Why does Figure 4 focus on gender-based clustering instead of emotional clustering?\n\n4. What would be the impact of removing the facial emotion condition in ablation studies?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QY98C8T4st", "reviewer": "Reviewer_4KNc", "summary": "This paper describes an approach for mapping silent video of a talking face to the synthesized voice.  The approach is based on a discrete diffusion transformer that is conditioned on the (visual) speaker identity and a learned representation of the facial expression of emotion.  Together these help to preserve speaker identity and improve the expressiveness of the speech.   In addition, residual vector quantization is used learn a coarse-to-fine tokenization to better capture voice characteristics at different levels of granularity.  The approach is evaluated using both objective and subjective assessment against a number of baselines.\n\n### Update\nI will maintain my score and rate this paper weak accept. The approach used makes sense, but I am disappointed in the evaluation in that the baselines the authors are comparing against are not strictly the true baselines.  The authors of the original baselines used different datasets, and the authors here can claim their approach beats the baselines on their experimental configuration, but the \"standard\" baselines do much better than the examples presented here.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your kind words and appreciating the significance of our contributions, and we try our best to address your questions as follows.\n\n\n**Q1: Impact of expressiveness variation** \n\nThank you for the insightful suggestion. In this paper, we use one-hot emotion labels to learn identity-agnostic emotional embeddings, ensuring that variations in expression do not significantly affect the generation results. In the future, we plan to incorporate emotional intensity as an additional condition to achieve more natural speech synthesis.\n\n\n\n**Q2: Extra user study**\n\nThank you for the valuable suggestion. Per your suggestion, we have supplemented an extra objective evaluation with larger participants and new MOS metrics for both attributes. Please refer to the Q5 response for Reviewer 7YBQ for the results, which show that DEmoFace outperforms in naturalness, identity timbre, and emotional prosody consistency. We will revise our manuscript accordingly.\n\n\n\n**Q3: Clarification on one-to-many mapping issue in continuous speech features**\n\nThe one-to-many mapping of speech generation denotes that multiple speech sequences can possibly correspond to the same text sequence with diverse pitch, duration and prosody, making the synthesis speech distribution multimodal instead of unimodal. The issue arises from attempting to generate continuous speech features, such as mel-spectrograms, which are highly correlated over time and frequency, leading to over-smoothing during frame-level prosody or linguistic predictions [1]. In addition, in this paper, we focus on extracting identity styles and emotions from facial features, rather than semantics from lip motion. We will certainly clarify this in future version.\n\n[1]: FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model\n\n \n\n**Q4: Clarification on t-SNE experiments**\n\nTo further compare our method with real speech data and ablation methods, we have added relevant visualizations referring to Figures 1 and 2 in the anonymous link https://anonymous.4open.science/r/demoface. The results demonstrate that our method shows a similar clustering distribution to that of real speech.\n\n \n\n**Q5: Implementation of baselines**\n\nTo ensure a fair comparison, we have included detailed implementation details in the appendix. The speech quality differences in some baselines, compared to original demos, stem from their limited dataset size and lenient evaluation, which may lead to overfitting. For example, they relied on the limited-vocabulary GRID dataset or the speaker-limited V2C dataset. In contrast, our method introduces a mixed dataset and is evaluated on unseen speakers,  reflecting more challenging real-world scenarios. \n\n \n\n**Q6: Clarification on related works, references, and writing**\n\nWe appreciate your valuable feedback, which has helped us refine our work. \n\n1. We will revise the related work section in detail: Compared with acoustic-guided methods, these non-autoregressive (NAR) methods suffer from over-smoothing, less diversity, and complex alignment issues, despite fast inference. Compared with visual-guided methods, they still pose the above issues by introducing NAR for coarse mel-spectrogram generation and diffusion for refinement and lacking an efficient conditioning paradigm.  In contrast, our DEmoFace can flexibly leverage both visual and acoustic conditions while dynamically aligning text for higher-quality and more diverse speech generation, offering practical guidance for optimizing the promising modeling paradigm.\n2. We will add the suggested references and compare them with ours.\n3. We will refine our writing to improve clarity and enhance readability.\n\n \n\n**Q7: Training cost comparison in Figure 5**\n\nFigure 5 of main text shows a comparison of metrics in the same model checkpoint, under the same training steps or costs, which demonstrates that curriculum learning can improve training efficiency. \n\n \n\n**Q8: Should facial expressions be called the ``most visual indicator''**\n\nOur proposed eF2S problem aims to infer timbre and emotional prosody solely from visual cues instead of acoustic cues, with facial expressions serving as the most relevant source of emotional information naturally.\n\n \n\n**Q9: Why need extra an LRS3 subset for learning more semantic units**\n\nThe \"semantic unit\" refers to text-rich units containing contextualized linguistic details. The audiovisual datasets used in this work exhibit limited linguistic diversity in acted emotional speech, which constrains the model to accurately generate speech for unseen text. Therefore, we incorporate a subset of LRS3 to enhance semantic learning from real-world scenario.", "claims_and_evidence": "I believe the claims are backed by the experiments in the paper.", "methods_and_evaluation_criteria": "The datasets used for training and evaluating are standard.  Reasonable baselines have been used to benchmark against too, including face-driven methods and speech-driven methods.", "theoretical_claims": "No, I did not check the derivations in the appendix beyond skimming them for information.", "experimental_designs_or_analyses": "I do not have specific concerns about the experimental design.  It is good to see a combination of objective and subjective assessment of the approach.  I did wonder about asking viewers about rating the identity and expressiveness separately.  How much does each individual attribute contribute to the goodness/degradation in the perceived consistency?", "relation_to_broader_scientific_literature": "There are two main contributions here:  1) conditioning the discrete diffusion transformer on the facial expression representation in addition to the identity representation, and 2) using residual vector quantization, which has been shown to be important in learning speaker representations.", "essential_references_not_discussed": "A potentially relevant reference that is missing:  Lu, J.; Sisman, B.; Liu, R.; Zhang, M.; and Li, H. . VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over. In Proceedings of ICASSP.  2022.\n\nMost of the baselines (e.g., from Table 1) are not discussed in the related work section.  It would be useful to have the work situated within that broader literature.  Explain what their individual limitations are, etc.", "other_strengths_and_weaknesses": "Strengths:\n+ The use of RVQ/curriculum learning and conditioning on the representation of the facial expression clearly improve the quality of the generated speech.\n+ The approach is significantly better then the baseline as evidenced by the results of the objective and subjective tests, and listening to the provided samples (albeit with the concern I have about the provided samples from the baselines that I highlight elsewhere),.\n\nWeaknesses:\n- It would have been nice to have more examples to see how other factors affect the approach.  For example, varying the degree of expressiveness from mid-expressive to very expressive.  How does this affect generation quality, and does the relative degree affect the contribution of the identity/expression conditioning on the network?", "other_comments_or_suggestions": "There are places where the word choice or use of incomplete sentences are use makes the paper difficult to follow without having to re-reading.  For example, the first sentence of the description of the forward diffusion process (line 141) is not a sentence, the sentence beginning at line 152 is poorly written,  and the last sentence before the description of the training objective in Section 3 is also not a sentence.\n\nExpand the caption for Figure 4 to more completely describe the figure.\n\nIn Figure 5, the relative training cost is provided.  Does this mean that “baseline” and “ours” are using a different number of steps, or are they using the same number of steps within a sub-figure, but a different number between the WER, emotion/speaker similarity figures?\n\nIn the introduction you mention: “Considering that facial expressions are the most direct indicators of emotion“ — it might be worth qualifying this and saying facial expressions are the most VISUAL indictor — tone of voice and other acoustic cues are equally indicative of emotion state too.\n\nIn the Datasets section of Section 5, what does “Additionally, these datasets lack sufficient semantic units in real-world environments, making it challenging to train a TTS model.” mean?  What are the “sufficient semantic units”?  Do you mean these audiovisual datasets are too small in terms of sample size (hours of speech) to train a high-quality TTS system?", "questions_for_authors": "Q1:  In the introduction, the wording refers to “... the one-to-many issues inherent in continuous speech features.\" — what are these one-to-many issues?  Given the context of the work (lip motion to speech mapping) I was wondering if it is the mapping of a visual lip gesture, e.g., lip closure of a bilabial plosive, mapping to many speech sounds, e.g., /b/, /p/, /m/.  This does not sit with \"continuous speech features.\" though so I am unclear.\n\nQ2:  For Figure 4 — these are all data points from synthetic speech.  How do data points for synthetic vs. real speech align in this low-dimensional projection?  For example, are all of the data points for the synthetic voice of a speaker aligned with/distinct from the data points for the corresponding real voice.  What is presented here shows that the performance of DemoFace aligns better than the baseline, but that is only part of the story.\n\nQ3:  I have a concern over the examples used in the baselines that are provided by way of comparison.  The voice quality in some of the baselines is considerably worse than the quality suggested on the original demo page(s).  Why is this the case?  From what I understand you are using the original implementations provided by the authors, and in many cases the same datasets (which suggests it is not a distribution shift that might require different hyper-parameters).,", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: The paper introduces Emotional Face-to-Speech (eF2S) as a novel task that synthesizes emotional speech solely from expressive facial cues. Then, the DEmoFace, a generative framework leveraging a discrete diffusion transformer (DiT) with curriculum learning,  is proposed, which is integrated with a multi-level neural audio codec. Key contributions include a multimodal DiT block, a coarse-to-fine curriculum learning strategy, and an enhanced predictor-free guidance mechanism. Experimental results demonstrate improved naturalness and emotional consistency compared to baselines, even surpassing speech-driven methods. The paper can be further improved by adding experiments to validate the generalizability and do ablation studies to justify the importance of facial emotion in speech generation.
Decision: Accept (poster)