Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis
Authors: Tianrui Wang
Haoyu Wang
Meng Ge
Cheng Gong
Chunyu Qiang
Ziyang Ma
Zikang Huang
Guanrou Yang
Xiaobao Wang
EngSiong Chng
Xie Chen
Longbiao Wang
Jianwu Dang
Venue: NeurIPS 2025 spotlight ()
Keywords: Emotion Control
Speech Rate Control
Controllable Speech Synthesis
Word-Level Control
Primary area: applications

Abstract: While emotional text-to-speech (TTS) has made significant progress, most existing research remains limited to utterance-level emotional expression and fails to support word-level control. Achieving word-level expressive control poses fundamental challenges, primarily due to the complexity of modeling multi-emotion transitions and the scarcity of annotated datasets that capture intra-sentence emotional and prosodic variation. In this paper, we propose WeSCon, the first self-training framework that enables word-level control of both emotion and speaking rate in a pretrained zero-shot TTS model, without relying on datasets containing intra-sentence emotion or speed transitions.
Our method introduces a transition-smoothing strategy and a dynamic speed control mechanism to guide the pretrained TTS model in performing word-level expressive synthesis through a multi-round inference process. To further simplify the inference, we incorporate a dynamic emotional attention bias mechanism and fine-tune the model via self-training, thereby activating its ability for word-level expressive control in an end-to-end manner. Experimental results show that WeSCon effectively overcomes data scarcity, achieving state-of-the-art performance in word-level emotional expression control while preserving the strong zero-shot synthesis capabilities of the original TTS model.

Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis
Tianrui Wang 1.2,3 Haoyu Wang', Meng Ge', Cheng Gong Chunyu Qiang Ziyang Ma Zikang Huang', Guanrou Yang", Xiaobao Wang
Eng Siong Chng', Xic Chen", Longbiao Wang-.7, Jianwu Dang Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing. Tianjin University, *Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ). "Nanyang Technological University, TTeleA,, China Telecom, 'Kuaishou Technology, "Shanghai Jiao Tong University, 7 Huiyan Technology (Tianjin), Shenzhen Institute of Advanced Technology
Abstract
While emotional text-to-speech (TTS) has made significant progress, most existing research remains limited to utterance-leve emotional expression and fails to support word-level control. Achieving word-level expressive control poses fundamental challenges, primarily due to the complexity of modeling multi-emotion transitions and the scarcity of annotated datasets that capture intra-sentence emotional and prosodic variation. In this paper. we propose WeSCon, the first self-training frame- work that enables word-level control of both emotion and speaking rate pre- trained zero-shot TTS model, without relying on datasets containing intreseeneence emotion or speed transitions. Our method introduces transition-smovt smoothhng strat- egy and dynamic speed control mechanism to guide the pretrained TTS model in performing word-level expressive synthesis through a multi-round inference process. To further simplify the inference, incorporate dynamic emotional at- tention bias mechanism and fine-tune the model via self self-training. thereby activating its ability for word-level expressive control in an end-to-end manner. Experimental results show that WeSCon effectively overcomes data searcity, achieving state-of- the-art performance in word-level emotional expression control while preserving the strong zero-shot synthesis capabilities of the original TTS model.
Introduction
Humans possess the ability regulate emotional expression during speech flexibly M. To simulate this expressive capability, recent advances in text-to-speech synthesis (TTS) have increasingly focused on controllable generation of various aspects speech, such as timbre, emotion, and speaking rate 2 Such control is key objective in the development of human-likk and expressive TTS.
Most current TTS models exhibit zero-shot capabilities, enabling them to synthesize speech from text while cloning attributes such as timbre, emotion, and speaking rate from a reference speech Despite these advances, as shown in Figure TT emotional and speaking rate control in current models is typically limited to the utterance level. This differs significantly from how humans naturally express emotion in speech. Unlike global speaker identity, emotional expression and speaking rate are dynamic and often vary within a single sentence [6] 7 Therefore, word-level control these factors is essential for achieving more natural and expressive speech synthesis (8]. To address this limitation, some approaches have proposed phoneme-leved emotion prediction from target
sample 3 4
"Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Target Text
Current ITS
Emotion Prompt
You came late and
surprised but it's okay.-
2 @ 3mz |||||||||10 [ unf You came late and I was surprised but it's okay..
Speed Control (1-5: viow fast)
Ours
C 4+2
Figure 1: Word-level control of emotion and speaking rate aims to modulate both attributes within an utterance, guided by multiple emotional prompts and amotion-speed--ageee tagged text. Our approach, WeSCon, achieves this using only small-scale public dataset without emotion transitions.
While these methods show potential for word-level emotion control, relying solely on text makes it difficult to capture essential acoustic cues such as prosody and intensity, which are vital to emotional expression control [12 [3] [4]. To address this limitation, recent studies such as ELaTE [5] and EmoCtrl-TTS [T6] have demonstrated that reference speech with emotional content can support intra-utterance control of time-varying expressive patterns, such as transitions from laughter crying. These works reflect a growing interest in TTS with word-level control over both emotion and speaking rate, but they also underscore several fundamental challenges. First, word-level expression control requires multiple emotional speech prompts, which introduces the challenge of guiding the model to attend to the appropriate emotion each word. In addition, current methods for fine-grained expression control rely on large-scale emotional speech datasets with time-aligned emotion transitions. However, such datasets are limited in both scale and accessibility [17). making fine-graied control even more difficult to realize in practice. These challenges lead us to ask: Is it possible to achieve effective word-level control of both emotion and speaking rate without relying on speech datasets containing emotion or speed transitions?
text to guide expressive synthesis 9
10
In this work, motivated by the zero-shot potential of pretrained TTS models, we propose WeSCon, two-stage self-training framework that achieves Word-level Emotion and Speed Control for TTS using only a small amount of public speech data without emotion speed transitions. In the first stage, we design a multi-round inference framework that incorporates a transition transition-smoothing module and a dynamic speed control mechanism. Without relying any emotional training data, this approach enables a pretrained zero-shot TTS model to perform high-quality word-level emotional expression control in TTS. In the second stage, the original TTS model is repurposed as a student and trained under the supervision of the 1st-stage teacher. A dynamic emotional attention bias is introduced, enabling the student acquire word-level control of emotion and speed through simplified end-to- end inference process, without the need for complex iterative generation or smoothing. Experimental results show that WeSCon achieves state-of-the- -hrrt-rrrttt performance the task of word-level emotional expression control in TTS, while preserving the zero-shot generalization and generation capabilities of the pretrained TTS model. Our contributions are summarized as follows:
We propose multi-round inference mechanism equipped with transition smoothing and dynamic speaking rate control, which is the first to achieve word-level control of both emotion and speaking rate in TTS without relying on any emotional training data.
We further introduce novel self- eef-trrining framework with dynamic emotional attention bias mechanism that empowers a pretrained TTS model with end-to-end word-level emotion and speaking rate control, using limited data without intra-sentence emotion or speed transitions.
We conduct comprehensive experiments to validate the effectiveness of our proposed frame- work. Results show that our method enables a pretrained zero-shot TTS model achieve SOTA performance in word-level emotional expression control, while preserving its original zero-shot capabilities. Ablation studies further confirm the contribution of each key design component. Our samples are available at htttpp://wwagttammrte github. io io/wescon/
2 Related Work
Scarcity of Emotional Dataset
The development of controllable TTS, particularly for emotional
expression control, depends heavily high-quality emotional speech datasets [18 19 20 While

---

74 You came late
Zero-Shot TTS 69
DynSpeed Control
Transition Smooth
You came late and I was surprised but (A4 714-2-+3 it's okay..
A 2
Zero-Shot TTS +
DynSpeed Control
Transition
m
was surprised
Smooth
|
Zero-Shot TTS
DEAB
G
n but it's
Zero-Shot TTS &
DynSpeod
Transition
733
G4
m2
m3
okay..
Control
Smooth
||
1st- Stage:
Teacher Model
2nd-Stage: Self-Training
Figure 2: Overview of WeSCon. The Ist-stage teacher extends a zero-shot TTS model with dynamic speed control, transition smoothing, and multi-round inference to enable word-level emotion and speaking rate control. In the 2nd stage, supervises a student model with dynamic emotion attention bias (DEAB) to achieve the same control in end-to-end manner with reduced inference complexity.
public corpora such as ESD [2 IEMOCAP [22). and CREMA-D (23] are available, they primarily provide utterance-level annotations and lack word-level time-aligned emotional labels. These datasets are also limited in size and diversity, often consisting of scripted speech and covering narrow range of emotions and speakers 24 More importantly, emotional datasets with intra- sentence variation, which are essential for learning word-level control, remain extremely searce and typically restricted to private use [15). Creating such datasets is expensive, requiring detailed word- or frame-level annotation and subjective emotional labeling [25]. This lack of fine-graud gra-grained emotional data poses a major challenge for training models capable of word-level expressive TTS.
From Utterance-Level to Word-Level Controllability of Emotion and Speaking Rate Most controllable TTS systems support only utterance-level control, where single label or reference speech governs the entire sentence 26 27 To achieve word-level control, some methods attempt to
predict frame- or phoneme-level emotional indicators from text alone [28 29 30 but they often fail to capture expressive variability due to the lack of acoustic cues such as intensity and prosody 19 10
Other approaches, such as ELaTE [15] and EmoCullTTS [16]. introduce emotional reference speech to enable intra-utterance control of specific expressive patterns like laughter or crying. While these represent progress, they are typically limited in expressiveness or rely on large-scale emotional datasets that are rarely publicly available. Consequently, achieving general and flexible word-level control over both emotion and speaking rate remains major challenge.
Self-Trainin under Data Scarcity Self-training has become a promising approach for low- resource speech signal processing, enabling knowledge transfer without fine-grained datasets [31 32)
While it has been applied to tasks like speaker adaptation (33, paralinguistic modeling [44 and speech translation (6), its use for fine- ine-grained emotional control in TTS remains unexplored, especially without detailed expressive labels. To address the scarcity of fine- grained datasets for word-level expressive control, we propose a self-training framework where teacher model with multi-round inference, transition smoothing, and dynamic speed control generates expressive pseudo- labels. A student model, sharing the teacher's backbone, is then fine-tuned under its supervision to perform word-level emotion and speaking rate control through a simplified end-to-end inference process, using only small public dataset without intra-sentence emotion or speed transitions.
3 WeSCon
3.1
Overview
WeSCon is a two-stage self-training framework that enables word-level control of emotion and speaking rate in pretrained zero-shot TTS model, using only small amount of emotional speech data without intra-sentence emotion transitions as prompts. As shown in Figure [ in the first stage, we introduce multi-round inference process with transition smoothing and dynamic speaking rate control to generate speech with word-level expression variations. In the second stage, the lst-stage model acts as a teacher to guide the original TTS model, equipped with a dynamic emotional attention bias (DEAB), toward word-level control through a simplified end-to-end inference. Sections 3.2
and 3.3 describe the two stages, and Section 3.4 provides the training details.
3

---

Speaker emboodingg
1 Speach with diffesent
eroctional expression (colec)
Taken to Wavform
Froues pec-trained modale 4
Medial need to be trance
token of prompt
Turgei test token
Vocoder
+
Sperch soken of prompt
Clenerated speach token
embedding
Flow Matching
Text roken till of last round
Special token (BOS5
Generstol toless multi sound inforences
Speech soken tail lf'ast round
Special waken (SOS)
First Word Segment To Speech with Emotion! I
Second Word Segment to
Aligner
Speech with Emotion II
Aliginer
Sexe-Speech Language Model
+
TExt-Speech
Language Model
(Speed Dows)
Interpolatin
Downssmpling (Speed Up)
(Speed Downl
Interpolation
Dewmsarrpling (Spood Up)
X
Figure 3: Word-level emotion and speaking rate control using transition-smoothing module and dynamic speed adjustment. At each inference round, an emotional prompt is used to generate a speech segment, with the tail of the previous output appended to ensure continuity. Speaking rate is controlled by interpolating or downsampling prompt speech tokens. The final utterance is produced by concatenating all segments and decoding them through flow matching and vocoder.
3.2 Teacher Model
3.2.1 Word-Level Emotion Control
As discussed in Section 2 current TTS models can perform utterance-level emotion and speaker cloning. Building on this, we adopt the high-performance Cosy Voice2 (37] as our backbone (details of the backbone architecture are provided in Appendix A and propose a multi-round inference strategy, where the model synthesizes multiple segments using different emotional prompts to achieve word-level emotion control. While this approach enables flexible emotional modulation, it often causes unnatural acoustic discontinuities at segment boundaries. To address this, we introduce sansition-smoothingg mechanism that improves coherence across inference rounds, as illustrated in Figure 3 Without modifying CosyVoiced Voice2, we append a lightweight content aligner, composed of non-causa Transformer (38) and convolutional layers. Trained on ASR data, this module predicts the corresponding text token for each speech token and requires no emotional supervision. During inference, the input text is segmented based on a user-defined emotion plan. At each inference round, the final text and speech tokens from the previous round are appended to the current prompt, forming an explicit tail-to-head linkage. This aligns naturally with CosyVoice2 Voice2's s continuation-sty generation 39 40 enabling smooth and coherent emotional transitions.
3.2.2 Word-Level Speaking Rate Control
In Cosy Voice2, utterance-level temporal prosody, including speaking rate and duration, is entirely determined by the reference speech prompt. To support more flexible and word-level control of speaking rate within a single utterance, we introduce dynamic speed control mechanism as part of our multi-round inference framework, as illustrated in Figure 3 The core idea is to adjust the prompt speech tokens using either nearest-neighhor interpolation or downsampling, Interpolation extends the prompt length, which slows down the generated speech, while downsampling shortens the prompt, resulting faster speaking rate, As demonstrated in Appendix B this resampling method provides effective global prosody control. By integrating it into the multi-round inference process, the speaking rate can be dynamically controlled at the word level as needed.
3.2.3 Speaker Consistency
Although the speech tokens Voice2'ss language model (LM) are primarily designed to encode semantic information (as introduced in Appendix A). these speech tokens may still inadvertently leak a small amount of speaker-related information. contrast, the flow matching serves as voice conversion- based reconstructor that transforms the generated speech tokens into the voice of

---

Teacher
Emotion
Emotion
MarMal
Specch Feuture
Fixtuur
Target of Self- Self-Training
WesCon
Preset Blas
Aligner
Normalizz
Bia
Text-Speech
Language Model
Sofimax
MLP
Softraax
Imput of Self- Training
MalMul
Emotion Algnnr
Inpurs of Multiple- Times Inference
Cnterpolaiin/Dowwaange
Text- Text-Speoch Language Model
Emnotion Indicator
Prompe
Indicator
InterpllatooneDoaaannggg Downsammling
Figure 4: The proposed self-training strategy. A teacher model under a complex multi- found inference manner supervises student TTS model to enable word-level emotion and speaking rate control. The dynamic emotional attention bias mechanism further enhances expressive generation in a simplified end-to-end single-pas pass inference manner.
specified target speaker. This design implies that as long as speaker inconsistency is avoided during the multi-round inference process in the LM part, the flow matching can effectively enforce speaker consistency in the final output. To ensure this consistency, we adopt a speaker-aware prompt selection strategy. Specifically, during multi-round inference, we prioritize selecting emotional prompts from different emotions of the same speaker. Then, a reference sample from the target speaker is randomly selected to provide the speaker identity flow matching for generating the target speaker's speech.
3.3 Self-Training
In the previous section, we enabled word-level control of emotion and speaking rate by introducing a multi-round inference framework for Cosy Voice2 [37] However, components such as the non- causal content aligner, multi-round inference, and hil-t linkage introduce significant inference complexity. To reduce this overhead while preserving controllability, we adopt a self self-training strategy. As shown Figure 4 the enhanced first-stage model serves as a teacher to supervise the original TTS model. The student model, equipped with dynamic emotional attention bias, learns to achieve word-level emotion and speaking rate control through a simplified end-to-end inference.
3.3.1 Self-Training with Teacher-Generated -rantionnraasition Speech
Our teacher model achieves word-level control of emotion and speaking rate without modifying the original TTS parameters, relying instead on complex inference pipeline with dynamic speed control and multi-round generation. To transfer this fine- fine-grained control ability to a simplified end-to -n- model, we propose self-training strategy. Specifically, the Ist-stage teacher model guides the student model to learn word-level controllability. We first use GPT-4o [1] to generate emotion-transition text sequences (details are shown in Appendix (D). which are paired with public emotional speech samples (without emotion transitions) as prompts. The teacher then synthesizes speech with word-level variation in emotion and speaking rate. These outputs are filtered based on character accuracy and expressive similarity (details are introduced in Appendix [.), and the student model is fine-tuned on the filtered supervisions with a small learning rate. This enables word-level emotional expression control during inference without requiring multi-round generation or dynamic concatenation.
3.3.2 Dynamic Emotional Attention Bias
We aim to preserve the strong zero-shot capability of the original TTS model while enabling word- level control of emotional expression under the self-training framework. To achieve this, we formu- late the input structure as { prompt prommp II C B Sprompt I Sprompt II ..... StSt], where
5

---

Crompt 8 and Strompt i denote the text and speech tokens of the i-th emotional prompt, respectively. C is the target text token sequence, and S is the corresponding speech token sequence used as supervision. The symbols ( and ( indicate the beginning of text and speech. This design remains fully compatible with the original input format @ C, @. S) of Cosy Voice2, preserving the autoregressive pattern of the pretrained model. To further encode word-level emotional variation within this unified format, we extend the text-side input by inserting explicit emotion indicator tokens that mark the boundaries between emotional segments. As illustrated in Figure 4 the final input sequence preceding ( becomes ( E Cprompi E CPrompt II ...... -J. where each E acts as a soft anchor guiding the model modulate emotion transitions during generation.
While the above data formatting preserves Cosy Voice2's generalization by avoiding interference with learned knowledge, it introduces a new challenge: during synthesis, the model may incorrectly attend to amotion-inconsistent prompts. For instance, when generating speech aligned with Emotion I, attention may drift toward prompts labeled with Emotion II, leading to emotional inconsistency and degraded synthesis quality. To address this, we propose dynamic attention bias mechanism that constrains the model's focus to emotion-relevan prompt regions based on the predicted emotional trajectory, Concretely, we introduce a causal lightweight Transformer to predict token-level emotion labels Et for each speech token S from historical context. Using the predicted emotion sequence, we introduce dynamic attention bias mechanism at each Transformer layer. We first concatenate the current text-speech representation with the predicted emotion features and project through a linear layer. The output is processed in two ways: one path adds a residual and feeds into the next layer, while the other is passed to an MLP 42] and softmax to produce a weight vector w E 1R The w is then used to compute dynamic attention bias by linearly combining seven predefined attention bias templates Blemp E T
XT (see Appendix for details). The resulting bias is computed as:
B hias
w B!
1=0
Then we multiply the bias with the normaaz-normaaizzed attention selectively emphasize regions aligned with the current emotional context. The final self-attention output is computed as:
Softmax
[QK] Vd
Bbius
V
Softmax
QK3
bias
vi
where Q, K, V E Rxxxxx denote the multi-head (H) query, key, and value, respectively, and d is the attention head dimension. The operator @ denotes clement-wise multiplication. This formulation enables the model to dynamically focus on emotionally relevant prompt segments at each generation step, thereby improving alignment between the generation and the intended emotional trajectory.
3.4 Detail Training Setup
WeSCon is trained in two stages. The first stage trains a content aligner to ensure smooth transitions during multi-round inference. In the second stage, a self-training strategy is adopted to transfer the teacher model's ability to control word-level emotional expression to the original TTS model.
The First Stage (Teacher Model) We use forced alignment 43 to generate token-level alignments between transcripts and speech, which serve as supervision for the content aligner. The TTS model remains frozen throughout this stage. Training of the content aligner is conducted without multi- round forwards. Let C' and S denote the input text and speech token sequences, Yinken e N denote the aligned target token sequence, where each label corresponds to one of V12 token classes. Let bb E 7'x1 be the binary label sequence for content boundary detection. The content aligner is jointly trained with token-level content classification loss and a binary boundary detection loss:
75
Laligner Σ log (Ytoken 1=
S C.
B
000) log (Y:ad) $ C, S, 0ns (02)
(3)
where T and TS denote the last frame indices for text and speech, and T = 7S - 7 is the total number of speech tokens. The learnable parameters ( correspond to the content aligner, while 0nh the frozen TTS model parameter used during forward propagation. We also apply class weighting

---

during loss computation to reduce the impact of overrepresented silence tokens and address the imbalance in boundary label distribution [4
The Second Stage (Self- (rain-Trann The teacher model generates supervision via multi nound inference using -ggeeea-geer texts with emotion labels. Token-level emotion labels are aligned based on emotion- correspondence. The student model is optimized by two objectives. The first a negative oggllikelhhood for speech token prediction:
T Lus Σ
log
S Cormpr
C
Ele
Expeech
StImmpp
S 0th 0Pm)
(4)
where C' and S are text and speech tokens for prompt and target, E are text-level and token-level emotion labels, and trainable 4th 6th denote TTS model and emotion aligner parameters. The second a token-level cross-entropy loss for emotion prediction:
Lo =
logp
S
Crropt
C
E 1 Ppech
Sprompt St tTh ga)) 0ea))))
(5)
t= Thross
4 Experiments
4.1
Experimental Setup
Data and Model Configuration In the first stage, the content aligner is trained 200 hours of non- emotional
speech from sibriSpeech-100--C0 100-Clan [45] and AISHELL- 46 In the second stage, the teacher model uses non-transition emotional train-set from ESD 21
prompts
synthesize training samples based on enotion-transitio texts generated by GPT-4o (see Appendix [ for generation details and examples). We adopt CosyVoices [37 the backbone TTS model. The content aligner is composed of five non-causal Transformer layers and two 5x5 convolutional layers with stride 1 and batch normalization [47), following Cosy Voice2's configuration for architectural consistency. In the second stage, the emotion aligner is a lightweight two-layer causal Transformer. The emotional attention bias module includes a linear layer with hidden dimension of 14 and an MLP output dimension of
Setup of Training and Inference In the first stage, the content aligner is trained for 400k steps on 2 NVIDIA 3090 GPUs using Adam 48] with learning rate linearly warmed up to 2.5e-4 over the first 10% of steps, then linearly decayed O. Each batch contains 90 seconds of speech. In the second stage. the student model is trained for 600k steps on 4 NVIDIA 3090 GPUs. The TTS model frozen for the first 20k steps to focus on training the emotion aligner. Fach batch contains 40 seconds of speech, and Adam is used with a fixed learning rate of 5e-7. Repetition-awan top-k sampling is applied during inference, with
and temperature 0.9.
Evaluation To evaluate word-level control over emotion and speaking rate, we construct test sets based on test set of ESD and use outstanding zero-shot TTS models [50 ST 52 37 with multi-round concatenative inference as baselines (see Appendix G for details). We use objective and subjective metrics to assess system performance (see Appendix G.3 for details). For intelligibillit, we report WER using Whisper-Large [53] for English and CER using Paraformer (54) for Chinese. Speaker similarity (S-SIM) is computed via cosine similarity of WavLM-Large embeddings (55). evaluate prosody alignment, we use AutoPCP [56]. Emotion similarity metrics (Emo2v. and Aro.) are computed using emotion2vec-LLarg [57] and a wav2vec-based model (58), respectively. We use variance of DNSMOS-PPI Pro [59] (DNSV) to assess the naturalness of emotion transition. Subjective evaluation includes four kinds of Mean Opinion Score (MOS): SMOS (speaker similarity), NMOS (naturalness of emotion transition), EMOS (emotion match), and SPMOS (speed match), each rated on a 5-point scale. Both the mean and 95% confidence intervals of MOS are reported.
4.2
Experimental Results
4.2.1 Comparison with Reference Models
Objective Evaluation We evaluate our method on word-level emotion and speaking rate control in both English and Chinese TTS. shown Table WeSCon (Ist-stage) and WeSCon (2nd-stage) consistently outperform baselines expressive metrics. Notably, the 2nd-stage model achieves
7

---

Table 1: Objective results on English and Chinese test sets for TTS with word-level emotion and speaking rate control. The best results for each metric are in bold, and the second-bes are underlined.
Emotion
Method
WER/CER
DNSV
S-SIM
AutoPCPt
Emo2v.
Aro.
Index-TTS
2.611
8.967
0.387
2.436
0.858
0.434
English
F5-TTS
2.954
8.972
0,453
2,417
0.869
0.447
Spark-TTS
2.787
8.637
0.374
2.560
0.861
0.440
CosyVeice2
3.185
7.894
0.521
2.525
0.866
0.446
WeSCon (1st)
3.204
4.577
0.531
2.689
0.879
0.463
WeSCon (2nd)
3.192
4.361
0.532
2.707
0.882
0.468
Index-TTS
1.834
8,521
0,490
2,470
0.838
0.514
Chinese
F5-TTS
1.965
9.134
0.478
2.541
0.847
0,510
Spark-TTS
1.897
8.633
0,441
2.518
0.848
0.530
Cosy Voice2
2.119
7.612
0.581
2.514
0.843
0.537
WeSCon (Ist)
2.129
4.980
0.595
2.650
0.866
0.551
WeSCon (2nd)
2.122
4.210
0.599
2.663
0.872
0.556
the highest Emo2V. and Aro. scores in both languages, demonstrating strong word-level emotional expressiveness enabled by our self-training framework. Regarding transition smoothness, our models significantly reduce DNSV compared CosyVoice2, with values dropping from 7.894 to 4.361 in English and from 7.612 to 4.210 in Chinese. This highlights the effectiveness of our smoothing mechanism and the end-to-end continuous inference in the 2nd-stage model in mitigating acoustic discontinuities across transitions. While the character error rate is slightly higher than baselines, it remains comparable to CosyVoice2, Voice2, backbone model. Finally, the 2nd-stage model slightly surpasses the Ist-stage model, benefiting from self-training with selective filtering that retains high- quality supervision from the teacher. Overall, our approach consistently improves upon CosyVoice2 Voice2 and achieves SOTA performance in key aspects of word-level expressive controllable TTS.
Subjective Evaluation We conduct subjective evaluations covering emo- tional expressiveness (EMOS), speak- ing rate control (SPMOS), speaker sim- ilarity (SMOS), and naturalness of emo- tion transition (NMOS), with details provided in Appendix G.3 As shown in T our method, WeSCon, con- sistently outperforms all baselines. It achieves more expressive and control- lable speech while maintaining speaker identity, demonstrating effective word-level control in tional expression. Additionally, WeSCon delivers more natural-sounding speech with smoother and more accurate speaking rate modulation.
Table Subjective results evaluated by 15 listeners, with 95% confidence intervals computed from the t-test.
Method
EMOS
SPMOS 1 SMOS
NMOS f
Index-TTS F5-TTS
3.51=0.19
3.50
3.06++..3
0.97+:0.25
3.63=0.15
3.51+0.21
3.11
0.84+0.26
Spark-TTS
3.55=0.19
3.63+0.18
2.96+0.24
2.99+0.26
Cosy Voice2
3.61:0.17
3.56.10.20
3.54
3.29
WeSCon
3.70 3.89 3.96 3.23
Capability on Zero-shot TTS In addition introduc- ing word-level controllability, we evaluate the perfor- mance of our method on the standard zero-shot TTS task using the SEED test set (test-zh) [60]. As shown in T the WeSCon (Ist) model yields results identical to Cosy Voice2, as the backbone TTS is frozen during this stage. The 2nd-stage model also achieves compara- ble results. Together with the findings in Table [ these results demonstrate that our method enables word-level emotion and speaking rate control without significantly degrading the original zero-shot TTS performance of the pretrained model.
Table 3: Objective evaluation on standard zero- shot TTS performance using character error rate (CER) and speaker similarity SIM).
Method
CER
S-SIM f
CosyVoice2 [37
1.45
0.748
WeSCon (Ist)
with Cosy Voice2
WeSCon (2nd)
1.47
0.744
4.2.2 Ablation Study
Transition-Smoone -Smoothinn Mechanism We evaluate the impact transition-smoothing mechanism by removing the tail-to-head alignment during multi-round inference in the 1st-stage model. As shown in Table removing this mechanism ("woo smoothing") leads substantial increase in DNSV (from 4.980 to 7.5800tt7776600000 7.568), indicating degraded smoothness between expressive transitions. Additionally, speaker

---

(S-SIM) and emotion similarity (Emo2V. and Aro.) drop notably, suggesting that the discontinuity negatively affects both emotional expression and speaker consistency. These results confirm that our smoothing strategy plays crucial role in ensuring coherent segment transitions during generation.
Speaking Rate Control To exam- ine the effectiveness of our dynamic speaking rate control, we remove this component from the 1st-stage model ("w/o speed control"). As shown in Ta- ble 4 DNSV slightly increases from 4.980 to 5.067, and performance drops are observed across most expressive metrics, such AutoPCP (2.650 to 2.499) and Emo2v. (0.866 0.844). This suggests that speaking rate varia- tion provides important prosodic cues for emotional expression in TTS. In addition, we further investigate the in- teraction between speaking rate control and emotional expression in Appendix
Table 4: Ablation study on two stages for word-level controllability on Chinese testset.
A PCP
Emotion
Method
CER DNSVL S-SIM
Emo2v.
Aro.
WeSCon (Ist)
2.129
4.980
0.595
2.650
0.866
0.551
w/o smoothing
2.209
7.568
0.576
2.596
0.851
0.531
w/o speed control
2.126
5.067
0,582
2.499
0.844
0.526
WcSCon (2nd)
2.122
4.210
0.599
2.663
0.872
0.556
w/o attention bias
2.398
5.534
0.575
2.511
0.837
0.519
w/o emotion flag
2,455
5.880
0,573
2.492
0.831
0.515
w/o datafilter
2.237
4.494
0,592
2.627
0.859
0.542
w/o dataformat
4.141
5.697
0.579
2.504
0.819
0.509
In the 2nd-stage model, we evaluate the effect of removing the dynamic emotional attention bias ("w/o attention bias"). As shown in Table 4 this results in a clear performance drop across all metrics, especially emotion similarity. DNSV also increases, indicating reduced smoothness. The results confirm the importance of the attention bias module in enabling the 2nd-stage model to focus on the correct emotional prompt during inference.
Dynamic Emotional Attention Bias
Data format of Self-training We further investigate the importance of data formatting in self- training. As shown in Table removing the emotion flags ("w/o emotion flag") results performance drops across all metrics, indicating that these flags play a crucial role in signaling the locations of emo- tional shifts the model. Furthermore, replacing our input data format with naive one that simply concatenates prompts and targets ("w/o data format"), as [Crrompt 1, @ Sprompt II C B S leads to the most significant degradation expressive metrics, including sharp increase in CER from 2. 66 to 4. 141. These results suggest that aligning the data organization with the structure used during pretraining allows the model to better leverage its pre-trained knowledge.
Self- Hull-Triiing Data Size We evaluate the im- pact of training data size the self-training pro- cess by varying the amount of synthetic speech used fine-tune the 2nd-stage model. Metrics are normalized between O and 1. As shown in Figure [5] performance improves with more data and peaks at 500 hours. Beyond this point, met- rics begin to decline. This trend attributed to the limited variety of emotional categories and speaker identities in the ESD, which restricts ex- pressive diversity and leads to overfitting when the data scale becomes overly redundant.
1.0
0.8
CBR
0.6
Emu2V.
DNSV
AntaPCP
2009
Figure 5: Performance trends on Chinese testset under different self-training data sizes.
Out-of-Domain Generalization and Alignment To further assess the model's 's robustness, we evaluate generalization ability on out-of-domain data (Appendix H). addition, we report the alignment accuracy achieved in both training stages (Appendix IT.
Conclusion, Limitations, and Broader Impact
Conclusion In this paper, we propose WeSCon, the first method to overcome expressive data scarcity and enable word-level emotional expression control through end-to-end inference, under a self-training framework with dynamic emotional attention bias mechanism. Experimental results show that WeSCon achieves state-of-the-art performance using only limited data without emotion or speed transitions, while maintaining strong zero-shot TTS capabilities.

---

Limitations and Future Work 1) Gradual emotion transitions. While WeSCon achieves smooth signal-level transitions, lacks semantic modeling of emotional evolution. In human speech, emo- tional changes often involve intermediate states. 2) Emotion diversity and composition. The model is limited a fixed set of discrete emotions and does not support compositional blended expressions, such as combining anger and sadness to convey despair. 3) Conditioned control. Emotional transitions are currently predefined by GPT-4o-based plans, which restricts flexibility. Future work will explore more dynamic, context-aware control strategies enable natural, interactive emotional expression.
Broader Impact WeSCon can be applied to expressive speech synthesis, virtual agents, and emotional storytelling. However, it may also pose risks related to speaker impersonation, especially when specific content and speaker prompts are combined. Like other generative models, it may produce biased or inappropriate outputs, although no such cases were observed during testing.
6
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grant U23B2053 and Grant 62176182 and the Open Research Fund from Guangdong Laboratory of Artificial Intelli- gence and Digital Economy (SZ) (No.GML-KK-24-16)
References
[1] Klaus R Scherer. Expression of emotion in voice and music. Journal of voice, 933):235-248 1995.
[2] Tianxin Xie, Yan Rong, Pengfei Zhang, Wenwu Wang, and Li Liu. Towards controllable speech synthesis in the era of large language models: A survey. arXiv preprint arXiv:2412.0660 2024.
[3] Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speedt with minimal supervision. Transactions of the Association for Computational Linguistics, 1 11:1 10:17731718, 2023.
[4] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.021/1 2023.
[5] Zeqian Ju. Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech. 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint 2xXiv:2433033100 2024.
[6] Sylvie Mozziconacci. Prosody and emotions. In Speech prosody, volume 2002, pages 1-9, 2002.
[7] Haotian Guan, Zhilei Liu, Longbiao Wang, Jianwu Dang, and Ruiguo Yu. Speech emotion recognition considering local dynamic features. In Studies on Speech Production: I 11th Interna- tional Seminar, ISSP 2017, Tianjin, China, October 16-19, 2017, Revised Selected Papers 11, pages 14-23. Springer, 2018.
[8] Yiwei Guo, Chenpeng Du, and Kai Yu. Unsupervised word-level prosody tagging for control- lable speech synthesis. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7597-7601, 2022.
[9] Haobin Tang, Xulong Zhang, Ning Cheng, Jing Xiao, and Jianzong Wang. ED-TTS: Multi-scale emotion modeling using cross-domain emotion diarization for emotional speech synthesis. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing IEEE, 2024.
(ICASSP), pages
[10] Yi Lei, Shan Yang, and Lei Xie. Fine-grained emotion strength transfer, control and prediction for emotional speech synthesis. 202 IEEE Spoken Language Technology Workshop (SLT), pages 423-430 2021.
10

---

[11] Chenpeng Du and Kai Yu. Rich prosody diversity modelling with phone-level mixture density network. In Interspeech 2021, pages 3136-31 -140, 2021.
[12] Li-Wei Chen and Alexander Rudnicky. Fine- line-grained style control in transformer- cansformer-based text-to- speech synthesis. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)) pages 7907-7911, IEFE, 2022.
[13] Shreeram Suresh Chandra, Zongyang Du, and Berrak Sisman. Exploring speech style spaces with language models: Emotional tts without emotion labels. arXiv preprint arXiv:2405.11413, 2024.
[14] Wei Zhao and Zheng Yang. An emotion speech synthesis method based on vits. Applied Sciences, 13(4)22225, 2023.
[15] Naoyuki Kanda, Xiaofei Wang, Sefik Emre Eskimez, Manthan Thakker, Hemin Yang. Zirun Zhu, Min Tang, Canrun Li, Baung--Hser Tsai, Zhen Xiao, al. Making wow-matching-bass zero-shot text-to-speech laugh you like. arXiv preprint arXiv:2402.07383, 2024.
[16] Haibin Wu, Xiaofei Wang, Sefik Emre Eskimez, Manthan Thakker, Daniel Tompkins, Chung- Hsien Tsai, Canrun Li, Zhen Xiao, Sheng Zhao, Jinyu Li, et al. Laugh now cry later: Controlling vime-varying emotional states of flow-matching- based zero-shot text-to-speech, In 2024 IEEE Spoken Language Technology Workshop (SLT), pages
IEEE, 2024.
[17] Tarun Rathi and Manoj Tripathy. Analyzing the influence of different speech data corpora and speech features on speech emotion recognition: A review. Speech Communication, page 103102, 2024.
[18] Xinfa Zhu, Lei, Tao Li, Yongmao Zhang, Hongbin Zhou, Heng Lu, and Lei Xie. Metts: Multilingual emotional text-to-speech by cross-speaker and cross-lingual emotion transfer. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 32:1506-151 1518, 2024.
[19] Yiwei Guo, Chenpeng Du, Xie Chen, and Kai Yu. Emodiff: Intensity controllable emotional text-to-speech with soft-label guidance. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[20] Guanrou Yang. Chen Yang, Qian Chen, Ziyang Ma, Wenxi Chen, Wen Wang. Tianrui Wang, Yifan Yang, Zhikang Niu, Wenrui Liu, Fan Yu, Zhihao Du, Zhifu Gao, ShiLiang Zhang, and Xie Chen. Emovoice: Llm-based emotional text-to-speech model with freestyle text prompting, 2025.
[21] Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset. In ICASSP 2021-202) IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 920-924. IEEE, 2021.
[22] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359, 2008
[23] Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390, 2014.
[24] Ziyang Ma, Mingjie Chen, Hezhao Zhang, Zhisheng Zheng, Wenxi Chen, Xiquan Li, Jiaxin Ye, Xie Chen, and Thomas Hain. Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark. In Interspeech 2024, pages 1580-1584, 2024.
[25] Rui Liu, Zhenqi Jia, Jie Yang. Yifan Hu, and Haizhou Li. Emphasis rendering for conversational text-to-speech with multi-modal multi-scale context modeling, 2024.
[26] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tt:: family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.0224 02430, 2024.

---

[27] Zhihao Du, Qian Chen, Shiliang Zhang. Kai Hu, Heng Lu. Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, al. Cosyvoice: A scalable multilingual zero-shot ext-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.0555 055007 2024.
[28] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong. Chen Zhang, Yanqing Liu, Xi Wang. Yichong Leng, Yuanhao Yi. Lei He, et al. Naturalspecch: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(6):4234 4245, 2024.
[29] Chae- Bin Im, Sang-Hoon Lee, Seung-Bin Bin Kim, and Seong-W Whan Lec. Emoq-tts: Emotion intensity quantization for fine-grained controllable emotional text-to-speech. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
pages
[30] Xuan Luo, Shinnosuke Takamichi, Yuki Saito, Tomoki Koriyama, Hiroshi Saruwatari, et al. Emootin--controllable speech synthesis using emotion soft label, utterance-level prosodic factors, and word-level prominence. APSIPA Transactions on Signal and Information Processing, 13(1),
2024.
[31] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. Advances in neural information processing systems, 33:3833-3845, 2020.
[32] Massih-Reza Amini, Vasilii Feofanov, Loic Pauletto, Lies Hadjadj, Emilie Devijver, and Yury Maximov. Self-training: A survey. Neurocomputing, 616:128904, 2025.
[33] Sameer Khurana, Niko Moritz, Takaaki Hori, and Jonathan Le Roux. Unsupervised domain adaptation for speech recognition via uncertainty driven self-training. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6553-6557. IEEE, 2021.
[34] Dong Yang, Tomoki Koriyama, and Yuki Saito. Frame-wise breath detection with self- training: An exploration of enhancing breath naturalness in text-to-speech. arXiv preprint arXiv:2402.00288, 2024.
[35] Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang, Self-training for end-to-end speech translation. In Interspeech 2020, pages 1476-1480, 2020.
[36] Qingkai Fang, Rong Ye. Lei Li, Yang Feng, and Mingxuan Wang. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 7050-7062, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[37] Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao. Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10111 101 2024.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[39] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier. Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31:2523 2533, 2023.
[40] Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang. Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are shro text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing, 33:705-718, 2025.
12

---

[41] Aaron Hurst. Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprini arXiv:2410.21276 2024.
[42]
Popescu, Valentina E Balas, Liliana Verescu-Popesse and Nikos Mastorakis. Multilayer perceptron and neural networks. WSEAS Transactions on Circuits and Systems, 8(7):579-588, 2009.
[43] Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pages 498-502, 2017.
[44] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9): 2009.
[45] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[46] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open- mandarin speech corpus and speech recognition baseline. In 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment (O-COCOSDA), pages 1-5. IEEE, 2017.
[47] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. pmlr, 2015.
[48] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint 111iv:1412.6980, 2014.
[49] Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers, 2024.
[50] Wei Deng. Siyi Zhou, Jingchen Shu, Jinchao Wang, and Lu Wang. Indextts: An industrial-level controllable and efficient zero-shot text-to-speech system. arXiv preprint 1555 2025.
[51] Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024.
[52] Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, al. Spark-tts: An efficient Ilm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710, 2025.
[53] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492-285188 PMLR, 2023.
[54] Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. arXiv preprint arXiv:2206.08317, 2022.
[55] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, al. Wavlm: Large-scale self-supervised pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505-15 2022.
[56] Loic Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Dup- penthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, al. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint 157 2023.
13

---

[57] Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15188 15185, 2023.
[58] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. framework for self-supervised learning of speech representations. Advances in neural information processing systems, 13:12449-12444 12460, 2020.
wav2vec 2.0:
[59] Fredrik Cumlin, Xinyu Liang, Victor Ungureanu, Chandan KA Reddy, Christian Schüldt, and Saikat Chatterjee. Dnsmos pro: A reduced-size dnn for probabilistic mos of speech. In 25th Interspeech Conferece 2024, Kos Island, Greece, Sep 2024-Sep 5 2024, pages 4818 3818-4822. International Speech Communication Association, 2024.
[60] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong. Lelai Deng, Chuang Ding. Lu Gao, Mingging Gong, Peisong Huang, Qingqing Huang. Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, Yuping Wang, Yuxuan Wang, Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang. Yuanhao Yi, Junteng Zhang. Qidi Zhang, Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, and Xiaobin Zhuang. Seed-tts: A family of high-quality versatile speech generation models, 2024.
[61] Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, et al. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint MrXXiv2407.04055 2024.
[62] Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206. 2.K.:22006006658, 2022.
[63] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. urXiv preprint arXiv:2412. 15115, 2024.
[64] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4195 4205, October 2023.
[65] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[66] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (6133-16142, 2023.
[67] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt MeVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. SciPy, 2015:18-24, 2015.
[68] Ya Li, Jianhua Tao, Linlin Chao, Wei Bao, and Yazhu Liu. Cheavd: chinese natural emotional audio-visual database. Journal of Ambient Intelligence and Humanized Computing, 8:913-924, 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have ensured that the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have thoroughly discussed the limitations of our work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
We present no theoretical results or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The model architecture is described in detail in Section 3 and the Appendix F
The experimental settings are also thoroughly outlined in Section and the Appendix G Appendix presents the key training curves. We confirm that the information provided is comprehensive.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The data we use can be accessed through the cited references. We include the code and data preparation scripts the supplementary material, and we plan to open-source them in the near future.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
haboli/c/giiies/dddesunninssisss oilicll for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//abps.cc/pullic/uiiiee
swssssssiioyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are included in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
We report the mean opinion scores (MOS) along with 95% confidence intervals (CI95), computed using the t-distribution over ratings from independent human listeners in Table 2
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed description of the computational resources used for experiments in Section 4.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips. http://///eeerpp.cccp////c//iiii inne
Answer: [Yes]
Justification: We confirm that our research conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive socictal impacts and negative socictal impacts of the work performed in Section
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have credited all the assets by listing the URL in the footnote, citing the paper, and explicitly noting the license it exists one.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsssie thcode wwwccom/aaaoo//c datasetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We write instructions in the code README about how to prepare data, launch training, and rum inference.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We conducted crowdsourced experiments, and Appendix G3 includes the full text of the instructions provided to participants, along with illustrative screenshots. No compensation was provided to the annotators.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We disclosed all potential risks to the subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We use large language model (LLM) to augment our experimental dataset. The specific usage of the LLM is described in detail in Appendix [ Although the LLM is not core model component, its use contributes directly to the experimental design and data quality, and thus we provide a clear explanation of its role.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM http:p///baaurecccoo/o/oooreeees for what should or should not be described.
21

---

A Details of Cosy Voice2
CosyVoice! 37 is zero-shot TTS model based on language model (LM) and flow matching. It first converts speech into discrete tokens through a supervised speech tokenizer module. Its core architecture identical to the base structure illustrated in Figure [] excluding the additional modules introduced in this work. The supervised speech tokenizer is jointly trained with an ASR task, which encourages the LM component foeus more on semantic modeling, particularly in terms of content, emotional expression, and duration. The flow matching component incorporates speaker embedding and target speech to provide speaker characteristics. transforms the speech tokens produced by the language model into mel-spectrogramss, primarily controlling global aspects of speech, especially speaker identity. Finally, the vocoder converts the mel-spectroggams into waveform signals. Cosy Voice2's CosyVoice2's disentangled modeling of semantic content and speaker identity provides an important foundation for our method. In addition, since training data is primarily in Chinese, demonstrates significantly better performance in Chinese than in English.
B Speed Control
As described in Section 3 control the speaking rate of synthesized speech by applying simple interpolation and downsampling to the prompt speech tokens. To assess whether this dynamic mechanism supports time-varying modulation, we visualize six types of control patterns in Figure 6 Numeric labels indicate the ratio between the transformed and original token lengths, where ratio of indicates no change, 0.5 indicates downsampling half the length, and 2 represents interpolation that doubles it. The left panel illustrates three downsampling patterns: gradually increasing interval, a decreasing interval, and uniform interval. The right panel shows corresponding interpolation patterns. These results demonstrate that global pnterollatio/dowwsaamppi effects comparable to time- varying anterpolation/downsampliii inherent randomness introduced by LM sampling. Because both methods provide only utterance-level control over speaking rate, word-level modulation requires integration with our multi-round inference framework.
can produce particularly when accounting for the
Prompt: 1-5 Downsampling
Prompt: 1-2
Interpolation
xC
3.304
6:755
0,22
6255
0.00
0:204
6:305
Prompt: 1 Downsampling
Prompt: 2 Interpolation
Prompt: 0.75 Downsampling
Prompt: 1.5 Interpolation
Figure 6: Visualization of six dynamic speaking rate control patterns, including time-varying and uniform interollatoo/doowsamppiin transformation relative to the original prompt. Blue numbers represent the duration (in frames) assigned each character. All synthesized speech shares the same content, which is marked in blue text within the figure.
The numerical labels indicate the ratio of the token length
cam++
22

---

We further investigate how different resampling ratios influence the speaking rate of the generated speech. As shown in Figure / the results reveal a clear correlation between the resampling factor and the output speed. When the token length is reduced to less than 40% of the original through down- sampling, the model fails to produce intelligible speech, as indicated by the red circles. Conversely, interpolation beyond three times the original length has minimal additional effect speaking rate. Notably, the most stable and effective control achieved when the token length lies between 50% and 200% of the original, suggesting this range practical bound for reliable modulation.
Characters per Second vs. Mulliple
Number of c'hhareteen second
Multiple of length ufter resumpling
Figure 7: Correlation between resampling ratio and output speaking rate. The most effective control observed when token lengths range from 50% to 200% of the original. Red circles mark failure cases where intelligibility is lost. The green dashed line indicates the character- character-per-scoond ser-sc rate of the prompt audio.
C Interaction Between Speaking Rate and Emotion
This section further examines the relationship between speaking rate and emotional expression. Speaking rate and emotional state are strongly coupled in human speech, different emotions are typically associated with distinct prosodic rhythms and energy patterns. Since both rate and emotional cues derived from the same prompt speech, temporal resampling inevitably alters the perceived emotional expression. To quantify the effect of speaking rate on perceived emotion, 100 emotional utterances are randomly selected from the test as prompts. The resampling ratio is systematically varied from 0.5 to 2.0 in increments of 0.25, using the same target text for all conditions, summarized in Table [5] For each condition, the emotion similarity between the generated speech and both the original and rate-mat matched (re-rated) reference speech is calculated using the Emo2v. score.
Table 5: Effect of speaking rate variation on emotion similarity.
Resampling Ratio
Emo2v. 1
Emo2v. (Re-rated) 1
0.5 (downsampled to half, speed up)
0.57
0.86
0.75
0.85
0.88
1,0
0.90
0.90
1.25
0.83
0.89
1.5
0.75
0.90
1.75
0.68
0.91
2.0 (interpolated twice, speed down)
0.51
0.87
The results show that emotion similarity declines substantially when the reference is not rate- nate-matched, whereas it remains stable when compared with rate-adjusted references. Notably, when the resampling ratio deviates significantly from the natural range (e.g., 0.75 -1.5), the perceived emotion becomes less consistent, likely due to distortion of spectral dynamics and pitch contours caused by excessive time-stretching or compression. These findings confirm that speaking rate provides essential prosodic cues for emotional perception, consistent with the observations in Section 4.2.2
23

---

D Generation of Emotionally Varying Texts
We employ GPT-4o 4 to generate the corpus of sentences containing intra-sentence emotional transitions. To ensure the emotional transitions contextually plausible, construct prompts based on predefined scenarios, character relationships, and conversation topics. An example of the prompt is shown in Listing ?? Specifically, first create large pool of randomly generated environments, contexts, and interpersonal relationships with personality traits. During generation, three elements randomly selected and injected into the prompt to guide GPT-4o in producing scripts.
[caption=<Example prompt for generating sentences with emotion shifta using GPT-40.), [abel-prroapt]][sson
You are a scriptwriter tasked with creating emotionally expressive single sentene dialogues with internal emotion shifts**. Your output should be grounded in the following:
**Dialogue environment and external factors**,
**Dialogue content and situational context**, **Interpersonal relationships and character traits**.
Output Format Template
Each dialogue entry consists of list of sentence segmenta**, where **each segment is labeled with its corresponding emotion and speaking speed**. The entire list represents a single sentence spoken by a character.
Example:
[
"ines_seg": "I trusted you", eemttion": ": "sad",
"speed": "1.25"
}
{
:e "but you",
"emotion" ; "surprise", "speed": "0.9"
>
{ "lines_seg": "lied to me!", "emotion", "angry", "speed": "1.5"
}
}.
]
Key Task Requirements
There are spemkere) characters: (', >, join (sjiakeee)))ss) Dialogue alternates between speakers; speaker may speak twice in succession**
Each sentence must be internally segmented (2"4 segments) and exhibit **clear emotion transitions**
Each segment must include:
wwline_seg*** a span 2"4 words, with punctuation only at the end of the last segment
**emotion**: the expressed emotion in this segment, chosen from: { emotions)
**spedd** the speaking where 0.5 very fast, 1 normal, 2.0 very slow) Dialogue Environment and External Factors (environment)
for this segment (range: 0.5 2.0.
Dialogue Content and Context
(context)
Interpersonal Relationships and Character Traits
(character_traite traits)
24

---

T0T 01T072007722 0T0.1100001000000
TOT 52505182
70T 1T5 52505152
TOTT T00.200012000S10005555
10
11
72
50
(1)
(2)
(3)
TI
T2
TO
50
3
51
52
F
(6)
(7)
Figure 8: Illustration of seven predefined emotional attention bias patterns. Red elements denote prompt inputs, green elements denote target text and speech. indicates text tokens, and S indicates speech tokens. Numbers represent emotional pairs indices. The light blue regions are preset the dark blue regions are preset to 5, and the upper-right triangular region is entirely set to 0.
E Data Filtering in Self-Training
During the self-training process, we introduce data filtering mechanism to ensure the reliability of the teacher model's guidance. Specifically, we adopt three metries for evaluating the quality of generated speech: CER for Chinese and WER for English, speaker similarity, and emotion similarity. The first-stage teacher model has explicit access to the aligmment among content, speech, emotion prompts, and speaker prompts, allowing to directly compute these metrics with the prompt. To avoid introducing bias from the final objective evaluation metrics prematurely, we deliberately use models that differ from those employed during evaluation. For speech recognition,
adopt the Sense Voice mode [61. J. For emotion representation, we use Whisper model fine-tuned for speech Resemblyzer" We normalize all three metrics for each data point and compute a combined score by summing them. Only the top 50% of data, ranked by this composite score, are selected for self-training. In other words, as shown in Figure [ for a 500-hour training set, we actually generate approximately 1000 hours of data. Similarly, for 2000-hour training set, we generate around 4000 hours of data.
emotion recognition For speaker embedding,
F Predefined Emotion Attention Bias
Since the emotional alignment of the student sequence input can be obtained from the output of the emotion aligner, we introduce seven predefined attention bias patterns to reduce the modeling burden of the emotional attention shift module. These typical patterns are illustrated in Figure g and described below.
(1) Standard GPT-style Causal Attention. Each token attends to all previous tokens in standard autoregressive manner without any emotional constraints.
(2) Striet Emotion-Allined Attention. This corresponds to the original training strategy of Cosy Voice2. For instance, when decoding the second emotion segment (green S2), the model is only allowed to attend to the corresponding emotional prompt and its associated text, specifically red and green T2, and red S2.
(3) Full Text History + Emotion-Al Aligned Speech Attention. On top of (2), this setting allows tokens attend the full text history, while speech tokens remain strictly aligned with their respective emotional segments.
otchaopenai--- soimai-whiipprr mehisper-largee large-v2
25

---

(4) Full History Access for Prompt Speech Encoding, Extending (3), this setting additionally allows each prompt speech token access all previous tokens during encoding.
(5) Prompt Speech Attends to Its Own History During Target Speech Generation. During the generation of target speech, when prompt speech tokens are revisited, each token allowed to attend to all previous prompt speech tokens.
(6) Prompt Speech Self-Attentin allows prompt speech tokens to attend to the full history during encoding, but during target speech generation, they attend only previously encoded prompt speech tokens.
in Encoding. Combining (4) and (5), this configuration
Although some attention bias configurations, such as (5), (6), and (7), are relatively uncommon in standard architectures, our predefined template-basee computation allows the Emotional Attention Bias module to focus solely on selecting and composing from these candidate biases. This design sig- nificantly reduces computational overhead and prevents the generation of implausible or inconsistent attention patterns.
G Evaluation Setup
G.1 Details of Dataset
We use the train-set, dev-set, and test-set of ESD (11] for training and evaluation. This dataset contains 350 parallel utterances, averaging 2.9 seconds in duration, spoken by 20 speakers: 10 native English and 10 native Mandarin (5 male and 5 female for each language). Each speaker expresses five emotions: happy, sad, neutral, angry, and surprised. All audio is sampled 16 kHz.
For evaluation, we generate 1,000 ammotion-speed-valt varying text samples (500 in Chinese and 500 in English) using the script provided in Appendix [ For each text sample, randomly select emotional prompts from the ESD test set to match the emotion transitions required by the sentence. All emotion prompts within single sentence The reference audio for the target speaker is also randomly selected from the same language-speaker subset. As result, approximately I out of every samples features emotional prompts and target speaker from the same speaker-emotie setting, given that the ESD dataset contains 5 emotions.
drawn from the same speaker ensure consistency.
G.2 Baselines
We adopt four strong zero-shot TTS systems as baselines:
Index-TTS" R 50 is GPT-style TTS model enhanced with pinyin-based pronunciation cor- rection for Chinese characters and punctuation-basel pause control. It integrates improved speaker condition modeling and Big [62] for high-quality audio synthesis. Trained on tens of thousands of hours of data, it supports multilingual zero-shot generation.
Spark-TT's [52) is a large language model- TTS system built upon Qwen2.5 (63
It directly reconstructs waveforms from LLM-predicted codes, eliminating the need for separate acoustic models. This design simplifies the pipeline and improves inference efficiency. It supports zero-shot voice cloning, wwasssiingaal/odesviiiiii switching synthesis, and virtual speaker customization via controllable parameters such as gender, pitch, and speaking rate.
FS-TTS 10
51 is a non-autoregressivve TTS system based on Diffusion Transformer
(DiT) (44) and flow matching (65). It forgoes duration models and alignment by padding text to match speech length, using ConvNeXt V2 [66] to refine text features. An inference-time Sway Sampling strategy improves decoding efficiency without retraining. Trained on 100K-hour multilingual dataset, F5-TTS supports zero-shot synthesis, expressive speech generation, speed control, and seamless sode-switching.
httpp////gaaaaoooo///iiiirreeeer Speechh-ppeeeheeee Data
https: httpas//ggthbbbbooo///xeeetttttr
http:://gtthab..oo///Wv/////s/
26

---

Cosy Voice2 [37] is language model-based TTS system designed for zero-shot control of both emotion and speaker identity. Further architectural and training details are provided in Appendix A
All baseline systems share the same inference procedure: each sentence is divided into multiple word- level segments with specified emotional states and speaking rates. These segments synthesized separately using emotion cloning combined with their respective speaking rate control strategies, and then concatenated form the final speech.
G.3 Evaluation Metries
MOS Demo Test
lorge
lagee reotac Supeii
Rige
Exctox
Areay
Teget leet duubieeatooo
Scoring Guidelines (1 -5, with 0.5 pos-t intervals)
this scoring systen is usad esdkata the perforennc gnttesited sporch acoss lour dimensions: Browtos tenorieeal deriluity, Speedtios opealiing sate mute king). Speoker Oppeater dlmilartyy, and NaturannaattOS prooohhvess of speach tranaltiond. Higher targe equctad
EnokiD
Soendhics
ge recutedd beced omitioral refercsce sudios und speaker reforence audio, oneure well accasey plnsen conduct as abject and desaleed asoesmment barad the snfreenca audion and the tolloding maltrttion crterts
Seen
Erokkdd
Spedbiis
Scoring Criteria
hatur
(BtrokiC)) Erokkos compleiiyy misraalheedd atronely insomiilent: Sppeudh (D) Speakdnng rate dowatos sowtrdy from the reference descripiinn ttg. ihould be fost out slons Mo Campht byy oilluse lindar, gondos idankis dieooiiieo Hight abhnd transkions, oolicaalle lecals.
Biecchttio tmooiio is diee denallllll sirs but rot cookly dffieeet cspeedistios Speok ligiing sate eprticetty enty discitee from the setvence ne decoriptiee Speaibetatii Scess inrrrr bur dearity albentt thet MCt trenstano woard traudiane with abrapinaws.
Spuodilds
(Banoittt tmadaas geeaeally oligmed with deveatannc Dipeedtatio Sponit close the refeennce or discrpttinn with sight lifennee Sppaabstiics ticeunc ofiin featuree and lanscy simnka wilh minenc dilllvoncce (Hutanall cullttes burallion moody nutta di with siigtt discan
speadrous
(EEvollc Emotton highly coneletnnt with only rabite con, (ppooduiiii Spenting rate dlwely naiches the rebrrncc with raturel thee (ionsleehoos) Thri hew very close to the barget spedbe that indirooiiiiy Mod mend lons smowth and thaed.
Noturnncsh/O
dliooittt tnedkas in compreweyy grawi shong and curade (Spee (hetled the rete voue deverted speeking rate: Opealsstac Nouth indiaiinquinn ruuiiaaaee to in target spedkr that inalooaiiiyy soar loo tranihine with partectty aatural liww
Figure 9: The MOS evaluation interface used for rating emotion consistency, speaking rate consistency, speaker similarity, and transition smoothness.
Objective Metrics
The objective evaluation is conducted in two groups: Group 1. Given the generated speech, the target speaker's prompt, and the reference transcript, we compute three utterance-evell metrics: character accuracy, speaker similarity, and DNSV (the variance of DNSMOS-
P 55 scores). Character accuracy computed by comparing the output of an automatic speech recognition (ASR) model against the target transcript. Specifically, we use Paraformer IT] 54 to calculate character error rate (CER) for Chinese and Whisper Large V to compute word error rate (WER) for English. Speaker similarity is measured by extracting utterance-level embeddings from the generated speech and the target prompt using La (55), followed by computing the cosine similarity between them. DNSV is used to assess transition smoothness. DNSMOS-PRO scores are calculated over the generated speech using a 2-second window and a 1-second stride. The
https///gttab..cc//ppaaa////sss
verification
27

---

variance of these scores is used to quantify transition smoothness, with higher variance indicating lower smoothness. Since the value of the variance often relatively small, we multiply by 100 for display purposes. Group 2. Based on the ASR transcription obtained in Group 1, we perform foreed alignment to determine word-level timestamps. align each generated word-level segment with its corresponding emotional prompt, according to the original text-emotion-speed mapping. For each aligned pair, to evaluate expressive similarity, the emotional prompt is first adjusted to the target speaking rate using phase vocoder algorithm [67] The generated segment is then compared to the rate-adjusted prompt using AutoPC
marting strategy then used to
[56] to
compute prosodic similarity. Emotion embeddings are extracted using emotion2vec--uee a wav2vec-bars baaed mode 58). and cosine similarity is calculated to quantify emotion similarity.
[57 and
Subjective Evaluation We conduct Mean Opinion Score (MOS) evaluations from four perspectives: emotional consistency, speaking rate consistency, speaker similarity, and smoothness of emotional transitions. For each aspect, we provide participants with detailed evaluation criteria and report both the mean scores and 95% confidence intervals. A total of 15 graduate students with research back- grounds in speech emotion recognition or emotional speech synthesis participated in the evaluation. Prior to the test, all participants were provided with detailed explanation of the interface and task. They were also informed that the data would used for scientific research purposes. Each participant rated 20 sets of results (10 in Chinese and 10 in English) generated by five different systems. The complete evaluation took an average of approximately 49 minutes per participant, Scores were assigned on a to 5 scale with 0.5-point intervals. The evaluation interface is shown in Figure 9
H Out-of-Domain Evaluation
To evaluate the generalization ability of our approach under out-of-domain dataset, we conduct word-level emotion and speaking rate control experiments on the CASIA dataset [68], as organized according to Appendix G The CASIA corpus is a Mandarin emotional speech dataset recorded by four native speakers and covers six emotion categories: neutral, angry, fear. happy, sad, and surprise. Some of these emotions are not during training, which makes CASIA suitable for testing the cross-domain robustness of controllable speech synthesis. The results are shown in Table 0 Our method, WeSCon, demonstrates strong performance across nearly all evaluation metries, achieving lower DNSV and higher speaker similarity (S-SIM), emotional similarity (Emo2vec). and arousal scores compared to other baselines. The overall results consistent with those in Table [ further confirming that our method generalizes well to unseen speakers and novel emotional patterns.
Compared to Table the student model (WeSCon 2nd) shows slightly weaker performance than the teacher model on the out-of-domain test set, in some metrics. This degradation is primarily caused by the data filtering strategy adopted during self-training, which improves performance in-domain speakers and emotions but may introduce subtle biases, resulting in mild overfitting. Nevertheless, such performance fluctuations are acceptable given that the second-stage model significantly simplifies the inference process.
Table Objective evaluation results on the CASIA-based evaluation dataset for word-level emotion and speaking rate control.
Emotion Emo2v.
Method
WER/CERJ
DNSV
S-SIM
AutoPCPt
Aro.
Index-TTS
1.217
8.887
0.468
2.444
0.824
0.502
F5-TTS
1.374
8,940
0.462
2.539
0.845
0.526
Spark-TTS
1.299
8.720
0.439
2.496
0.841
0,523
Cosy
1.405
8.093
0.542
2.503
0.835
0.517
WeSCon (1st)
1.411
4.680
0.587
2.670
0.869
0.548
WeSCon (2nd)
1.478
4.641
0.590
2.624
0.867
0.552
28

---

I Training Progress
We present the evolution of key validation metries throughout the two-stage training process, as illustrated in Figure I0 and Figure 11 Figure I0 displays the frame-level accuracy of the aligner model in the first stage, covering both text token prediction and boundary detection. Figure T reports the accuracy of speech token prediction and the frame-level emotion prediction by the emotion aligner the second stage. As shown, aligner consistently achieves high frame-level accuracy both stages. This is expected, the target classes for both text and emotion are provided as input, and the aligner's primary objective to learn accurate alignments, which is relatively straightforward task given the model's underlying text-to-speech capabilities.
Aligned Text Accuracy vs. Update Steps
Boundary Accuracy vs. Update Steps
0.9
Aligned Text Accunac
0.8
Boundary Accuniyy
0.6
0.5
0.5
0.3
71
0.2
0.2
0.0
50k
100k
150k
200%
250k
300k
350k
400k
.0kk
100k
150k
200k
250k
300k
350k
400k
Update Steps
Update Steps
Figure 10: Validation accuracy of frame-level text token and boundary prediction by the aligner during the first stage training.
Speech Token Accuracy Update Steps
Aligned Emotion Accuracy Update Steps
0.256
(3 Emaoronn Aggurgey
Speech Token Accurncy
0.255
00.944
1..940
0.254
60.9939
0.253
00.9938
30.9937
0.252
00.9936
0.251
0.9935
0.9934
100k
200k
300k
400%
500k
600k
100k
200k
300k
400k
500k
600%
Update Steps
Update Steps
Figure 11: Validation accuracy of speech token prediction and aligner's frame-level emotion label prediction during the second stage self-training.
29


Reviews:
[{"review_id": "EqBbPAIdwV", "reviewer": "Reviewer_uPUi", "summary": "The paper introduces WeSCon, a novel two-stage self-training framework for enabling word-level control over both emotion and speaking rate in a zero-shot text-to-speech (TTS) model. The primary challenge addressed is the scarcity of datasets with fine-grained, intra-sentence emotional annotations. Experimental results, on both English and Chinese, demonstrate that WeSCon achieves state-of-the-art performance in word-level expressive control. It significantly improves transition smoothness and emotional accuracy compared to strong baselines, all while preserving the original model's zero-shot speaker cloning capabilities.", "questions": "The paper's core design choice—a teacher-student framework—is justified by the need for a more efficient inference process. This claim is currently unsubstantiated. Can you provide a quantitative comparison of inference speed (e.g., Real-Time Factor) for: (a) the teacher model, (b) the final student model, and (c) the baseline models. Without this data, the central motivation for the entire second stage of the work is unconvincing.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper addresses an important challenge in TTS. Achieving fine-grained, word-level control over expressive attributes is a logical and necessary next step for creating more natural-sounding synthetic speech.\n- The paper is generally well-written and structured. The figures and descriptions of the architecture provide a clear high-level understanding of the proposed pipeline.\n\nWeaknesses\n- The entire self-training process is critically dependent on texts generated by a proprietary model GPT-4o. In other words, the system can only learn emotional transitions that the LLM can imagine and articulate, potentially limiting its diversity and introducing stylistic artifacts.\n- The title claims \"Word-Level Emotional Expression Control,\" which suggests a high degree of flexibility. However, the control is limited to a small, predefined set of discrete emotions, and the transitions are pre-scripted by LLMs. The system does not model the evolution of emotion (e.g., a gradual shift from surprise to anger) but rather concatenates discrete emotional states. This is a significant limitation on the naturalness of the expression and may not generalize to more complex, real-world emotional dynamics.\n- A primary motivation for the complex two-stage design is to overcome the inefficiency of the teacher's multi-round inference. However, the paper does not show any quantitative analysis of inference speed (e.g., Real-Time Factor). Without this result, it is impossible to judge whether the student model provides a meaningful practical benefit. The entire premise of the second stage rests on an unproven assumption of efficiency gain.", "was_revised": true, "final_justification": "Thanks for reviewers' rebuttal, it solve my concerns, I have raised my point to 4.", "rebuttal": "We thank you for your thoughtful feedback and constructive criticisms. We appreciate that you acknowledged the significance of fine-grained expressive control in TTS, and the clarity of our proposed pipeline. Below, we respond to the main concerns raised.\n\n- W1: Reliance on GPT-4o-generated text may limit the diversity and naturalness of emotional\n  transitions.\n\n> A1: We agree that relying on GPT-4o-generated text introduces limitations in the diversity and naturalness of emotional transitions. This concern is valid and has been explicitly acknowledged in the Limitations section of our paper.\n> \n> Our goal is not to evaluate the LLM itself, but to show that a TTS model can achieve word-level expressive control using only non-switching emotion data. In our framework, GPT-4o serves solely as a tool to create pseudo-supervision for transitions absent in the original dataset.\n> \n> To improve the diversity of generated expressions, we independently constructed over 2000 options for each of dialogue settings, contextual situations, and character relationships, allowing for up to 8 billion unique combinations. GPT-4o was prompted with carefully crafted instructions to ensure appropriate emotional flow. For example, the first case on our demo page involves a formal discussion between a leader and an employee about project progress.\n> \n> While GPT-4o may not fully capture the richness of real-world transitions, and this may partly explain our method's performance bottleneck (see Figure 5 in our paper). Our results show that the model still learns to generate smooth and perceptually natural emotional shifts. This demonstrates the practical effectiveness of our approach despite GPT-4o's limitations. In addition, since emotional planning in our method is determined by the sequence of emotion labels, but the detail expressiveness is primarily driven by the emotional speech prompts, our framework does not heavily rely on GPT-4o as a source of emotional intent. For instance, such intent can also be derived from narrative texts such as novels or scripts.\n\n- W2: Discrete emotion concatenation limits the naturalness and flexibility of expression.\n\n> A2: We appreciate the reviewer’s concern and would like to clarify that our method does not inherently suffer from the issue described. When the speech prompt contains different emotions, the model is able to handle such emotional switches and generate coherent and natural speech. Our framework does not rely on artificial concatenation and supports expressive control across distinct emotional states.\n> \n> As demonstrated in our demo page, the model supports speech prompts composed of sentences from different speakers as well as from the same speaker with different emotional tones. This shows that our system can successfully generate speech based on prompts containing emotional changes, even across sentences with varying styles or intensities. Therefore, when the prompt comes from a single utterance that includes an emotional switch, the model is equally capable of producing natural and consistent results.\n> \n> In our experiments, we constructed prompts using segments with discrete emotions due to the lack of datasets annotated with intra-sentence emotional switches. This allowed us to conduct evaluations at scale while simulating diverse emotional scenarios. Despite this limitation, the model still achieves high naturalness and fluency, as supported by both subjective and objective results in Table 1 and Table 2 (Both objective and subjective metrics achieve state-of-the-art performance).\n\n- W3&Q1: The paper lacks quantitative validation of the second-stage model’s efficiency on inference speed.\n\n> A3: The second-stage model is not designed to be faster than the first-stage pipeline. Instead, it brings several practical benefits, including simplified end-to-end inference, reduced error propagation from the multi-stage process, and improved transition consistency by leveraging full contextual emotional prompt, as shown in our experimental results.\n> \n> To address the reviewer’s concern, we conducted a quantitative comparison of inference speed under consistent conditions. All models were evaluated using the same test set (as in Table 1), on a single NVIDIA 2080Ti GPU. Results are shown below:\n> \n> **RTF** (Real-Time Factor): The ratio of total inference time (including prompt processing and target speech generation) to the duration of the generated speech. This metric reflects the real-time performance of the system when handling word-level expressive-controllable speech synthesis.\n> \n> **Token/s**: The number of tokens (including both the prompt and generated tokens) processed per second during inference. This reflects the architectural complexity.\n> \n> | Model | RTF | Token/s |\n> | --- | --- | --- |\n> | CosyVoice2 | 0.70 | 114.97 |\n> | WeSCon (1st) | 0.53 | 114.13 |\n> | WeSCon (2nd) | 0.68 | 109.38 |\n> \n> The first-stage model achieves a lower RTF than CosyVoice2, primarily due to fewer Flow Matching operations when handling multiple emotional transitions. The second-stage model introduces lightweight dynamic attention, which results in only a slight decrease in token/s, indicating that the overall architectural complexity remains similar. However, because the model attends to the all emotional prompts, the input sequence becomes longer during inference, leading to a more noticeable increase in RTF. Despite this, the inference speed remains comparable to CosyVoice2 while providing significantly better expressive smoothness and naturalness, as shown in Table 1 and Table 4. These results demonstrate the practical value of our self-training framework, even though inference efficiency is not its primary objective."}, {"review_id": "fvDXCl5BY0", "reviewer": "Reviewer_LGsJ", "summary": "This paper proposed a word-level emotion control-based zero-shot TTS. Instead of using utterance level emotion prompt to derive the emotional speech, this paper proposed a multi-stage inference method that the word-level emotion embeddings and speaking rate is automatically learnt which were used to help to predict the proper expressive speech. Furthermore, to achieve a end-to-end system, this paper adopted the self-training scheme to use the multi-stage system as a teacher to guide the end-to-end model training. Experiment and results show the proposed method is very promissing in terms of both objective and subjective evalution metrics.", "questions": "Can the emotion transfer across language? For example, if you use English speech as prompt and generate Chinees speeh, can the proposed method still generate controlled expressive speech?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n1, This paper is addrssing a challenging problem of zero-shot expressive text to speech syntheis with an innovative method, which is word-level emotion control. This method has advantage over the utterance-based prompt method as it can have a more precise emotion control, particular for generating speech with changing tone in a single sentence. \n2, The paper is very well-written with a clear clarity.\n3, The techniques are sounding.   \n4, The experiments are comprehensive and the results are very promissing compared to the SOTA methods in the expressive TTS field. The sound quality and naturalness of the samples in the demo page is very impressive. \n\nWeakness\nOne of the small issue is that for the speech rate control, the paper proposed to do either downsampling or nearest-neigbor interpolation. However, the paper did not mention any method to predict the speech length. There is a concern that if you downsample or interpolate the speech samples with an arbitary factor, it could generate unnatural speech.", "was_revised": true, "final_justification": "This paper proposed an innovative word-level emotion control-based zero-shot TTS rather than a conventional utterance-based method. The method is sounding and the results are promissing. In the rebutal stage, the authors clarfied two questions I rasied about generated speech speed and cross lingual emotion transfer, which are reasonable and clear. Based on these, I will keep my original score which is 6-Strong Accept.", "rebuttal": "We sincerely thank you for your positive and encouraging feedback. We are glad that you found our method for word-level emotion control to be innovative and effective, especially in addressing the challenge of generating expressive speech with intra-sentence variation. We also appreciate your recognition of the writing quality, technical soundness, and the strength of our experimental results and audio demos. Below we respond to the specific concern and question raised:\n\n- W1: The paper proposes using downsampling or interpolation to adjust speech rate, but does not describe how target durations are determined. Arbitrary rescaling may result in unnatural speech.\n\n> A1: Thank you for highlighting this important detail. In our system, the speaking rate is not arbitrarily defined but computed through a structured process informed by both LLM outputs and statistics of the non-emotion-transition training dataset:\n> \n> 1. During supervision generation of teacher model, GPT-4o specifies the desired speaking rate for each word (ranging from 0.5 to 2.0, where 1.0 denotes normal speed), as part of the scripted prompt generation (see Appendix C).\n>   \n> 2. For each emotion category, we calculate the average phoneme duration (i.e., average duration per phoneme) from non-emotion-transition training data.\n>   \n> 3. The target phoneme duration is obtained by multiplying the LLM-specified rate with the emotion-specific average duration.\n>   \n> 4. We then compare this target to the phoneme duration of the prompt speech, and apply either linear interpolation or downsampling accordingly.\n>   \n> \n> This approach ensures that duration modification is linguistically informed, emotion-sensitive, and statistically regularized. In experiments, we observed that this method preserves the naturalness and rhythm of generated speech without introducing artifacts. We will include this implementation detail in the appendix of the final version.\n\n- Q1: Can the emotion transfer across language? For example, if you use English speech as prompt and generate Chinese speeh, can the proposed method still generate controlled expressive speech?\n\n> QA1: Our backbone model, CosyVoice2, supports cross-lingual zero-shot TTS. In our preliminary experiments, the first-stage WeSCon model successfully handles cross-lingual intra-sentence emotional variation. The second-stage model can also generate expressive speech in cross-lingual settings, but exhibits noticeable accent-related distortions. This is mainly due to the fact that the student model in Stage 2 was not explicitly trained with cross-lingual supervision or data organization.\n> \n> Theoretically, the student model can achieve comparable cross-lingual performance by incorporating structured multilingual self-training data. However, this extension falls beyond the scope of the current paper. We consider this a promising direction and plan to explore it in future work."}, {"review_id": "oHoelFhVGR", "reviewer": "Reviewer_zpVF", "summary": "This paper proposes the WeSCon framework, which achieves end-to-end word-level emotional expression control for the first time without requiring emotion-annotated data, through a self-training framework and a dynamic emotional attention bias mechanism. The method generates mixed-emotion training data (combining speech segments with different emotional prompts) via a multi-round inference mechanism and designs a dynamic emotional attention bias module to enable pretrained TTS models to support fine-grained word-level emotion and speech rate joint control. Experiments demonstrate that WeSCon achieves state-of-the-art (SOTA) performance in word-level emotional expression under limited data while preserving the zero-shot generation capability of the pretrained model, offering a novel approach for emotional speech synthesis in low-resource scenarios.", "questions": "see Weakness", "limitations": "see Weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Emotional speech synthesis is a critical direction in the field of text-to-speech (TTS), and data scarcity remains a key challenge. This work addresses emotion control without relying on emotion-annotated data, demonstrating notable application value and research significance.\n\n2. The multi-round inference mechanism generates mixed-emotion training data by combining speech with different emotional prompts, eliminating the need for large-scale emotion-annotated data and improving data utilization efficiency. The dynamic emotional attention bias mechanism enables pretrained TTS models to perceive emotion boundaries and speech rate variations, supporting joint control of word-level emotion and speech rate during inference, thereby enhancing expressive capability.\n\n3. The experiments verify the effectiveness of the proposed method in word-level emotional expression control. Comparative results with baseline models demonstrate its performance advantages. Ablation studies further validate the contributions of individual components, reflecting the rationality of the design.\n\nWeaknesses:\n\n1. While high-expressive emotional speech synthesis is an important research direction, the baseline selection in the experiments exhibits bias. Compared models (e.g., CosyVoice, Spark-TTS, F5-TTS) are not specifically designed for emotional synthesis. The sentence-level concatenation strategy may unfairly advantage the proposed method in comparisons, as baseline models generate utterances holistically without boundary artifacts. This asymmetry could affect performance evaluation.\n\n2. The current implementation adjusts speech rate merely by upsampling/downsampling speech tokens without explicitly linking it to emotional expression features (e.g., stress, rhythm, prosodic boundaries). This decoupled design may result in acoustically similar attributes for the same emotion at different speech rates, contradicting linguistic principles (where emotions should exhibit distinct prosodic patterns at varying rates). Further analysis of the emotion-speech rate interaction or supplementary experiments is recommended.\n\n3. The reliance on GPT-4o for generating text and emotion labels introduces definitional ambiguity. The \"reasonable emotion labels\" lack clear criteria. If \"reasonable\" refers to statistical consistency with common scenarios, labels may overfit textual semantics (as modern LLMs can infer basic emotion shifts from context).\n\n4. In my opinion, the application of this proposed method is limited. The acoustic features of the generated speech should come from the semantic understanding of the given text rather than a cherry-picked, complicated prompt waveform. For the scenario where the given emotion prompt conflicts with the semantic information of the given text, this paper did not provide an in-depth analysis. To me, this paper is fancy but lacks practical applicability.", "was_revised": true, "final_justification": "After reading other reviewers' comments and your rebuttal, I have decided to improve my rating to 4, a positive score. I agree that the emotional interpretation does not always follow directly from the literal wording. I think the proposed method is a promising way to synthesize speech with given dynamic emotional expressions.", "rebuttal": "We sincerely thank the reviewer for the detailed and thoughtful feedback. We appreciate your recognition of the novelty, technical design, and experimental rigor of our work, as well as the practical significance of word-level emotional expression control under low-resource conditions. Below we respond point-by-point to the concerns raised.\n\n- W1： On the fairness of baseline comparisons.\n\n> A1: We believe our comparisons are fair and representative.While CosyVoice2, Spark-TTS, IndexTTS, and F5-TTS are primarily designed for general-purpose zero-shot TTS, they are among the state-of-the-art open-source zero-shot TTS models, and all offer strong emotion cloning capabilities. In contrast, open-source models specifically designed for emotional synthesis (e.g., EmotiVoice) do not support speaker cloning, making direct comparisons under a unified evaluation protocol infeasible.\n> \n> Furthermore, as shown in Table 3, our focus is to preserve the strong zero-shot generalization of pretrained TTS while enabling fine-grained expressive controlling. While we acknowledge that differences in model capabilities may introduce some asymmetry, we have followed a consistent evaluation protocol and believe the selected baselines are the most relevant and competitive open-source alternatives for assessing fine-grained emotional expression.\n\n- W2: On the interaction between speaking rate and emotion.\n\n> A2: This is an excellent point. We agree that speech rate and emotion are closely interrelated, and emotional states often exhibit distinct prosodic patterns depending on speaking tempo. We would like to clarify that our method does not assume independent control of speech rate and emotional expression. In fact, our findings support the hypothesis that the two are interdependent. Since both speaking rate and emotional features are provided through the same prompt speech, adjusting the rate through upsampling or downsampling inevitably affects emotional expression as well.\n> \n> As shown in Table 4 (row 3), removing speech rate control leads to noticeable drops in both emotional expression similarity and naturalness (During evaluation, we adjust the target speech to match the target speaking rate before computing emotion similarity, see Appendix F.3, line 769).\n> \n> To further investigate the effect of speaking rate on perceived emotional expression, we conducted an additional experiment (will be added in the appendix). We randomly selected 100 emotional utterances from the test set as prompts and systematically varied the resampling ratio from 0.5 to 2.0 in steps of 0.25, using the same target text (this range was chosen to ensure performance stability; see Appendix B). For each result, we computed the emotion similarity between the generated speech and both the original and rate-adjusted (re-rated) target using Emotion2Vec Score (Emition similarity):\n> \n> | Resampling Ratio | Emotion2Vec Score ↑ | Emotion2Vec Score (re-rated) ↑ |\n> | --- | --- | --- |\n> | 0.5 (Downsampled to half, speed up) | 0.57 | 0.86 |\n> | 0.75 | 0.85 | 0.88 |\n> | 1.0 | 0.90 | 0.90 |\n> | 1.25 | 0.83 | 0.89 |\n> | 1.5 | 0.75 | 0.90 |\n> | 1.75 | 0.68 | 0.91 |\n> | 2.0 (Interpolated to twice, speed down) | 0.51 | 0.87 |\n> \n> The results show that emotional expression similarity drops significantly if the reference is not adjusted for speaking rate, while similarity remains stable when compared against rate-matched references. This confirms that speech rate provides important prosodic cues for emotional expression, consistent with our findings in Section 4.2.2.\n\n- W3: On concerns about GPT-4o-generated text and emotion labels.\n\n> A: We appreciate the reviewer’s insightful question. In our paper, *“reasonable emotion labels”* refer to emotional changes that fit the scenario, context, and character relationships defined in the prompt. Rather than assigning labels manually, we let GPT-4o generate emotional expressions based on structured input such as background, context, and role pairs.\n> \n> To reduce semantic overfitting, we independently prepared over 2000 options for each of backgrounds, contexts, and roles, enabling up to 8 billion combinations through free permutation. This diversity helps guide GPT-4o to produce contextually grounded emotion shifts.\n> \n> We acknowledge that the diversity of generated data is still limited by the LLM’s capacity and the size of our emotion prompt dataset, as discussed in Figure 5. Fully addressing this would require richer emotional corpora, which is beyond the scope of this work. In addition, since emotional planning in our method is determined by the sequence of emotion labels, but the detail expressiveness is primarily driven by the emotional speech prompts, our framework does not heavily rely on GPT-4o as a source of emotional intent. For instance, such intent can also be derived from narrative texts such as novels or scripts.\n> \n> Our goal is to show that, even without emotion-transition data, a pretrained zero-shot TTS model can learn fine-grained expressive control through self-training. We believe that a detailed discussion of GPT-4o’s performance is beyond the scope of our paper.\n\n- W4: On the applicability and practicality of the proposed method and the possible mismatch between prompt emotion and text semantics.\n\n> A: We appreciate the reviewer’s concern regarding the practicality of our method and the possible mismatch between prompt emotion and text semantics.\n> \n> As the reviewer pointed out, emotional expression should reflect the semantic meaning of the text. We fully agree with this view. Our method assumes that the emotional intent has already been determined based on the semantic content, whether through human interpretation or external models (GPT-4o). The speech prompt provides expressive cues such as prosody, rhythm, and speaking rate. These cues help to deliver the intended emotion more effectively and work together with the semantics rather than replacing it.\n> \n> Regarding the possible mismatch between prompt speech emotion and text meaning, we believe such cases often reflect the gap between literal text and the speaker’s intended meaning in context. Emotional interpretation does not always follow directly from the literal wording. For example, the sentence \"What a beautiful day\" may be used sincerely or sarcastically depending on the situation. Similarly, \"I'm fine\" can express calmness, annoyance, or resignation. In these cases, a prompt that seems emotionally different from the surface text can still match the speaker's true intent. Therefore, a surface-level mismatch does not necessarily mean a conflict with semantics.\n> \n> How the emotional intent is obtained, whether through semantic analysis, narrative planning, or other upstream processes, is not the focus of our work. Our goal is to generate word-level expressive speech once the emotion and text are given. Our experiments show that this framework produces natural and coherent speech across a range of emotional contexts. This supports its practical value in real-world applications such as audiobook synthesis, dialogue systems, and storytelling."}, {"review_id": "3Q8msT5qiC", "reviewer": "Reviewer_5szS", "summary": "The paper introduces WeSCon (Word-level Emotion and Speed Control for TTS), a novel two-stage framework to achieve word-level emotion and speaking rate control using a pre-trained zero-shot TTS model (CosyVoice 2) without an explicitly annotated dataset, i.e, without fine-grained emotional information. In Stage one, they train a content aligner (supervised by the output of a forced aligner) and use a multi-inference method with different dynamic speed control mechanisms (interpolating prompts to slow or faster the speaking rate), and a Transition-smoothing strategy (tail-to-head linkage) to guide the model to do word-level expressive synthesis through multi-round inference. However, this method requires multiple inference steps which can be time consuming, thus, they use a teacher-student self training paradigm with Dynamic Emotional Attention Bias (DEAB) (essentially predicting weight for 7 different predefined attention masks during generation) and emotion prediction resulting in end-to-end word-level emotion and speed control for TTS with minimal to no performance drop.", "questions": "None", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strength\n- The paper is well written, logically structured and most design choices are supplemented with ablations resulting in a very through experimentation.\n- The paper requires a small utterance-level annotated dataset and can still achieve good quality word-level emotion and speed control, which is especially useful, as the authors mention that such datasets can be expensive to obtain or annotate.\n- The model consistently outperforms strong baselines, and the effect of the dynamic speed control mechanism, transition smoothing, and DEAB is also objectively ablated, showing their benefits clearly.\n- Again, for the experiments, a comprehensive and diverse set of objective metrics were used to evaluate all the models, providing robust evaluations.  \n\n### Weakness\n* The subjective evaluations have overlapping confidence intervals, resulting in insignificant differences between comparing systems because of a small number of evaluators.\n* Limited emotional granularity (mainly because of available datasets) as we can sample only from a discrete number of emotions and not walk in a continuous space.\n* Training overhead, to train despite of use of a small dataset, one has to do multi-round inference, do force alignment, generate synthetic data (from an LLM), train CosyVoice 2, etc. However, the stage 2 training helps mitigate it.", "was_revised": true, "final_justification": "The authors addressed my questions and pointed out weaknesses.", "rebuttal": "We are very encouraged by your recognition of our paper's clarity, technical design, and comprehensive experimentation. We are also grateful for your constructive observations, which help us further clarify our contributions and limitations. Below we respond to each of the concerns raised.\n\n- W1: On subjective evaluation confidence intervals.\n\n> A1: Thank you for the observation. While some subjective scores, such as EMOS, show overlapping confidence intervals, this is common in MOS evaluations with a limited number of raters. In our case, 15 professionally trained evaluators assessed 20 comparison groups, each consisting of 5 samples. This setup represents a typical configuration that balances statistical reliability and practical cost. Overlap in confidence intervals does not necessarily imply that the systems are indistinguishable, particularly when the mean scores consistently favor our model across all perceptual dimensions, including SPMOS, SMOS, NMOS, and EMOS.\n> \n> These trends are also strongly supported by objective results in Table 1, where our model outperforms all baselines across expressive metrics. The ablation studies further confirm that each component contributes to measurable gains in both control and expressiveness.\n\n- W2: On limited emotional granularity, as we can sample only from a discrete number of emotions and not walk in a continuous space.\n\n> A2: We appreciate the reviewer’s insightful comment. While our test set is constructed using discrete emotion labels, the model is not inherently restricted to fixed emotional categories. When the prompt speech contains natural emotional variation, including subtle or gradual shifts, the model is able to capture and reproduce these changes in a coherent and expressive manner.\n> \n> As shown on our demo page, the system effectively handles prompts that include emotionally diverse segments, whether from different speakers or from the same speaker expressing different emotional tones. The resulting speech demonstrates smooth transitions and appropriate emotional flow, suggesting that the model can respond flexibly to emotional variation in realistic scenarios.\n> \n> The use of discrete labels in our evaluation was mainly intended to support large-scale testing and controllable supervision, in light of the lack of datasets annotated with continuous or intra-sentence emotional changes. We agree that finer-grained or continuous emotional modeling is a valuable direction, and we believe our framework offers a strong foundation for future extensions in this area.\n\n- W3: On training overhead. However, the stage 2 training helps mitigate it.\n\n> A3: We appreciate the reviewer’s thoughtful observation. While our two-stage self-training framework introduces additional steps, including multi-round inference, forced alignment, and synthetic data generation, this design is primarily motivated by the lack of labeled emotion-transition data. The first stage plays a crucial role in constructing pseudo-supervised signals that activate word-level expressive control in a pretrained zero-shot TTS model. As the reviewer rightly noted, the second-stage student model successfully absorbs this supervision into an end-to-end architecture. This not only simplifies inference but also mitigates potential cascading errors from the teacher pipeline while maintaining controllability.\n> \n> In terms of training overhead, although the overall pipeline involves several components, each step is relatively efficient. For example, as shown in Appendix H, the content aligner can be trained quickly with a small amount of data, due to the strong text-speech alignment capabilities of pretrained TTS models. Additionally, the LLM-based script generation is not time-consuming, as the total number of emotion scripts required remains manageable even when covering thousands of hours of speech. Finally, the inference process of the teacher model can be accelerated through parallelization. With eight A100 GPUs and vLLM support, the system can synthesize approximately 40 hours of supervision data per hour, enabling efficient data generation at scale."}]

Author rebuttal:

Meta-review: After rebuttal, all reviewers recommend acceptance based on motivation, novelty and quality of results.
Decision: Accept (spotlight)