Title: Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis
Venue: NeurIPS 2025
Accepted: yes
Keywords: Emotion Control

Speech Rate Control

Controllable Speech Synthesis

Word-Level Control
Abstract: While emotional text-to-speech (TTS) has made significant progress, most existing research remains limited to utterance-level emotional expression and fails to support word-level control. Achieving word-level expressive control poses fundamental challenges, primarily due to the complexity of modeling multi-emotion transitions and the scarcity of annotated datasets that capture intra-sentence emotional and prosodic variation. In this paper, we propose WeSCon, the first self-training framework that enables word-level control of both emotion and speaking rate in a pretrained zero-shot TTS model, without relying on datasets containing intra-sentence emotion or speed transitions.
Our method introduces a transition-smoothing strategy and a dynamic speed control mechanism to guide the pretrained TTS model in performing word-level expressive synthesis through a multi-round inference process. To further simplify the inference, we incorporate a dynamic emotional attention bias mechanism and fine-tune the model via self-training, thereby activating its ability for word-level expressive control in an end-to-end manner. Experimental results show that WeSCon effectively overcomes data scarcity, achieving state-of-the-art performance in word-level emotional expression control while preserving the strong zero-shot synthesis capabilities of the original TTS model.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces WeSCon (Word-level Emotion and Speed Control for TTS), a novel two-stage framework to achieve word-level emotion and speaking rate control using a pre-trained zero-shot TTS model (CosyVoice 2) without an explicitly annotated dataset, i.e, without fine-grained emotional information. In Stage one, they train a content aligner (supervised by the output of a forced aligner) and use a multi-inference method with different dynamic speed control mechanisms (interpolating prompts to slow or faster the speaking rate), and a Transition-smoothing strategy (tail-to-head linkage) to guide the model to do word-level expressive synthesis through multi-round inference. However, this method requires multiple inference steps which can be time consuming, thus, they use a teacher-student self training paradigm with Dynamic Emotional Attention Bias (DEAB) (essentially predicting weight for 7 different predefined attention masks during generation) and emotion prediction resulting in end-to-end word-level emotion and speed control for TTS with minimal to no performance drop.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The paper is well written, logically structured and most design choices are supplemented with ablations resulting in a very through experimentation.\n- The paper requires a small utterance-level annotated dataset and can still achieve good quality word-level emotion and speed control, which is especially useful, as the authors mention that such datasets can be expensive to obtain or annotate.\n- The model consistently outperforms strong baselines, and the effect of the dynamic speed control mechanism, transition smoothing, and DEAB is also objectively ablated, showing their benefits clearly.\n- Again, for the experiments, a comprehensive and diverse set of objective metrics were used to evaluate all the models, providing robust evaluations.  \n\n### Weakness\n* The subjective evaluations have overlapping confidence intervals, resulting in insignificant differences between comparing systems because of a small number of evaluators.\n* Limited emotional granularity (mainly because of available datasets) as we can sample only from a discrete number of emotions and not walk in a continuous space.\n* Training overhead, to train despite of use of a small dataset, one has to do multi-round inference, do force alignment, generate synthetic data (from an LLM), train CosyVoice 2, etc. However, the stage 2 training helps mitigate it.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the WeSCon framework, which achieves end-to-end word-level emotional expression control for the first time without requiring emotion-annotated data, through a self-training framework and a dynamic emotional attention bias mechanism. The method generates mixed-emotion training data (combining speech segments with different emotional prompts) via a multi-round inference mechanism and designs a dynamic emotional attention bias module to enable pretrained TTS models to support fine-grained word-level emotion and speech rate joint control. Experiments demonstrate that WeSCon achieves state-of-the-art (SOTA) performance in word-level emotional expression under limited data while preserving the zero-shot generation capability of the pretrained model, offering a novel approach for emotional speech synthesis in low-resource scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Emotional speech synthesis is a critical direction in the field of text-to-speech (TTS), and data scarcity remains a key challenge. This work addresses emotion control without relying on emotion-annotated data, demonstrating notable application value and research significance.\n\n2. The multi-round inference mechanism generates mixed-emotion training data by combining speech with different emotional prompts, eliminating the need for large-scale emotion-annotated data and improving data utilization efficiency. The dynamic emotional attention bias mechanism enables pretrained TTS models to perceive emotion boundaries and speech rate variations, supporting joint control of word-level emotion and speech rate during inference, thereby enhancing expressive capability.\n\n3. The experiments verify the effectiveness of the proposed method in word-level emotional expression control. Comparative results with baseline models demonstrate its performance advantages. Ablation studies further validate the contributions of individual components, reflecting the rationality of the design.\n\nWeaknesses:\n\n1. While high-expressive emotional speech synthesis is an important research direction, the baseline selection in the experiments exhibits bias. Compared models (e.g., CosyVoice, Spark-TTS, F5-TTS) are not specifically designed for emotional synthesis. The sentence-level concatenation strategy may unfairly advantage the proposed method in comparisons, as baseline models generate utterances holistically without boundary artifacts. This asymmetry could affect performance evaluation.\n\n2. The current implementation adjusts speech rate merely by upsampling/downsampling speech tokens without explicitly linking it to emotional expression features (e.g., stress, rhythm, prosodic boundaries). This decoupled design may result in acoustically similar attributes for the same emotion at different speech rates, contradicting linguistic principles (where emotions should exhibit distinct prosodic patterns at varying rates). Further analysis of the emotion-speech rate interaction or supplementary experiments is recommended.\n\n3. The reliance on GPT-4o for generating text and emotion labels introduces definitional ambiguity. The \"reasonable emotion labels\" lack clear criteria. If \"reasonable\" refers to statistical consistency with common scenarios, labels may overfit textual semantics (as modern LLMs can infer basic emotion shifts from context).\n\n4. In my opinion, the application of this proposed method is limited. The acoustic features of the generated speech should come from the semantic understanding of the given text rather than a cherry-picked, complicated prompt waveform. For the scenario where the given emotion prompt conflicts with the semantic information of the given text, this paper did not provide an in-depth analysis. To me, this paper is fancy but lacks practical applicability.\n\nQUESTIONS:\nsee Weakness\n\nLIMITATIONS:\nsee Weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a word-level emotion control-based zero-shot TTS. Instead of using utterance level emotion prompt to derive the emotional speech, this paper proposed a multi-stage inference method that the word-level emotion embeddings and speaking rate is automatically learnt which were used to help to predict the proper expressive speech. Furthermore, to achieve a end-to-end system, this paper adopted the self-training scheme to use the multi-stage system as a teacher to guide the end-to-end model training. Experiment and results show the proposed method is very promissing in terms of both objective and subjective evalution metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1, This paper is addrssing a challenging problem of zero-shot expressive text to speech syntheis with an innovative method, which is word-level emotion control. This method has advantage over the utterance-based prompt method as it can have a more precise emotion control, particular for generating speech with changing tone in a single sentence. \n2, The paper is very well-written with a clear clarity.\n3, The techniques are sounding.   \n4, The experiments are comprehensive and the results are very promissing compared to the SOTA methods in the expressive TTS field. The sound quality and naturalness of the samples in the demo page is very impressive. \n\nWeakness\nOne of the small issue is that for the speech rate control, the paper proposed to do either downsampling or nearest-neigbor interpolation. However, the paper did not mention any method to predict the speech length. There is a concern that if you downsample or interpolate the speech samples with an arbitary factor, it could generate unnatural speech.\n\nQUESTIONS:\nCan the emotion transfer across language? For example, if you use English speech as prompt and generate Chinees speeh, can the proposed method still generate controlled expressive speech?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces WeSCon, a novel two-stage self-training framework for enabling word-level control over both emotion and speaking rate in a zero-shot text-to-speech (TTS) model. The primary challenge addressed is the scarcity of datasets with fine-grained, intra-sentence emotional annotations. Experimental results, on both English and Chinese, demonstrate that WeSCon achieves state-of-the-art performance in word-level expressive control. It significantly improves transition smoothness and emotional accuracy compared to strong baselines, all while preserving the original model's zero-shot speaker cloning capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper addresses an important challenge in TTS. Achieving fine-grained, word-level control over expressive attributes is a logical and necessary next step for creating more natural-sounding synthetic speech.\n- The paper is generally well-written and structured. The figures and descriptions of the architecture provide a clear high-level understanding of the proposed pipeline.\n\nWeaknesses\n- The entire self-training process is critically dependent on texts generated by a proprietary model GPT-4o. In other words, the system can only learn emotional transitions that the LLM can imagine and articulate, potentially limiting its diversity and introducing stylistic artifacts.\n- The title claims \"Word-Level Emotional Expression Control,\" which suggests a high degree of flexibility. However, the control is limited to a small, predefined set of discrete emotions, and the transitions are pre-scripted by LLMs. The system does not model the evolution of emotion (e.g., a gradual shift from surprise to anger) but rather concatenates discrete emotional states. This is a significant limitation on the naturalness of the expression and may not generalize to more complex, real-world emotional dynamics.\n- A primary motivation for the complex two-stage design is to overcome the inefficiency of the teacher's multi-round inference. However, the paper does not show any quantitative analysis of inference speed (e.g., Real-Time Factor). Without this result, it is impossible to judge whether the student model provides a meaningful practical benefit. The entire premise of the second stage rests on an unproven assumption of efficiency gain.\n\nQUESTIONS:\nThe paper's core design choice—a teacher-student framework—is justified by the need for a more efficient inference process. This claim is currently unsubstantiated. Can you provide a quantitative comparison of inference speed (e.g., Real-Time Factor) for: (a) the teacher model, (b) the final student model, and (c) the baseline models. Without this data, the central motivation for the entire second stage of the work is unconvincing.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}