GenAI4ELab/papercli-papers huggingface dataset:
Speaker-Aware Multi-Task Learning for Speech Emotion Recognition
Authors: Xiaohan Shi; Xingfeng Li; Tomoki Toda
Venue: Interspeech 2025 (main)

Abstract: Speaker representations play a crucial role in achieving accurate speech emotion recognition (SER). Previous studies have primarily relied on generic speaker recognition (SR) models to extract speaker representations. However, these approaches struggle in speaker-dependent SER tasks, as they fail to capture speaker-specific characteristics effectively. To address this limitation, we propose a Speaker-Aware Multi-Task (SAMT) model, which is designed to effectively model speaker-specific and emotion-specific representations for SER. Additionally, we introduce a speaker-emotion disentanglement loss to explicitly separate speaker and emotion information, further enhancing speaker representation. Extensive experiments demonstrate the effectiveness of our approach, achieving performance gains of 1.76% in speaker-dependent and 1.99% in speaker-independent settings over the baseline. Moreover, the speaker-emotion disentanglement loss further improves SER performance.
Multimodal Emotion Diarization: Frame-Wise Integration of Text and Audio Representations
Authors: Ziv Tamir; Thomas Thebaud; Jesus Villalba; Najim Dehak; Oren Kurland
Venue: Interspeech 2025 (main)

Abstract: Speech emotion diarization (SED) is the task of segmenting an audio stream into time-continuous emotional states, akin to speaker diarization but for emotions. While traditional speech emotion recognition (SER) assigns a single emotion label to a given utterance, real-world conversations exhibit dynamic emotional transitions that require a more granular approach. In this work, we propose a novel multimodal SED framework that uses frame-wise integration of text and audio embeddings using temporal synchronization and direct concatenation, followed by a context-aware sliding window smoothing mechanism. Audio representations are extracted using WavLM, and EmoBERTa generates text embeddings aligned to spoken words.  We evaluate our approach using Emotion Diarization Error Rate (EDER), a metric designed for SED. Experimental results show that our proposed method significantly improves diarization performance, with respect to score fusion and cross-attention methods yielding an EDER of 25%.
Analysis of Phonetic Level Similarities Across Languages in Emotional Speech
Authors: Pravin Mote; Abinay Reddy Naini; Donita Robinson; Elizabeth Richerson; Carlos Busso
Venue: Interspeech 2025 (main)

Abstract: Unsupervised domain adaptation offers significant potential for cross-lingual speech emotion recognition (SER). Most relevant studies have addressed this problem as a domain mismatch without considering phonetical emotional differences across languages. Our study explores universal discrete speech units obtained with vector quantization of wavLM representations from emotional speech in English, Taiwanese Mandarin, and Russian. We estimate cluster-wise distributions of quantized wavLM frames to quantify phonetic commonalities and differences across languages, vowels, and emotions. Our findings indicate that certain emotion-specific phonemes exhibit cross-linguistic similarities. The distribution of vowels varies with emotional content. Certain vowels across languages show close distributional proximity, offering anchor points for cross-lingual domain adaptation. We also propose and validate a method to quantify phoneme distribution similarities across languages.
Label Semantic-Driven Contrastive Learning for Speech Emotion Recognition
Authors: Jiaxi Hu; Leyuan Qu; Haoxun Li; Taihao Li
Venue: Interspeech 2025 (main)

Abstract: Speech Emotion Recognition (SER) is crucial for human-computer interaction applications. However, SER remains a challenging task due to limited datasets and ambiguous emotion boundaries. While Self-Supervised Learning (SSL) models have demonstrated considerable success in speech processing tasks, existing approaches still struggle to distinguish subtle emotional variations. In this paper, we propose a novel Label Semantic-driven Contrastive Learning framework (LaSCL) that integrates emotion label semantic embeddings into speech representation learning. Our method uses label embeddings as semantic anchors to explicitly model relationships between emotions and employ a label divergence loss to better establish clearer emotion boundaries. Experiments on the widely used IEMOCAP benchmark indicate that LaSCL achieves state-of-the-art performance compared with previous methods.
Pitch Contour Model (PCM) with Transformer Cross-Attention for Speech Emotion Recognition
Authors: Minji Ryu; Ji-Hyeon Hur; Sung Heuk Kim; Gahgene Gweon
Venue: Interspeech 2025 (main)

Abstract: Pitch is important for distinguishing emotional states through intonation. To incorporate pitch contour patterns into Speech Emotion Recognition (SER) task, we propose the Pitch Contour Model (PCM), which integrates pitch features with Transformer-based speech representations. PCM processes pitch features via linear embedding and combines them with Wav2Vec 2.0 extracted features using cross-attention. Experimental results show that PCM enhances SER performance, achieving state-of-the-art (SOTA) Valence-Arousal-Dominance (V-A-D) scores with V:0.627, avg:0.571 in MSP-Podcast v1.11 and V:0.646, A:0.744, D:0.557 in IEMOCAP datasets. We observe that the effect of z-score normalization on pitch varies across datasets, with lower pitch variability conditions benefiting more from raw pitch values. Furthermore, our study suggests how pretraining and finetuning language mismatches, between English and Korean, affect the choice between CNN-based and linear embeddings for pitch representation.
EATS-Speech: Emotion-Adaptive Transformation and Priority Synthesis for Zero-Shot Text-to-Speech
Authors: Jingyuan Xing; Zhipeng Li; Shuaiqi Chen; Xiaofen Xing; Xiangmin Xu
Venue: Interspeech 2025 (main)

Abstract: Zero-shot text-to-speech (TTS) supports diverse speech synthesis without speaker-specific data but struggles to accurately transfer emotions from reference to target text. Traditional approaches treat emotion as part of a global style, leading to inconsistent emotional expressiveness. To address this, we propose EATS-Speech, an Emotion-Adaptive Transformation Synthesis framework. EATS-Speech employs Emotion Priority Synthesis through a parallel pipeline that decomposes speech into non-emotion style, emotion, and content. It prioritizes emotion generation to enhance expressiveness. Furthermore, it introduces Emotion-Adaptive Transformation Synthesis, where an LLM-based converter learns text-emotion mapping patterns from the reference speech and transfers them to the target text. Experiments on the LibriTTS dataset demonstrate the improvements in emotional expressiveness and accurate emotion adaptation.
Voice Impression Control in Zero-Shot TTS
Authors: Kenichi Fujita; Shota Horiguchi; Yusuke Ijima
Venue: Interspeech 2025 (main)

Abstract: Para-/non-linguistic information in speech is pivotal in shaping the listeners' impression. Although zero-shot text-to-speech (TTS) has achieved high speaker fidelity, modulating subtle para-/non-linguistic information to control perceived voice characteristics, i.e., impressions, remains challenging. We have therefore developed a voice impression control method in zero-shot TTS that utilizes a low-dimensional vector to represent the intensities of various voice impression pairs (e.g., darkâbright). The results of both objective and subjective evaluations have demonstrated our method's effectiveness in impression control. Furthermore, generating this vector via a large language model enables target-impression generation from a natural language description of the desired impression, thus eliminating the need for manual optimization.
EME-TTS: Unlocking the Emphasis and Emotion Link in Speech Synthesis
Authors: Haoxun Li; Leyuan Qu; Jiaxi Hu; Taihao Li
Venue: Interspeech 2025 (main)

Abstract: In recent years, emotional Text-to-Speech (TTS) synthesis and emphasis-controllable speech synthesis have advanced significantly. However, their interaction remains underexplored. We propose Emphasis Meets Emotion TTS (EME-TTS), a novel framework designed to address two key research questions: (1) how to effectively utilize emphasis to enhance the expressiveness of emotional speech, and (2) how to maintain the perceptual clarity and stability of target emphasis across different emotions. EME-TTS employs weakly supervised learning with emphasis pseudo-labels and variance-based emphasis features. Additionally, the proposed Emphasis Perception Enhancement (EPE) block enhances the interaction between emotional signals and emphasis positions. Experimental results show that EME-TTS, when combined with large language models for emphasis position prediction, enables more natural emotional speech synthesis while preserving stable and distinguishable target emphasis across emotions. Synthesized samples are available online.
DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speech
Authors: Deok-Hyeon Cho; Hyung-Seok Oh; Seung-Bin Kim; Seong-Whan Lee
Venue: Interspeech 2025 (main)

Abstract: Cross-speaker emotion transfer in speech synthesis relies on extracting speaker-independent emotion embeddings for accurate emotion modeling without retaining speaker traits. However, existing timbre compression methods fail to fully separate speaker and emotion characteristics, causing speaker leakage and degraded synthesis quality. To address this, we propose DiEmo-TTS, a self-supervised distillation method to minimize emotional information loss and preserve speaker identity. We introduce cluster-driven sampling and information perturbation to preserve emotion while removing irrelevant factors. To facilitate this process, we propose an emotion clustering and matching approach using emotional attribute prediction and speaker embeddings, enabling generalization to unlabeled data. Additionally, we designed a dual conditioning transformer to integrate style features better. Experimental results confirm the effectiveness of our method in learning speaker-irrelevant emotion embeddings.
Spotlight-TTS: Spotlighting the Style via Voiced-Aware Style Extraction and Style Direction Adjustment for Expressive Text-to-Speech
Authors: Nam-Gyu Kim; Deok-Hyeon Cho; Seung-Bin Kim; Seong-Whan Lee
Venue: Interspeech 2025 (main)

Abstract: Recent advances in expressive text-to-speech (TTS) have introduced diverse methods based on style embedding extracted from reference speech. However, synthesizing high-quality expressive speech remains challenging. We propose Spotlight-TTS, which exclusively emphasizes style via voiced-aware style extraction and style direction adjustment. Voiced-aware style extraction focuses on voiced regions highly related to style while maintaining continuity across different speech regions to improve expressiveness. We adjust the direction of the extracted style for optimal integration into the TTS model, which improves speech quality. Experimental results demonstrate that Spotlight-TTS achieves superior performance compared to baseline models in terms of expressiveness, overall speech quality, and style transfer capability. Our audio samples are publicly available.
Speaker-agnostic Emotion Vector for Cross-speaker Emotion Intensity Control
Authors: Masato Murata; Koichi Miyazaki; Tomoki Koriyama
Venue: Interspeech 2025 (main)

Abstract: Cross-speaker emotion intensity control aims to generate emotional speech of a target speaker with desired emotion intensities using only their neutral speech. A recently proposed method, emotion arithmetic, achieves emotion intensity control using a single-speaker emotion vector. Although this prior method has shown promising results in the same-speaker setting, it lost speaker consistency in the cross-speaker setting due to mismatches between the emotion vector of the source and target speakers. To overcome this limitation, we propose a speaker-agnostic emotion vector designed to capture shared emotional expressions across multiple speakers. This speaker-agnostic emotion vector is applicable to arbitrary speakers. Experimental results demonstrate that the proposed method succeeds in cross-speaker emotion intensity control while maintaining speaker consistency, speech quality, and controllability, even in the unseen speaker case.
SA-RAS: Speaker-Aware Style Retrieval Augmented Generation for Expressive Zero-Shot Text-to-Speech Synthesis
Authors: Xueru Li; Jingyuan Xing; Xiaofen Xing; Zhipeng Li; Xiangmin Xu
Venue: Interspeech 2025 (main)

Abstract: Zero-shot text-to-speech (TTS) models can clone an unseen speaker by reference speech. Given multiple reference speeches, Retrieval Augmented Generation (RAG) shows potential in enhancing synthesis quality. The existing method faces two key limitations: semantically-biased retrieval tends to introduce style mismatch, and the deterministic mapping between text and speech fails to fully capture the unique style of the speaker. To solve these, we propose a Speaker-Aware Retrieval Augmented TTS Synthesis (SA-RAS) model. It consists of a two-stage TTS model, Split-TTS and a zero-shot CLAP. The former embeds style and timbre separately to obtain a style space for retrieval. The latter optimizes the correlation between text and style by incorporating speaker information into the retrieval model. Experiments prove that SA-RAS boosts the style expressiveness of synthesized speech.
DiffEmotionVC: A Dual-Granularity Disentangled Diffusion Framework for Any-to-Any Emotional Voice Conversion
Authors: Xiaosu Su; BoWen Yang; Xiaowei Yi; Yun Cao
Venue: Interspeech 2025 (main)

Abstract: Emotional Voice Conversion (EVC) plays a vital role in improving human-computer interaction but faces challenges due to the complexity of emotion features, which are entangled with speaker and content characteristics. To overcome these challenges, we propose DiffEmotionVC, a diffusion-based framework for any-to-any EVC. Our approach integrates a dual-granularity emotion encoder that captures both utterance-level emotional context and frame-level acoustic details. It also employs an orthogonality-constrained condition encoder that disentangles emotion features through gated cross-attention while preserving feature independence with an orthogonal loss. Additionally, multi-objective diffusion training enhances both reconstruction fidelity and emotion discriminability via contrastive learning. Experimental results show a UTMOS score of 4.04 and 80% emotion recognition accuracy, demonstrating the framework's effectiveness in speech quality and optimizing emotional expression.
ZSDEVC: Zero-Shot Diffusion-based Emotional Voice Conversion with Disentangled Mechanism
Authors: Hsing-Hang Chou; Yun-Shao Lin; Ching-Chin Sung; Yu Tsao; Chi-Chun Lee
Venue: Interspeech 2025 (main)

Abstract: The human voice conveys not just words but also emotional states and individuality. Emotional voice conversion (EVC) modifies emotional expressions while preserving linguistic content and speaker identity, improving applications like human-machine interaction. While deep learning has advanced EVC models for specific target speakers on well-crafted emotional datasets, existing methods often face issues with emotion accuracy and speech distortion. In addition, the zero-shot scenario, in which emotion conversion is applied to unseen speakers, remains underexplored. This work introduces a novel diffusion framework with disentangled mechanisms and expressive guidance, trained on a large emotional speech dataset and evaluated on unseen speakers across in-domain and out-of-domain datasets. Experimental results show that our method produces expressive speech with high emotional accuracy, naturalness, and quality, showcasing its potential for broader EVC applications.
MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt
Authors: Zhichao Wu; Yueteng Kang; Songjun Cao; Long Ma; Qiulin Li; Qun Yang
Venue: Interspeech 2025 (main)

Abstract: Most existing Zero-Shot Text-To-Speech (ZS-TTS) systems generate the unseen speech based on single prompt, such as reference speech or text descriptions, which limits their flexibility. We propose a customized emotion ZS-TTS system based on multi-modal prompt. The system disentangles speech into the content, timbre, emotion and prosody, allowing emotion prompts to be provided as text, image or speech. To extract emotion information from different prompts, we propose a multi-modal prompt emotion encoder. Additionally, we introduce an prosody predictor to fit the distribution of prosody and propose an emotion consistency loss to preserve emotion information in the predicted prosody. A diffusion-based acoustic model is employed to generate the target mel-spectrogram. Both objective and subjective experiments demonstrate that our system outperforms existing systems in terms of naturalness and similarity.
MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognition
Authors: Yinfeng Xia; Huiyan Li; Chenyang Le; Manhong Wang; Yutao Sun; Xingyang Ma; Yanmin Qian
Venue: Interspeech 2025 (main)

Abstract: Applying large pre-trained speech models like Whisper has shown promise in reducing training costs for various speech tasks. However, integrating these models into streaming systems remains a challenge. This paper presents a novel prefix-to-prefix training framework for streaming recognition by fine-tuning the Whisper. We introduce the Continuous Integrate-and-Fire mechanism to establish a quasi-monotonic alignment between continuous speech sequences and discrete text tokens. Additionally, we design Monotonic Finite Look-ahead Attention, allowing each token to attend to infinite left-context and finite right-context from the speech sequences. We also employ the wait-k decoding strategy to simplify the decoding process while ensuring consistency between training and testing. Our theoretical analysis and experiments demonstrate that this approach achieves a controllable trade-off between latency and quality, making it suitable for various streaming applications.
Delayed-KD: Delayed Knowledge Distillation based CTC for Low-Latency Streaming ASR
Authors: Longhao Li; Yangze Li; Hongfei Xue; Jie Liu; Shuai Fang; Kai Wang; Lei Xie
Venue: Interspeech 2025 (main)

Abstract: CTC-based streaming ASR has gained significant attention in real-world applications but faces two main challenges: accuracy degradation in small chunks and token emission latency. To mitigate these challenges, we propose Delayed-KD, which applies delayed knowledge distillation on CTC posterior probabilities from a non-streaming to a streaming model. Specifically, with a tiny chunk size, we introduce a Temporal Alignment Buffer (TAB) that defines a relative delay range compared to the non-streaming teacher model to align CTC outputs and mitigate non-blank token mismatches. Additionally, TAB enables fine-grained control over token emission delay. Experiments on 178-hour AISHELL-1 and 10,000-hour WenetSpeech Mandarin datasets show consistent superiority of Delayed-KD. Impressively, Delayed-KD at 40 ms latency achieves a lower character error rate (CER) of 5.42% on AISHELL-1, comparable to the competitive U2++ model running at 320 ms latency.
Parameter-efficient Fine-tuning of Conformer-based Streaming Speech Recognition into Non-streaming Models
Authors: Yunjae Nam; Jeong U Han; Kiyeon Kim; Jaemin Lim
Venue: Interspeech 2025 (main)

Abstract: In this paper, we propose a parameter-efficient fine-tuning method for converting streaming speech recognition models into non-streaming models. We introduce the Full-Context Adapter, a modified residual adapter designed to incorporate full-context information, enabling non-streaming conversion while keeping the pre-trained streaming model. The proposed method integrates the Full-Context Adapter into a Conformer-based RNN-Transducer streaming model, allowing it to operate as a non-streaming model. We conduct experiments to evaluate the effectiveness of our method, demonstrating that it achieves comparable performance to a non-streaming baseline model while requiring <10% additional parameters. Furthermore, we analyze optimal hyperparameter configurations and explore performance enhancement strategies. Our results show that the proposed approach provides a simple yet effective solution for efficiently operating both streaming and non-streaming ASR.
On-device Streaming Discrete Speech Units
Authors: Kwanghee Choi; Masao Someki; Emma Strubell; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: Discrete speech units (DSUs) are derived from clustering the features of self-supervised speech models (S3Ms). DSUs offer significant advantages for on-device streaming speech applications due to their rich phonetic information, high transmission efficiency, and seamless integration with large language models. However, conventional DSU-based approaches are impractical as they require full-length speech input and computationally expensive S3Ms. In this work, we reduce both the attention window and the model size while preserving the effectiveness of DSUs. Our results demonstrate that we can reduce floating-point operations (FLOPs) by 50% with only a relative increase of 6.5% in character error rate (CER) on the ML-SUPERB 1h dataset. These findings highlight the potential of DSUs for real-time speech processing in resource-constrained environments.
Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding
Authors: Haoran Zhou; Xingchen Song; Brendan Fahy; Qiaochu Song; Binbin Zhang; Zhendong Peng; Anshul Wadhawan; Denglin Jiang; Apurv Verma; Vinay Ramesh; Srivas Prasad; Michele M. Franceschini
Venue: Interspeech 2025 (main)

Abstract: OpenAI Whisper is a family of robust Automatic Speech Recognition (ASR) models trained on 680,000 hours of audio. However, its encoder-decoder architecture, trained with a sequence-to-sequence objective, lacks native support for streaming ASR. In this paper, we fine-tune Whisper for streaming ASR using the WeNet toolkit by adopting a Unified Two-pass (U2) structure. We introduce an additional Connectionist Temporal Classification (CTC) decoder trained with causal attention masks to generate streaming partial transcripts, while the original Whisper decoder reranks these partial outputs. Our experiments on LibriSpeech and an earnings call dataset demonstrate that, with adequate fine-tuning data, Whisper can be adapted into a capable streaming ASR model. We also introduce a hybrid tokenizer approach, which uses a smaller token space for the CTC decoder while retaining Whisperâs original token space for the attention decoder, resulting in improved data efficiency and generalization.
Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization
Authors: Luong Ho; Khanh Le; Vinh Pham; Bao Nguyen; Tan Tran; Duc Chau
Venue: Interspeech 2025 (main)

Abstract: Inverse Text Normalization (ITN) is crucial for converting spoken Automatic Speech Recognition (ASR) outputs into well-formatted written text, enhancing both readability and usability. Despite its importance, the integration of streaming ITN within streaming ASR remains largely unexplored due to challenges in accuracy, efficiency, and adaptability, particularly in low-resource and limited-context scenarios. In this paper, we introduce a streaming pretrained language model for ITN, leveraging pretrained linguistic representations for improved robustness. To address streaming constraints, we propose Dynamic Context-Aware during training and inference, enabling adaptive chunk size adjustments and the integration of right-context information. Experimental results demonstrate that our method achieves accuracy comparable to non-streaming ITN and surpasses existing streaming ITN models on a Vietnamese dataset, all while maintaining low latency, ensuring seamless integration into ASR systems.