GenAI4ELab/papercli-papers huggingface dataset:
Token-Level Logits Matter: A Closer Look at Speech Foundation Models for Ambiguous Emotion Recognition
Authors: Jule Valendo Halim; Siyi Wang; Hong Jia; Ting Dang
Venue: Interspeech 2025 (main)

Abstract: Emotional intelligence in conversational AI is crucial across domains like human-computer interaction. While numerous models have been developed, they often overlook the complexity and ambiguity inherent in human emotions. In the era of large speech foundation models (SFMs), understanding their capability in recognizing ambiguous emotions is essential for the development of next-generation emotion-aware models. This study examines the effectiveness of SFMs in ambiguous emotion recognition. We designed prompts for ambiguous emotion prediction and introduced two novel approaches to infer ambiguous emotion distributions: one analysing generated text responses and the other examining the internal processing of SFMs through token-level logits. Our findings suggest that while SFMs may not consistently generate accurate text responses for ambiguous emotions, they can interpret such emotions at the token level based on prior knowledge, demonstrating robustness across different prompts.
Assessing the feasibility of Large Language Models for detecting micro-behaviors in team interactions during space missions
Authors: Ankush Raut; Projna Paromita; Sydney Begerowski; Suzanne Bell; Theodora Chaspari
Venue: Interspeech 2025 (main)

Abstract: We explore the feasibility of large language models (LLMs) in detecting subtle expressions of micro-behaviors in team conversations using transcripts collected during simulated space missions. Specifically, we examine zero-shot classification, fine-tuning, and paraphrase-augmented fine-tuning with encoder-only sequence classification LLMs, as well as few-shot text generation with decoder-only causal language modeling LLMs, to predict the micro-behavior associated with each conversational turn (i.e., dialogue). Our findings indicate that encoder-only LLMs, such as RoBERTa and DistilBERT, struggled to detect underrepresented micro-behaviors, particularly discouraging speech, even with weighted fine-tuning. In contrast, the instruction fine-tuned version of Llama-3.1, a decoder-only LLM, demonstrated superior performance, with the best models achieving macro F1-scores of 44% for 3-way classification and 68% for binary classification. These results have implications for the development of speech technologies aimed at analyzing team communication dynamics and enhancing training interventions in high-stakes environments such as space missions, particularly in scenarios where text is the only accessible data.
A-SMiLE: Affective Sparse Mixture-of-Experts Adapter with Multi-Task Learning for Spoken Dialogue Models
Authors: Yi-Wen Chao; Yizhou Peng; Dianwen Ng; Yukun Ma; Chongjia Ni; Eng Siong Chng; Eng Siong Chng
Venue: Interspeech 2025 (main)

Abstract: Large language models have made remarkable progress in generating coherent and contextually relevant dialogue. However, they still struggle to capture fine-grained paralinguistic cues, limiting emotional coherence and contextual appropriateness. To address this challenge, we propose an Affective Sparse Mixture-of-experts adapter with multi-task Learning (A-SMiLE) to enhance the spoken dialogue model's capability for affective perception across multiple dimensions. Our approach integrates Valence, Arousal, and Dominance (VAD) modeling with response generation, thereby ensuring expressive and contextually aware interactions. Evaluations on DailyTalk and a hard-case benchmark demonstrate significant improvements over baselines in both emotion prediction and response generation. It shows the potential of cognitive-inspired affective modeling to enhance intelligent speech interactions on the spoken dialogue models.
Non-Intrusive Binaural Speech Intelligibility Prediction Using Mamba for Hearing-Impaired Listeners
Authors: Katsuhiko Yamamoto; Koichi Miyazaki
Venue: Interspeech 2025 (main)

Abstract: Speech intelligibility prediction (SIP) models have been used as objective metrics to assess intelligibility for hearing-impaired (HI) listeners. In the Clarity Prediction Challenge 2 (CPC2), non-intrusive binaural SIP models based on transformers showed high prediction accuracy. However, the self-attention mechanism theoretically incurs high computational and memory costs, making it a bottleneck for low-latency, power-efficient devices. This may also degrade the temporal processing of binaural SIPs. Therefore, we propose Mamba-based SIP models instead of transformers for the temporal processing blocks. Experimental results show that our proposed SIP model achieves competitive performance compared to the baseline while maintaining a relatively small number of parameters. Our analysis suggests that the SIP model based on bidirectional Mamba effectively captures contextual and spatial speech information from binaural signals.
No Audiogram: Leveraging Existing Scores for Personalized Speech Intelligibility Prediction
Authors: Haoshuai Zhou; Changgeng Mo; Boxuan Cao; Linkai Li; Shan Xiang Wang
Venue: Interspeech 2025 (main)

Abstract: Personalized speech intelligibility prediction is challenging. Previous approaches have mainly relied on audiograms, which are inherently limited in accuracy as they only capture a listener's hearing threshold for pure tones. Rather than incorporating additional listener features, we propose a novel approach that leverages an individual's existing intelligibility data to predict their performance on new audio. We introduce the Support Sample-Based Intelligibility Prediction Network (SSIPNet), a deep learning model that leverages speech foundation models to build a high-dimensional representation of a listener's speech recognition ability from multiple support (audio, score) pairs, enabling accurate predictions for unseen audio. Results on the Clarity Prediction Challenge dataset show that, even with a small number of support (audio, score) pairs, our method outperforms audiogram-based predictions. Our work presents a new paradigm for personalized speech intelligibility prediction.
Feature Importance across Domains for Improving Non-Intrusive Speech Intelligibility Prediction in Hearing Aids
Authors: Ryandhimas E. Zezario; Sabato M. Siniscalchi; Fei Chen; Hsin-Min Wang; Yu Tsao
Venue: Interspeech 2025 (main)

Abstract: Given the critical role of non-intrusive speech intelligibility assessment in hearing aids (HA), this paper enhances its performance by introducing Feature Importance across Domains (FiDo). We estimate feature importance on spectral and time-domain acoustic features as well as latent representations of Whisper. Importance weights are calculated per frame, and based on these weights, features are projected into new spaces, allowing the model to focus on important areas early. Next, feature concatenation is performed to combine the features before the assessment module processes them. Experimental results show that when FiDo is incorporated into the improved multi-branched speech intelligibility model MBI-Net+, RMSE can be reduced by 7.62% (from 26.10 to 24.11). MBI-Net+ with FiDo also achieves a relative RMSE reduction of 3.98% compared to the best system in the 2023 Clarity Prediction Challenge. These results validate FiDo's effectiveness in enhancing neural speech assessment in HA.
Intelligibility Prediction for Time-Modified Speech Signals Using Spectro-Temporal Modulation Features
Authors: Aymen Bashir; Haolan Wang; Amin Edraki; Wai-Yip Chan; Jesper Jensen
Venue: Interspeech 2025 (main)

Abstract: Reference-based speech intelligibility prediction algorithms (RB-SIPAs) are limited to speech degradations that maintain the time alignment between the clean and degraded speech signals. We address this limitation by augmenting the existing RB-SIPAs framework with time alignment. To achieve robust time alignment at low signal-to-noise ratios, we propose using spectro-temporal modulations (STM) of the speech signals for dynamic time warping (DTW). Our experiments demonstrate that in DTW, the use of specific STM components as features for clean and time-modified degraded speech achieves the baseline time alignment obtained using clean speech and noise-free time-modified speech. Moreover, we propose two methods to incorporate time alignment into existing RB-SIPAs. Using these methods, we compare the output scores of the RB-SIPA with listening test scores and show better correlation results using the STM features as compared to MFCCs.
French Listening Tests for the Assessment of Intelligibility, Quality, and Identity of Body-Conducted Speech Enhancement
Authors: Thomas Joubaud; Julien Hauret; VÃ©ronique Zimpfer; Ãric Bavu
Venue: Interspeech 2025 (main)

Abstract: This study evaluates the Extreme Bandwidth Extension Network (EBEN) model on body-conduction sensors through listening tests. Using the Vibravox dataset, we assess intelligibility with a French Modified Rhyme Test, speech quality with a MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor) protocol and speaker identity preservation with an A/B identification task. The experiments involved male and female speakers recorded with a forehead accelerometer, rigid in-ear and throat microphones. The results confirm that EBEN enhances both speech quality and intelligibility. It slightly degrades speaker identification performance when applied to female speakers' throat microphone recordings. The findings also demonstrate a correlation between Short-Time Objective Intelligibility (STOI) and perceived quality in body-conducted speech, while speaker verification using ECAPA2-TDNN aligns well with identification performance. No tested metric reliably predicts EBEN's effect on intelligibility.
Benchmarking Neural Speech Codec Intelligibility with SITool
Authors: Anna Leschanowsky; Kishor Kayyar Lakshminarayana; Anjana Rajasekhar; Lyonel Behringer; Ibrahim Kilinc; Guillaume Fuchs; EmanuÃ«l A. P. Habets
Venue: Interspeech 2025 (main)

Abstract: Speech intelligibility assessment is essential for evaluating neural speech codecs, yet most evaluation efforts focus on overall quality rather than intelligibility. Only a few publicly available tools exist for conducting standardized intelligibility tests, like the Diagnostic Rhyme Test (DRT) and Modified Rhyme Test (MRT). We introduce the Speech Intelligibility Toolkit for Subjective Evaluation (SITool), a Flask-based web application for conducting DRT and MRT in laboratory and crowdsourcing settings. We use SITool to benchmark 13 neural and traditional speech codecs, analyzing phoneme-level degradations and comparing subjective DRT results with objective intelligibility metrics. Our findings show that, while neural speech codecs can outperform traditional ones in subjective intelligibility, only STOI and ESTOI â not WER â significantly correlate with subjective results, although they struggle to capture gender and wordlist-specific variations observed in subjective evaluations.
AISHELL-5: The First Open-Source In-Car Multi-Channel Multi-Speaker Speech Dataset for Automatic Speech Diarization and Recognition
Authors: Yuhang Dai; He Wang; Xingchen Li; Zihan Zhang; Shuiyuan Wang; Lei Xie; Xin Xu; Hongxiao Guo; Shaoji Zhang; Hui Bu; Wei Chen
Venue: Interspeech 2025 (main)

Abstract: This paper delineates AISHELL-5, the first open-source in-car multi-channel multi-speaker Mandarin automatic speech recognition (ASR) dataset.  AISHLL-5 includes two parts:  (1) over 100 hours of multi-channel speech data recorded in a hybrid electric vehicle across more than 60 real driving scenarios. This audio data consists of four far-filed speech signals captured by microphones located on each car door, as well as near-field signals obtained from high-fidelity headset microphones worn by each speaker. (2) a collection of 40 hours of real-world environmental noise recordings, which supports the in-car speech data simulation. Moreover, we also provide an open-access, reproducible baseline system based on this dataset. This system features a speech frontend model that employs speech source separation to extract each speaker's clean speech from the far-filed signals, along with a speech recognition module that accurately transcribes the content of each individual speaker.
Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR
Authors: Weiqing Wang; Taejin Park; Ivan Medennikov; Jinhan Wang; Kunal Dhawan; He Huang; Nithin Rao Koluguri; Jagadeesh Balam; Boris Ginsburg
Venue: Interspeech 2025 (main)

Abstract: We propose a self-speaker adaptation method for streaming multi-talker automatic speech recognition (ASR) that eliminates the need for explicit speaker queries. Unlike conventional approaches requiring target speaker embeddings or enrollment audio, our technique dynamically adapts individual ASR instances through speaker-wise speech activity prediction. The key innovation involves injecting speaker-specific kernels generated via speaker supervision activations into selected ASR encoder layers. This enables instantaneous speaker adaptation to target speakers while handling fully overlapped speech even in a streaming scenario. Experiments show state-of-the-art performance in both offline and streaming scenarios, demonstrating that our self-adaptive method effectively addresses severe speech overlap through streamlined speaker-focused recognition. The results validate the proposed self-speaker adaptation approach as a robust solution for multi-talker ASR under severe overlapping speech conditions.
Speaker-Distinguishable CTC: Learning Speaker Distinction Using CTC for Multi-Talker Speech Recognition
Authors: Asahi Sakuma; Hiroaki Sato; Ryuga Sugano; Tadashi Kumano; Yoshihiko Kawai; Tetsuji Ogawa
Venue: Interspeech 2025 (main)

Abstract: This paper presents a novel framework for multi-talker automatic speech recognition without the need for auxiliary information. Serialized Output Training (SOT), a widely used approach, suffers from recognition errors due to speaker assignment failures. Although incorporating auxiliary information, such as token-level timestamps, can improve recognition accuracy, extracting such information from natural conversational speech remains challenging. To address this limitation, we propose Speaker-Distinguishable CTC (SD-CTC), an extension of CTC that jointly assigns a token and its corresponding speaker label to each frame. We further integrate SD-CTC into the SOT framework, enabling the SOT model to learn speaker distinction using only overlapping speech and transcriptions. Experimental comparisons show that multi-task learning with SD-CTC and SOT reduces the error rate of the SOT model by 26% and achieves performance comparable to state-of-the-art methods relying on auxiliary information.
Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios
Authors: Aswin Shanmugam Subramanian; Amit Das; Naoyuki Kanda; Jinyu Li; Xiaofei Wang; Yifan Gong
Venue: Interspeech 2025 (main)

Abstract: We extend the frameworks of Serialized Output Training (SOT) to address practical needs of both streaming and offline automatic speech recognition (ASR) applications. Our approach focuses on balancing latency and accuracy, catering to real-time captioning and summarization requirements. We propose several key improvements: (1) Leveraging Continuous Speech Separation (CSS) single-channel front-end with end-to-end (E2E) systems for highly overlapping scenarios, challenging the conventional wisdom of E2E versus cascaded setups. The CSS framework improves the accuracy of the ASR system by separating overlapped speech from multiple speakers. (2) Implementing dual modelsâConformer Transducer for streaming and Sequence-to-Sequence for offlineâor alternatively, a two-pass model based on cascaded encoders. (3) Exploring segment-based SOT (segSOT) which is better suited for offline scenarios while also enhancing readability of multi-talker transcriptions.
Efficient Streaming TTS Acoustic Model with Depthwise RVQ Decoding Strategies in a Mamba Framework
Authors: Joun Yeop Lee; Sangjun Park; Byoung Jin Choi; Ji-Hyun Lee; Min-Kyung Kim; Hoon-Young Cho
Venue: Interspeech 2025 (main)

Abstract: Recent advances in neural codec-based text-to-speech (TTS) systems have achieved remarkable synthesized speech quality. However, their reliance on large model size and heavy computational requirements limits CPU-based on-device deployment. In this work, we present a Mamba-based streaming acoustic model with two novel depthwise decoding strategies for residual vector quantization (RVQ)-based codec: a Masked Language Model (MLM) approach and an Implicit Neural Representation (INR) approach. The MLM strategy iteratively refines tokens along the code depth axis to enhance speech quality, whereas the INR approach predicts all quantization levels in parallel to reduce computational costs. We further incorporate a speaker embedding conditioning mechanism for a zero-shot scenario, enabling robust performance on unseen speakers. Experimental results demonstrate comparable or even superior improvements in both objective and subjective metrics compared to other larger TTS baseline models.
APTTS: Adversarial Post-training in Latent Flow Matching for Fast and High-fidelity Text-to-Speech
Authors: Hyungchan Yoon; Chanwoo Lee; Hoodong Lee; Stanley Jungkyu Choi
Venue: Interspeech 2025 (main)

Abstract: Although recent text-to-speech (TTS) models based on flow matching have achieved remarkable generation quality, their reliance on numerous sampling steps hinders practical deployment. In this work, we introduce an adversarial post-training strategy for flow matching TTS that significantly reduces the required sampling steps. Our approach treats a pre-trained flow matching model as a few-step generator, optimizing it with reconstruction and adversarial objectives. We integrate this technique into APTTS, our novel latent flow matching framework for zero-shot TTS, and demonstrate its superiority over state-of-the-art baselines with real-time applicability.  Furthermore, we validate the scalability of our adversarial post-training approach by applying it to Matcha-TTS, a publicly available flow matching model. Evaluations on a multi-speaker dataset show that our method enhances audio quality while reducing inference time, underscoring its potential as a scalable solution for real-time TTS.
Eigenvoice Synthesis based on Model Editing for Speaker Generation
Authors: Masato Murata; Koichi Miyazaki; Tomoki Koriyama; Tomoki Toda
Venue: Interspeech 2025 (main)

Abstract: Speaker generation task aims to create unseen speaker voice without reference speech. The key to the task is defining a speaker space that represents diverse speakers to determine the generated speaker trait. However, the effective way to define this speaker space remains unclear. Eigenvoice synthesis is one of the promising approaches in the traditional parametric synthesis framework, such as HMM-based methods, which define a low-dimensional speaker space using pre-stored speaker features. This study proposes a novel DNN-based eigenvoice synthesis method via model editing. Unlike prior methods, our method defines a speaker space in the DNN model parameter space. By directly sampling new DNN model parameters in this space, we can create diverse speaker voices. Experimental results showed the capability of our method to generate diverse speakers' speech. Moreover, we discovered a gender-dominant axis in the created speaker space, indicating the potential to control speaker attributes.
Score-Based Training for Energy-Based TTS Models
Authors: Wanli Sun; Anton Ragni
Venue: Interspeech 2025 (main)

Abstract: Noise contrastive estimation (NCE) is a popular method for training energy-based models (EBM) with intractable normalisation terms. The key idea of NCE is to learn by comparing unnormalised log-likelihoods of the reference and noisy samples, thus avoiding explicitly computing normalisation terms. However, NCE critically relies on the quality of noisy samples. Recently, sliced score matching (SSM) has been popularised by closely related diffusion models (DM). Unlike NCE, SSM learns a gradient of log-likelihood, or score, by learning distribution of its projections on randomly chosen directions. However, both NCE and SSM disregard the form of log-likelihood function, which is problematic given that EBMs and DMs make use of first-order optimisation during inference. This paper proposes a new criterion that learns scores more suitable for first-order schemes. Experiments contrasts these approaches for training EBMs.
Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding
Authors: Zijian Lin; Yang Zhang; Yougen Yuan; Yuming Yan; Jinjiang Liu; Zhiyong Wu; Pengfei Hu; Qun Yu
Venue: Interspeech 2025 (main)

Abstract: Modern autoregressive speech synthesis models leveraging language models have demonstrated remarkable performance. However, the sequential nature of next token prediction in these models leads to significant latency, hindering their deployment in scenarios where inference speed is critical. In this work, we propose Speech Speculative Decoding (SSD), a novel framework for autoregressive speech synthesis acceleration. Specifically, our method employs a lightweight draft model to generate candidate token sequences, which are subsequently verified in parallel by the target model using the proposed SSD framework. Experimental results demonstrate that SSD achieves a significant speedup of 1.4x compared with conventional autoregressive decoding, while maintaining high fidelity and naturalness. Subjective evaluations further validate the effectiveness of SSD in preserving the perceptual quality of the target model while accelerating inference.
BitTTS: Highly Compact Text-to-Speech Using 1.58-bit Quantization and Weight Indexing
Authors: Masaya Kawamura; Takuya Hasumi; Yuma Shirahata; Ryuichi Yamamoto
Venue: Interspeech 2025 (main)

Abstract: This paper proposes a highly compact, lightweight text-to-speech (TTS) model for on-device applications. To reduce the model size, the proposed model introduces two techniques. First, we introduce quantization-aware training (QAT), which quantizes model parameters during training to as low as 1.58-bit. In this case, most of 32-bit model parameters are quantized to ternary values {-1, 0, 1}. Second, we propose a method named weight indexing. In this method, we save a group of 1.58-bit weights as a single int8 index. This allows for efficient storage of model parameters, even on hardware that treats values in units of 8-bit. Experimental results demonstrate that the proposed method achieved 83 % reduction in model size, while outperforming the baseline of similar model size without quantization in synthesis quality.
GTAnet: Geometry-Guided Temporal Attention for EEG-Based Sound Source Tracking in Cocktail Party Scenarios
Authors: Saurav Pahuja; Gabriel Ivucic; Siqi Cai; Dashanka Da Silva; Haizhou Li; Tanja Schultz
Venue: Interspeech 2025 (main)

Abstract: EEG-based sound source tracking in cocktail party scenarios is a challenging task due to the complex acoustic environment. To address this, we propose a novel approach, Geometry-guided Temporal Attention (GTAnet), which integrates the spatial configuration of multi-channel EEG signals with the temporal dynamics of neural activity. GTAnet constructs a geometry-based graph to capture the spatial relationships between electrodes while employing a temporal attention mechanism to highlight key intervals in auditory processing. The results show that GTAnet outperforms baseline models and offers interpretable insights into the neural mechanisms underlying auditory scene analysis.