GenAI4ELab/papercli-papers huggingface dataset:
TargetVoice: Single Channel Low-Latency Target Speaker Extraction
Authors: Arun Kumar Pallala; Nivedita Chennupati; Balaji Padmanaban; Rakesh Pogula; Uma Subhashini Ravuri; Naveen Ellanki; Harish Rajamani; Naveen Ambati
Venue: Interspeech 2025 (main)

Abstract: We present TargetVoice, a lightweight, low-latency target speaker extraction (TSE) model optimized for edge devices. It isolates a target speakerâs voice from multi-speaker and noisy environments, making it ideal for use in call centers, conference calls, hands-free communication, and smart speakers. By streaming only the enrolled speakerâs voice, TargetVoice also improves speech recognition accuracy in real-world conditions. Unlike existing models that struggle with similar-gender speakers or varying acoustic environments, TargetVoice leverages a robust in-house data strategy and a specialized speaker embedding extraction system. The model uses a compact 10MB speaker encoder to generate a reliable embedding from a single 3 second enrollment. This embedding is fused with the input mixture in a 12MB extraction block with 6G MACs to isolate the target voice efficiently, enabling real-time performance on resource-constrained platforms.
Rollback Speech: Smart Feedback Prompts for Lost Utterances in Unstable Online Calls
Authors: Yuni Amaloa Quintero Villalobos; Wafaa Wardah; Sebastian MÃ¶ller; Robert P. Spang
Venue: Interspeech 2025 (main)

Abstract: We present a prototype system designed to enhance the user experience of online conferencing in the presence of poor network conditions. When connection issues cause parts of a speakerâs audio to be lost, the system uses dual automatic speech recognition (ASR) outputsâone local and one remoteâto detect which utterances were not received. Upon reconnection, the speaker is provided with extracted keywords representing the information that did not reach the listener, enabling efficient and targeted repetition without full redundancy. This helps maintain conversational flow and reduces frustration during virtual communication. The system comprises real-time transcription, network degradation simulation, discrepancy detection, and lightweight keyword extraction in a modular architecture. Our âShow and Tellâ demonstration allows participants to experience how the system augments communication continuity, even under conditions of packet loss or delayed transmission.
Simultaneous Speech Translation Integrated Compact Multiple Sound Spot Synthesis System On A Laptop Carried Out With A Backpack
Authors: Takuma Okamoto; Michiyo Kono
Venue: Interspeech 2025 (main)

Abstract: Multiple sound spot synthesis, which can present different sounds in different zones simultaneously using a loudspeaker array, is an important spatial sound presentation technology for speech and audio applications. We have previously implemented a portable multiple sound spot synthesis system with a compact circular array of 16 loudspeakers, which can be carried out with a suitcase. To further improve the mobility, we implement a very small 16-channel amplifier directly mounted under the compact circular array. Additionally, we implement a system integrating multiple sound spot synthesis and multilingual simultaneous speech-to-speech translation on-premise on a laptop without network connection. Finally, the complete demo system can be carried out with a backpack. In the Show & Tell, we demonstrate four-language sound spot synthesis combined with multilingual simultaneous speech-to-speech translation using the compact demo system.
GenECA: A General-Purpose Framework for Real-Time Adaptive Multimodal Embodied Conversational Agents
Authors: Santosh Patapati; Aashrith Tatineni; Trisanth Srinivasan
Venue: Interspeech 2025 (main)

Abstract: We present GenECA, a general-purpose framework for real-time multimodal interaction with embodied conversational agents. GenECA captures audio and visual signals from standard devices to analyze nonverbal features such as facial expressions, vocal tone, gaze, and posture. This information is used to generate context-aware dialogue and synchronize the agent's speech with dynamic gestures and backchannel facial animations in real time. GenECA provides the first ECA system able to deliver context-aware speech and well-timed animations in real-time without reliance on human operators. Through modular design, it can support a wide variety of applications, such as education, customer service, and therapy. Our research enables rapid prototyping and deployment of interactive virtual agents.
Hybrid Expert Knowledge and Self-Supervised Learning for Diagnostic Modeling of Adductor Spasmodic and Primary Myotonic Dysphonia
Authors: Zhou Du; Hang Chen; Huijun Ding; Jun Du; Zhen Chen
Venue: Interspeech 2025 (main)

Abstract: Dysphonia encompasses a broad spectrum of vocal disorders with diverse etiologies, among which adductor spasmodic dysphonia (ADSD) and primary muscle tension dysphonia (pMTD) are particularly challenging to diagnose. Currently, the primary diagnostic method relies on subjective auditory perception by highly experienced clinicians. To alleviate the scarcity of diagnostic resources, this study develops a deep learning-based approach for automatically diagnosing ADSD and pMTD using patientsâ speech data. Our contributions are: (1) designing a convolutional neural network (CNN)-based diagnostic model that leverages handcrafted features derived from expert knowledge and (2) incorporating self-supervised learning (SSL) to extract more discriminative representations as input from raw waveforms adaptively. This marks the first application of deep learning techniques to ADSD and pMTD diagnostic modeling, achieving a classification accuracy of 83.3% on our newly constructed dataset.
MVP: Multi-source Voice Pathology detection
Authors: Alkis Koudounas; Moreno La Quatra; Gabriele Ciravegna; Marco Fantini; Erika Crosetti; Giovanni Succo; Tania Cerquitelli; Sabato Marco Siniscalchi; Elena Baralis
Venue: Interspeech 2025 (main)

Abstract: Voice disorders significantly impact patient quality of life, yet non-invasive automated diagnosis remains under-explored due to both the scarcity of pathological voice data, and the variability in recording sources. This work introduces MVP (Multi-source Voice Pathology detection), a novel approach that leverages transformers operating directly on raw voice signals. We explore three fusion strategies to combine sentence reading and sustained vowel recordings: waveform concatenation, intermediate feature fusion, and decision-level combination. Empirical validation across the German, Portuguese, and Italian languages shows that intermediate feature fusion using transformers best captures the complementary characteristics of both recording types. Our approach achieves up to +13% AUC improvement over single-source methods.
Phonetic Posteriorgram-Based Phoneme Selection for Vocal Cord Disorder Classification in Continuous Mandarin Speech
Authors: Chih-Ning Chen; Yu-Lan Chuang; Ming-Jhang Yang; Wei-Cheng Hsu; Yung-An Tsou; Yi-Wen Liu
Venue: Interspeech 2025 (main)

Abstract: Automatic classification of vocal cord disorders (VCDs) in dysphonia benefits society by enabling home screening when immediate clinical consultation is unavailable. Previous studies focused on VCD classification using single vowels or isolated words. This research advances the field by classifying VCDs from continuous speech, using data from diagnosed patients. By parsing continuous speech into phonemes based on phonetic posteriorgrams (PPGs), we investigated VCD classification using the Mel frequency cepstral coefficients (MFCCs) corresponding to each Mandarin phoneme as the features. Results show a 15% accuracy improvement over a baseline model that ignores phonetic context and a prior study on the same patients using single-word utterances and sustained vowels. Our findings enhance the understanding of phonetic characteristics in VCDs and underscore the significance of continuous speech in automatic classification.
Articulatory clarity and variability before and after surgery for tongue cancer
Authors: Thomas Tienkamp; Fleur van Ast; Roos van der Veen; Teja Rebernik; Raoul Buurke; Nikki Hoekzema; Katharina Polsterer; Hedwig Sekeres; Rob van Son; Martijn Wieling; Max Witjes; Sebastiaan de Visscher; Defne Abur
Venue: Interspeech 2025 (main)

Abstract: Surgical treatment for tongue cancer can negatively affect the mobility and musculature of the tongue, which can influence articulatory clarity and variability. In this study, we investigated articulatory clarity through the vowel articulation index (VAI) and variability through vowel formant dispersion (VFD). Using a sentence reading task, we assessed 11 individuals pre and six months post tongue cancer surgery, alongside 11 sex- and age-matched typical speakers. Our results show that while the VAI was significantly smaller post-surgery compared to pre-surgery, there was no significant difference between patients and typical speakers at either time point. Post-surgery, speakers had higher VFD values for /i/ compared to pre-surgery and typical speakers, signalling higher variability. Taken together, our results suggest that while articulatory clarity remained within typical ranges following surgery for tongue cancer for the speakers in our study, articulatory variability increased.
Hybrid HMM-SVM classifier using frication-based features for detection of non-normative sibilant articulation patterns in Polish childrenâs speech
Authors: Zuzanna Miodonska
Venue: Interspeech 2025 (main)

Abstract: This study proposes a method for detecting non-normative articulatory features of the retroflex voiceless fricative, which is among the last sounds to develop in Polish children's speech. The approach integrates time-based phone articulation models with SVM in a hybrid binary classification method. The novelty lies in incorporating noise-based acoustic features for improved assessment and employing statistical temporal models to enhance detection accuracy. Results show that the proposed approach achieved over 86% accuracy detection for interdental and over 90% for addental articulation patterns. Dental articulation was most distinguishable, nearing 100% accuracy. Our findings indicate that adding noise features improves classification accuracy. This research contributes to the development of sigmatism diagnostic tools tailored for preschool and early school-age children, a group for whom automated pronunciation assessment remains an open challenge.
Fine-Tuning ASR for Stuttered Speech: Personalized vs. Generalized Approaches
Authors: Dena Mujtaba; Nihar R. Mahapatra
Venue: Interspeech 2025 (main)

Abstract: Stuttering - characterized by involuntary disfluencies such as blocks, prolongations, and repetitions - is often misinterpreted by automatic speech recognition (ASR) systems, resulting in elevated word error rates and making voice-driven technologies inaccessible to people who stutter. The variability of disfluencies across speakers and contexts further complicates ASR training, compounded by limited annotated stuttered speech data. In this paper, we investigate fine-tuning ASRs for stuttered speech, comparing generalized models (trained across multiple speakers) to personalized models tailored to individual speech characteristics. Using a diverse range of voice-AI scenarios, including virtual assistants and video interviews, we evaluate how personalization affects transcription accuracy. Our findings show that personalized ASRs significantly reduce word error rates, especially in spontaneous speech, highlighting the potential of tailored models for more inclusive voice technologies.
SiamCTC: Learning Speech Representations through Monotonic Temporal Alignment
Authors: SooHwan Eom; Mark Hasegawa-Johnson; Chang D. Yoo
Venue: Interspeech 2025 (main)

Abstract: Self-supervised speech representation learning has made significant progress through Siamese networks, which leverage different views of the same input. However, existing methods often require frame-wise alignment between these views, overlooking the broader linguistic context invariance across different speaking styles. We introduce SiamCTC, a framework that integrates Siamese networks with Connectionist Temporal Classification (CTC) to learn speech representations without strict frame-level correspondence. By employing CTC loss to establish flexible, monotonic alignments between differing temporal realizations of the same content, SiamCTC accommodates speed perturbations and other temporal augmentations. This design relaxes frame-wise constraints while preserving temporal coherence and enhancing robustness to speaking-rate variations in downstream tasks. Our experiments demonstrate that SiamCTC leads to more adaptable speech representations, particularly at diverse speaking rates.
Improving Generalization of End-to-End ASR through Diversity and Independence Regularization
Authors: Ye-Eun Ko; Mun-Hak Lee; Dong-Hyun Kim; Joon-Hyuk Chang
Venue: Interspeech 2025 (main)

Abstract: Automatic speech recognition (ASR) has been driven by representative end-to-end model architectures, including connectionist temporal classification (CTC), attention-based encoder-decoder (AED), and recurrent neural network transducer (RNN-T). However, these models are prone to overfitting during training, which degrades their generalization performance. In this paper, we propose a novel regularization technique applicable to various ASR models: diversity loss and independence loss. Diversity loss reduces the similarity between feature representations, encouraging the model to learn diverse patterns. Independence loss minimizes the covariance between feature vectors, ensuring that they contain independent information and reducing redundancy. We apply these techniques to CTC, AED, and RNN-T models and demonstrate that the proposed regularization method effectively improves the model generalization performance and robustness through extensive experiments.
Exploring Linear Variant Transformers and k-NN Memory Inference for Long-Form ASR
Authors: Carlos Carvalho; Jinchuan Tian; William Chen; Yifan Peng; Alberto Abad; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: While transformer-based models excel in short-form (SF) automatic speech recognition, the quadratic complexity of the self-attention mechanism introduces significant challenges in long-form (LF) audio. To address this, we compare strong linear transformer variantsâFastformer, SummaryMixing, BiMamba, and E-Branchformer with Flash Attention (EBranch-FA). For the latter, we also explore rotary positional encodings. Additionally, we propose a new challenging LF benchmark derived from the LibriHeavy corpus, featuring development and test sets with varying average durations to enable comprehensive evaluation across different temporal scales. Furthermore, we propose a memory system, KNN-MAN, for inference, which can be applied to any existing encoder-decoder models, without additional training. For example, with BiMamba, we reduce the word error rate from 18.8% to 17.5% in our LF Test Clean set derived from LibriSpeech.
Attention-Free Dual-Mode ASR with Latency-Controlled Selective State Spaces
Authors: Takafumi Moriya; Masato Mimura; Kiyoaki Matsui; Hiroshi Sato; Kohei Matsuura
Venue: Interspeech 2025 (main)

Abstract: This paper introduces a novel encoder architecture designed to enhance transducer-based dual-mode automatic speech recognition (ASR). Our approach leverages the selective state-space model, Mamba, to enable attention-free dual-mode ASR. While bidirectional Mamba (BiMamba) captures full context and enables constant-time inference, unlike attention-based models with quadratic complexity, it is limited to offline processing. In contrast, using only unidirectional Mamba for the dual-mode ASR degrades ASR performance in both offline and streaming modes due to its restricted access to future contexts. To address this issue, we propose the latency-controlled BiMamba (LC-BiMamba); it enables chunk-wise processing in streaming mode while still accessing future contexts and functioning as standard BiMamba does in offline mode. Experimental results demonstrate that LC-BiMamba outperforms the baseline Conformer system, achieving faster and more accurate decoding in our dual-mode ASR framework.
Thinking Fast and Slow: Robust Speech Recognition via Deep Filter-Tuning
Authors: Dianwen Ng; Kun Zhou; Bin Ma; Eng Siong Chng
Venue: Interspeech 2025 (main)

Abstract: Self-supervised learning (SSL) models have revolutionized speech representation by extracting rich acoustic and phonetic features with minimal labeled data. However, their computational demands during fine-tuning and vulnerability to catastrophic forgetting pose challenges for practical deployment. Parameter-Efficient fine-tuning (PEFT) methods, such as prompt tuning, are often employed to address these challenges. While prompt tuning has been successful with large language models in natural language processing (NLP), it struggles to learn effective instructional signals when adapting to speech SSL models, likely due to insufficient a priori knowledge that hinders soft token learning during fine-tuning. We introduce Deep Filter Tuning (DFT), a soft-token adaptation strategy that selectively filters semantic information from noise-distorted representations. By modifying only 0.38% of model weights, DFT achieves a 12% performance gain in noisy environments, offering an efficient solution for robust speech recognition under challenging conditions such as noise adaptation.
Towards Efficiently Whisper Fine-tuning with Monotonic Alignments
Authors: Ziyang Zhuang; Tao Wei; Ming Fang; Ning Cheng; Shaojun Wang; Jing Xiao
Venue: Interspeech 2025 (main)

Abstract: Whisper is a popular large-scale model developed by OpenAI, which can perform robust speech recognition and speech translation tasks. Fine-tuning Whisper on various domains thus becomes a common approach to improve the model performance on specific tasks. This paper introduces an efficient fine-tuning method that enhances Whisper's fine-tuning quality by incorporating a training objective based on monotonic alignments, without modifying the model network. Experimental results show that the proposed fine-tuning approach achieves consistent and considerable improvements over normal fine-tuned models on speech recognition tasks and some of the speech translation tasks. In addition, a new dataset MDCCST is built to validate the performance of Cantonese-to-Mandarin translation.
Dynamic Acoustic Model Architecture Optimization in Training for ASR
Authors: Jingjing Xu; Zijian Yang; Albert Zeyer; Eugen Beck; Ralf SchlÃ¼ter; Hermann Ney
Venue: Interspeech 2025 (main)

Abstract: Architecture design is inherently complex. Existing approaches rely on either handcrafted rules, which demand extensive empirical expertise, or automated methods like neural architecture search, which are computationally intensive. In this paper, we introduce DMAO, an architecture optimization framework that employs a grow-and-drop strategy to automatically reallocate parameters during training.This reallocation shifts resources from less-utilized areas to those parts of the model where they are most beneficial. Notably, DMAO only introduces negligible training overhead at a given model complexity. We evaluate DMAO through experiments with CTC on LibriSpeech, TED-LIUM-v2 and Switchboard datasets. The results show that, using the same amount of training resources, our proposed DMAO consistently improves WER by up to ~6% relatively across various architectures, model sizes, and datasets. Furthermore, we analyze the pattern of parameter redistribution and uncover insightful findings.
Knowledge Distillation Method for Pruned RNN-T Models via Pruning Bounds Sharing and Losses Confusion
Authors: Xiaocan Zhang; Weiwei Jiang; Guibin Zheng; Chenhao Jing; Jiqing Han; Tieran Zheng
Venue: Interspeech 2025 (main)

Abstract: Although the advantages of large models have been widely proved in speech recognition, small models are still required in several applications due to the limited computational resources or training data. The recognition accuracy of small models has always been a challenging issue. This paper proposes a distillation method for the pruned RNN-T structure to enhance the generalization ability of small models by leveraging information from large models, where the small model shares the pruning bounds of the large model as well as the decoder and connector structures, and multi-loss fusion is used to distill. Utilizing the Chinese speech dataset Aishell-1, experimental results demonstrated that the small model distilled from pre-trained large model significantly outperforms the directly trained model of the same size by a notable relative reduction of 30.4% in Character Error Rate (CER), thereby validating the effectiveness of the proposed knowledge distillation method.
An Effective Training Framework for Light-Weight Automatic Speech Recognition Models
Authors: Abdul Hannan; Alessio Brutti; Shah Nawaz; Mubashir Noman
Venue: Interspeech 2025 (main)

Abstract: Recent advancement in deep learning encouraged developing large automatic speech recognition (ASR) models that achieve promising results while ignoring computational and memory constraints. However, deploying such models on low resource devices is impractical despite of their favorable performance. Existing approaches (pruning, distillation, layer skip etc.) transform the large models into smaller ones at the cost of significant performance degradation or require prolonged training of smaller models for better performance. To address these issues, we introduce an efficacious two-step representation learning based approach capable of producing several small sized models from a single large model ensuring considerably better performance in limited number of epochs. Comprehensive experimentation on ASR benchmarks reveals the efficacy of our approach, achieving three-fold training speed-up and up to 12.54% word error rate improvement.
Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filtering
Authors: AndrÃ©s Carofilis; Pradeep Rangappa; Srikanth Madikeri; Shashi Kumar; Sergio Burdisso; Jeena Prakash; EsaÃº Villatoro-Tello; Petr Motlicek; Bidisha Sharma; Kadri Hacioglu; Shankar Venkatesan; Saurabh Vyas; Andreas Stolcke
Venue: Interspeech 2025 (main)

Abstract: Fine-tuning pretrained ASR models for specific domains is challenging when labeled data is scarce. But unlabeled audio and labeled data from related domains are often available. We propose an incremental semi-supervised learning pipeline that first integrates a small in-domain labeled set and an auxiliary dataset from a closely related domain, achieving a relative improvement of 4% over no auxiliary data. Filtering based on multi-model consensus or named entity recognition (NER) is then applied to select and iteratively refine pseudo-labels, showing slower performance saturation compared to random selection. Evaluated on the multi-domain Wow call center and Fisher English corpora, it outperforms single-step fine-tuning. Consensus-based filtering outperforms other methods, providing up to 22.3% relative improvement on Wow and 24.8% on Fisher over single-step fine-tuning with random selection. NER is the second-best filter, providing competitive performance at a lower computational cost.