GenAI4ELab/papercli-papers huggingface dataset:
Bridging ASR and LLMs for Dysarthric Speech Recognition: Benchmarking Self-Supervised and Generative Approaches
Authors: Ahmed Aboeitta; Ahmed Sharshar; Youssef Nafea; Shady Shehata
Venue: Interspeech 2025 (main)

Abstract: Dysarthric speech presents significant challenges for Automatic Speech Recognition (ASR) due to phoneme distortions and high variability. While self-supervised ASR models like Wav2Vec, HuBERT, and Whisper have shown promise, their effectiveness in dysarthric speech remains unclear. This study systematically benchmarks these models with different decoding strategies, including CTC, seq2seq, and LLM-enhanced decoding (BART, GPT-2, Vicuna). Our contributions include (1) benchmarking ASR architectures for dysarthric speech, (2) introducing LLM-based decoding to improve intelligibility, (3) analyzing generalization across datasets, and (4) providing insights into recognition errors across severity levels. Findings highlight that LLM-enhanced decoding improves dysarthric ASR by leveraging linguistic constraints for phoneme restoration and grammatical correction.
Towards Inclusive ASR: Investigating Voice Conversion for Dysarthric Speech Recognition in Low-Resource Languages
Authors: Chin-Jou Li; Eunjung Yeo; Kwanghee Choi; Paula Andrea PÃ©rez-Toro; Masao Someki; Rohan Kumar Das; Zhengjun Yue; Juan Rafael Orozco-Arroyave; Elmar NÃ¶th; David R. Mortensen
Venue: Interspeech 2025 (main)

Abstract: Automatic speech recognition (ASR) for dysarthric speech remains challenging due to data scarcity, particularly in non-English languages. To address this, we fine-tune a voice conversion model on English dysarthric speech (UASpeech) to encode both speaker characteristics and prosodic distortions, then apply it to convert healthy non-English speech (FLEURS) into non-English dysarthric-like speech. The generated data is then used to fine-tune a multilingual ASR model, Massively Multilingual Speech (MMS), for improved dysarthric speech recognition. Evaluation on PC-GITA (Spanish), EasyCall (Italian), and SSNCE (Tamil) demonstrates that VC with both speaker and prosody conversion significantly outperforms the off-the-shelf MMS performance and conventional augmentation techniques such as speed and tempo perturbation. Objective and subjective analyses of the generated data further confirm that the generated speech simulates dysarthric characteristics.
Mitigating Overfitting During Speech Foundation Model Fine-tuning: Applications to Dysarthric Speech Detection
Authors: Yan Xiong; Visar Berisha; Julie Liss; Chaitali Chakrabarti
Venue: Interspeech 2025 (main)

Abstract: Speech foundation models have shown significant success in various speech-processing applications. However, fine-tuning these models on dysarthric speech is challenging due to overfitting caused by limited dataset sizes. This work proposes a modified multitask learning framework to mitigate overfitting in foundation model fine-tuning. Specifically, we train the model on a more complex task along with the task of interest and use gradient projection to preserve beneficial updates while resolving conflicts. We demonstrate that using automatic speech recognition as the main task and dysarthria detection as the auxiliary task improves model robustness and dysarthria detection performance. The proposed method1 reduces overfitting and improves in-corpus and cross-corpus detection accuracy by 5.4% to 13.4% compared to standard multi-task learning. These findings highlight the importance of structured multitask training for enhancing foundation model adaptability.
Towards Temporally Explainable Dysarthric Speech Clarity Assessment
Authors: Seohyun Park; Chitralekha Gupta; Michelle Kah Yian Kwan; Xinhui Fung; Alexander Wenjun Yip; Suranga Nanayakkara
Venue: Interspeech 2025 (main)

Abstract: Dysarthria, a motor speech disorder, affects intelligibility and requires targeted interventions for effective communication. In this work, we investigate automated mispronunciation feedback by collecting a dysarthric speech dataset from six speakers reading two passages, annotated by a speech therapist with temporal markers and mispronunciation descriptions. We design a three-stage framework for explainable mispronunciation evaluation: (1) overall clarity scoring, (2) mispronunciation localization, and (3) mispronunciation type classification. We systematically analyze pretrained Automatic Speech Recognition (ASR) models in each stage, assessing their effectiveness in dysarthric speech evaluation. Our findings offer clinically relevant insights for automating actionable feedback for pronunciation assessment, which could enable independent practice for patients and help therapists deliver more effective interventions.
Code Mix TTS: An Approach to Infer Human Like Speech for Multi-Lingual Input Texts
Authors: Vishal Gourav; Phanindra Mankale
Venue: Interspeech 2025 (main)

Abstract: We have come a far way in terms of producing high quality, human like audios for given input texts using TTS or Text to Speech Systems. We simply provide the TTS system with an input text and it generates an audio output. But, if we look at it closely, we provide the input text in a particular language only. However, as humans, while interacting with each other, do we actually speak in just one particular language? The answer is No, in most cases. Our speech in general, specially in multiple language countries, is a mix of native language of the specific country and a popular language(mostly English) and sometimes the other way round where we mix a bit of the native language words in our English conversation. In this paper, we talk about an approach to achieve the similar kind of "Code Mix" inference output for "Multi-Lingual" input texts from our TTS system without the need of any additional training data or fine tuning.
Turing's Echo: Investigating Linguistic Sensitivity of Deepfake Voice Detection via Gamification
Authors: Binh Nguyen; Thai Le
Venue: Interspeech 2025 (main)

Abstract: Recent advancements in text-to-speech technology have made it easier than ever to synthesize high-fidelity, human-like audio, commonly coined as deepfake speech. While this technology is beneficial to many applications, it has introduced significant risks, especially in enabling hyper-realistic impersonation threats. Existing research has developed robust algorithms capable of detecting deepfake speech under various acoustic conditions, such as pitch shift and background noise. However, the indirect impact of linguistic factors, such as word choice, grammar, and sentence structure of a deepfake speech's transcript, on the performance of deepfake detectors remains unexplored. As the first step to bridge this gap, this paper introduces a gamified research prototype, called TURING'S ECHO, to evaluate (1) how humans perceive such linguistic sensitivity in comparison with machine and (2) how robust is state-of-the-art deepfake speech detection under different linguistic nuances.
Unleashing the Inner Monster: Demonstrating High-Fidelity Human to Non-Human Voice Conversion
Authors: Namhyun Cho; Sunmin Kim; Minsu Kang; Seolhee Lee; Choonghyeon Lee; Yangsun Lee
Venue: Interspeech 2025 (main)

Abstract: Non-human monsters in massive multiplayer online role-playing games (MMORPGs) are essential for immersive game environments and significantly enhance player engagement. However, producing high-quality monster sounds demands considerable time and financial resources. AI-driven voice conversion offers a potential solution, but existing models rely on standard human voice training and human-centric audio features, limiting their ability to generate realistic non-human voices. This study presents a human-to-non-human (H2NH) voice conversion model designed to address these challenges. The model effectively generated high-quality non-human sounds by recording the voices of participants and converting them in real-time into selected monster vocalizations.
TungnaÃ¡ In Live Performance: An Implementation Of Interactive Artistic Text-To-Voice
Authors: Victor Shepardson; Jonathan Reus; Thor Magnusson
Venue: Interspeech 2025 (main)

Abstract: We demonstrate Tungnaa, a software musical instrument for controlling interactive artistic text-to-voice (IATV) models. IATV is a proposed low-resource text-conditional voice generation task for musical performance applications. TungnaÃ¡ is a Python package implementing IATV training, a real-time inference engine and graphical interface for performance with IATV models. We also describe an application of Tungnaa in collaboration with sound poet Jaap Blonk.
Hear Me Out: Interactive evaluation and bias discovery platform for speech-to-speech conversational AI
Authors: Shree Harsha Bokkahalli Satish; Gustav Eje Henter; Ãva SzÃ©kely
Venue: Interspeech 2025 (main)

Abstract: A new wave of speech foundation models is emerging, capable of processing spoken language directly from audio. These models promise more expressive and emotionally aware interactions by retaining prosodic information throughout conversations. âHear Me Outâ evaluates their ability to preserve crucial vocal cues, enabling users to explore how variations in speaker characteristics and paralinguistic features influence AI responses. Through real-time voice conversion, users can ask a question and then re-ask it with a modified one, immediately observing differences in response tone, phrasing, and behavior. The system presents paired responses side by side, offering direct comparisons of AI interpretations of both the original and transformed voices, thereby highlighting potential biases. By creating inquiry into speaker modeling, contextual understanding, and fairness, this immersive experience encourages users to reflect on identity, voice, and also promote inclusive future research.
Vocal-tract model with two directions: Static design for a dummy head and dynamic design for a speaking machine
Authors: Takayuki Arai
Venue: Interspeech 2025 (main)

Abstract: Physical models of the human vocal tract have been developed for many purposes, including education in acoustics and phonetics, speech and language pathology, and speech science/technology. To cover different application areas, variations are needed. When organizing the types of vocal-tract models developed for such a wide range, one of the major axes is static vs. dynamic aspects. In this study, we demonstrate two models that are the current extremities of the two directions of this axis. The first model is an ultimate static one in which the vocal-tract configuration is one vowel. This model is embedded in a dummy head so that it can be applied to produce exactly the same vowel repeatedly or for a very long duration with a realistic directional radiation pattern. The second model is a recently developed dynamic one with several blocks simulating articulatory movements. The cam mechanisms help to change the vocal-tract configurations in real time.
Dhvani: A Weakly-supervised Phonemic Error Detection and Personalized Feedback System for Hindi
Authors: Arnav Rustagi; Satvik Bajpai; Nimrat Kaur; Siddharth Siddharth
Venue: Interspeech 2025 (main)

Abstract: Computer-Assisted Pronunciation Training (CAPT) has been extensively studied for English. However, there remains a critical gap in its application to Indian languages with a base of 1.5 billion speakers. Pronunciation tools tailored to Indian languages are strikingly lacking despite the fact that millions learn them every year. With over 600 million speakers and being the fourth most-spoken language worldwide, improving Hindi pronunciation is a vital first step toward addressing this gap. This paper proposes 1) Dhvani---a novel CAPT system for Hindi, 2) synthetic speech generation for Hindi mispronunciations, and 3) a novel methodology for providing personalized feedback to learners. While the system often interacts with learners using Devanagari graphemes, its core analysis targets phonemic distinctions, leveraging Hindi's highly phonetic orthography to analyze mispronounced speech and provide targeted feedback.
Evaluating Wav2Vec2-Bert for Computer-Assisted Pronunciation Training for isiZulu
Authors: Alexandra Fort; Francis Tyers
Venue: Interspeech 2025 (main)

Abstract: Pronunciation training is essential for learning a second language, but tools for computer-assisted pronunciation training have only been developed for a small subset of languages. Through creating and evaluating three fine-tuned versions of Wav2Vec2-Bert, this paper investigates the performance of Wav2Vec2-Bert for detecting language learner errors in low-resource settings. The results provide insight into how the data used for fine-tuning can impact performance and a thorough analysis of erroneous predictions. The evaluation of Wav2Vec2-Bert for this task offers a case study of an under-resourced language and suggestions for how a large-language model can be used to develop pronunciation training tools in under-resourced settings.
Towards Adaptable and Intelligible Speech Synthesis in Noisy Environments
Authors: Lubos Marcinek; Jonas Beskow; Joakim Gustafsson
Venue: Interspeech 2025 (main)

Abstract: We present an investigation into adaptable speech synthesis for noisy environments. Leveraging a zero-shot TTS we synthesized a corpus of 1,200 speech samples from 100 sentences of varying complexity, each generated at six distinct levels of vocal effort. To simulate realistic listening conditions, the synthesized speech is merged with environmental noise recordings from a diverse range of indoor and transportation settings at nine different signal-to-noise ratios. We assess the intelligibility of the resulting noisy speech using the ASR word error rates across conditions. Additionally, the input text was evaluated using four metrics on sentence complexity and word predictability. A number of regression models that used noise type, SNR, vocal effort and text as input were trained to predict ASR WER. Results show that increased vocal effort improves intelligibility, with benefits up to 30% in adverse conditions, most most pronounced in environments with competing speech at low SNRs.
Harnessing Text-to-Speech Voice Cloning Models for Improved Audiological Speech Assessment
Authors: Lidea Shahidi; Erdem Baha Topbas; Thu Ngan Dang; Tobias Goehring
Venue: Interspeech 2025 (main)

Abstract: Conventional audiological speech assessments are limited in their predictive utility, due to the small number of available stimuli and the restricted communications experiences that they represent. To enhance the capabilities of audiological speech assessments, this work evaluates the ability of several text-to-speech voice cloning models on the task of replicating a standard UK open-set speech test used clinically. Models are evaluated using complementary measures of speech perception: speech intelligibility in background noise, speech quality, and speaker discrimination, in a large-scale online study (N = 73). To ensure speech intelligibility measurements are comparable, the psychometric functions are characterized for each model. Results indicate models which accurately and consistently replicate speaker characteristics and produce speech that is similarly intelligible and natural for audiological speech assessment.
75-Speaker Annot-16: A benchmark dataset for speech articulatory rt-MRI annotation with articulator contours and phonetic alignment
Authors: Xuan Shi; Yubin Zhang; Yijing Lu; Marcus Ma; Tiantian Feng; Asterios Toutios; Haley Hsu; Louis Goldstein; Shrikanth Narayanan
Venue: Interspeech 2025 (main)

Abstract: High-quality speech articulatory databases are essential for advancing speech science and technology research. However, the lack of standardized annotations limits their full potential use and broad accessibility. In this context, we introduce 75-Speaker Annot-16, a comprehensive annotation dataset derived from the 75-Speaker vocal tract MRI database. Annot-16 provides phonetic alignments, articulator contour annotations, and handmade ground-truth articulator contours. Our annotation process integrates automated algorithms with expert verification to ensure accuracy and efficiency. To demonstrate its utility, we establish three benchmark tasks: speech phoneme recognition, articulatory contour segmentation, and articulatory phoneme recognition. Annot-16 can serve as a valuable resource for speech modeling, computer vision, and cross-modal learning, bridging engineering applications, speech science, and linguistic research.
Representing Speech Through Autoregressive Prediction of Cochlear Tokens
Authors: Greta Tuckute; Klemen Kotar; Evelina Fedorenko; Daniel Yamins
Venue: Interspeech 2025 (main)

Abstract: We introduce AuriStream, a biologically inspired model for encoding speech via a two-stage framework inspired by the human auditory processing hierarchy. The first stage transforms raw audio into a time-frequency representation based on the human cochlea, from which we extract discrete cochlear tokens. The second stage applies an autoregressive sequence model over the cochlear tokens. AuriStream learns meaningful phoneme and word representations, and state-of-the-art lexical semantics. AuriStream shows competitive performance on diverse downstream SUPERB speech tasks. Complementing AuriStream's strong representational capabilities, it generates continuations of audio which can be visualized in a spectrogram space and decoded back into audio, providing insights into the model's predictions. In summary, we present a two-stage framework for speech representation learning to advance the development of more human-like models that efficiently handle a range of speech-based tasks.
Reasoning-Based Approach with Chain-of-Thought for Alzheimerâs Detection Using Speech and Large Language Models
Authors: Chanwoo Park; Anna Seo Gyeong Choi; Sunghye Cho; Chanwoo Kim
Venue: Interspeech 2025 (main)

Abstract: Societies worldwide are rapidly entering a super-aged era, making elderly health a pressing concern. The aging population is increasing the burden on national economies and households. Dementia cases are rising significantly with this demographic shift. Recent research using voice-based models and large language models (LLM) offers new possibilities for dementia diagnosis and treatment. Our Chain-of-Thought (CoT) reasoning method combines speech and language models. The process starts with automatic speech recognition to convert speech to text. We add a linear layer to an LLM for Alzheimer's disease (AD) and non-AD classification, using supervised fine-tuning (SFT) with CoT reasoning and cues. This approach showed an 16.7% relative performance improvement compared to methods without CoT prompt reasoning. To the best of our knowledge, our proposed method achieved state-of-the-art performance in CoT approaches.
Finding the Human Voice in AI: Insights on the Perception of AI-Voice Clones from Naturalness and Similarity Ratings
Authors: Linda Bakkouche; Charles McGhee; Emily Lau; Stephanie Cooper; Xinbing Luo; Madeleine Rees; Kai Alter; Brechtje Post; Julia Schwarz
Venue: Interspeech 2025 (main)

Abstract: AI-generated voice clones are important tools in language learning, audiobooks, and assistive technology, but often struggle to replicate key prosodic features such as dynamic F0 variation. The impact of these differences on speech perception remain underexplored. To address this, we conducted two behavioural tasks, evaluating listenersâ ratings of naturalness and similarity for human speech, three AI voice clones (ElevenLabs, StyleTTS-2, XTTS-v2), and a 30% F0 variation condition. ElevenLabs was rated comparably to human speech, while StyleTTS-2 and XTTS-v2 received lower ratings. Reduced F0 variation also led to lower ratings, suggesting that prosody is key to perceived naturalness and similarity. Listener ratings were further influenced by speaker accent and sex, but not by AI tool experience. These findings suggest that prosodic features and speaker-specific characteristics could be drivers for the varying performance of AI-voice clones.
Prosodically Enhanced Foreign Accent Simulation by Discrete Token-based Resynthesis Only with Native Speech Corpora
Authors: Kentaro Onda; Keisuke Imoto; Satoru Fukayama; Daisuke Saito; Nobuaki Minematsu
Venue: Interspeech 2025 (main)

Abstract: Recently, a method for synthesizing foreign-accented speech only with native speech data using discrete tokens obtained from self-supervised learning (SSL) models was proposed. Considering limited availability of accented speech data, this method is expected to make it much easier to simulate foreign accents. By using the synthesized accented speech as listening materials for humans or training data for automatic speech recognition (ASR), both of them will acquire higher robustness against foreign accents. However, the previous method has a fatal flaw that it cannot reproduce duration-related accents. Durational accents are commonly seen when L2 speakers, whose native language has syllable-timed or mora-timed rhythm, speak stress-timed languages, such as English. In this paper, we integrate duration modification to the previousÂ method to simulate foreign accents more accurately. Experiments show that the proposed method successfully replicates durational accents seen in real L2 speech.
Weight Factorization and Centralization for Continual Learning in Speech Recognition
Authors: Enes Ugan; Ngoc-Quan Pham; Alexander Waibel
Venue: Interspeech 2025 (main)

Abstract: Modern neural network based speech recognition models are required to continually absorb new data without re-training the whole system, especially in downstream applications using foundation models, having no access to the original training data. Continually training the models in a rehearsal-free, multilingual, and language agnostic condition, likely leads to catastrophic forgetting, when a seemingly insignificant disruption to the weights can destructively harm the quality of the models. Inspired by the ability of human brains to learn and consolidate knowledge through the waking-sleeping cycle, we propose a continual learning approach with two distinct phases: factorization and centralization, learning and merging knowledge accordingly. Our experiments on a sequence of varied code-switching datasets showed that the centralization stage can effectively prevent catastrophic forgetting by accumulating the knowledge in multiple scattering low-rank adapters.
Dysfluent WFST: A Framework for Zero-Shot Speech Dysfluency Transcription and Detection
Authors: Chenxu Guo; Jiachen Lian; Xuanru Zhou; Jinming Zhang; Shuhe Li; Zongli Ye; Peter Park; Anaisha Das; Zoe Ezzes; Jet Vonk; Brittany Morin; Rian Bogley; Lisa Wauters; Zachary Miller; Maria Gorno-Tempini; Gopala Anumanchipalli
Venue: Interspeech 2025 (main)

Abstract: Automatic detection of speech dysfluency aids speech-language pathologists in efficient transcription of disordered speech, enhancing diagnostics and treatment planning. Traditional methods, often limited to classification, provide insufficient clinical insight, and text-independent models misclassify dysfluency, especially in context-dependent cases. This work introduces Dysfluent-WFST, a zero-shot decoder that simultaneously transcribes phonemes and detects dysfluency. Unlike previous models, Dysfluent-WFST operates with upstream encoders like WavLM and requires no additional training. It achieves state-of-the-art performance in both phonetic error rate and dysfluency detection on simulated and real speech data. Our approach is lightweight, interpretable, and effective, demonstrating that explicit modeling of pronunciation behavior in decoding, rather than complex architectures, is key to improving dysfluency processing systems.