GenAI4ELab/papercli-papers huggingface dataset:
A Copula-Based Generative Score-Level Fusion Model for Speaker Verification
Authors: Sandro Cumani
Venue: Interspeech 2025 (main)

Abstract: In this work we present a novel generative approach for the score-level fusion of speaker verification systems. The proposed method employs a copula-based representation of the joint score distribution of multiple speaker recognizers that allows decoupling the dependency structure from the characterization of the marginal densities of the scores of different systems. This allows us to combine complex Variance-Gamma marginals with a simple Gaussian copula to obtain a characterization of the joint target and non-target score distribution that can be effectively employed for the score-level combination of multiple recognizers. Our results on NIST SRE 2019 and SITW datasets show that our approach is competitive with respect to state-of-the-art discriminative score fusion techniques, providing both accurate and well-calibrated scores, with a measured Cllr reduction of up to 7% relative with respect to discriminative linear fusion methods.
Acoustic similarities, articulatory uniqueness: Speech production mechanisms in individuals with congenital lip paralysis
Authors: Anne Hermes; Ivana DidirkovÃ¡; Philipp Buech; Gilles Vannuscorps
Venue: Interspeech 2025 (main)

Abstract: Moebius syndrome is a rare congenital neuromuscular disorder characterized by facial paralysis, which can severely impact speech. Despite these constraints, some individuals with congenital lip paralysis develop very natural speech intelligibility by employing strategies that remain underexplored. This study examines the acoustic-articulatory adaptations in three French-speaking individuals with Moebius syndrome compared to three controls. Using articulatory and acoustic data, we analyzed speech patterns in disyllabic words. Results indicate that Moebius speakers achieve acoustically similar vowel targets to controls despite significant articulatory differences, including increased tongue movement to compensate for lip paralysis. These findings highlight the adaptability of the speech motor system, supporting the notion of motor equivalence in achieving intelligible speech.
Relationship between objective and subjective perceptual measures of speech in individuals with head and neck cancer
Authors: Bence Mark Halpern; Thomas Tienkamp; Teja Rebernik; Rob J.J.H. van Son; Martijn Wieling; Defne Abur; Tomoki Toda
Venue: Interspeech 2025 (main)

Abstract: Meaningful speech assessment is vital in clinical phonetics and therapy monitoring. This study examined the link between perceptual speech assessments and objective acoustic measures in a large head and neck cancer (HNC) dataset. Trained listeners provided ratings of intelligibility, articulation, voice quality, phonation, speech rate, nasality, and background noise on speech. Strong correlations were found between subjective intelligibility, articulation, and voice quality, likely due to a shared underlying cause of speech symptoms in our speaker population. Objective measures of intelligibility and speech rate aligned with their subjective counterpart. Our results suggest that a single intelligibility measure may be sufficient for the clinical monitoring of speakers treated for HNC using concomitant chemoradiation.
Evaluating the Usefulness of Non-Diagnostic Speech Data for Developing Parkinson's Disease Classifiers
Authors: Terry Yi Zhong; Esther Janse; Cristian Tejedor-Garcia; Louis ten Bosch; Martha Larson
Venue: Interspeech 2025 (main)

Abstract: Speech-based Parkinson's disease (PD) detection has gained attention for its automated, cost-effective, and non-intrusive nature. As research studies usually rely on data from diagnostic-oriented speech tasks, this work explores the feasibility of diagnosing PD on the basis of speech data not originally intended for diagnostic purposes, using the Turn-Taking (TT) dataset. Our findings indicate that TT can be as useful as diagnostic-oriented PD datasets like PC-GITA. We also investigate which specific dataset characteristics impact PD classification performance. The results show that concatenating audio recordings and balancing participants' gender and status distributions can be beneficial. Cross-dataset evaluation reveals that models trained on PC-GITA generalize poorly to TT, whereas models trained on TT perform better on PC-GITA. Furthermore, we provide insights into the high variability across folds, which is mainly due to large differences in individual speaker performance.
Multimodal Assessment of Speech Impairment in Amyotrophic Lateral Sclerosis Using Audio-Visual and Machine Learning Approaches
Authors: Francesco Pierotti; Andrea Bandini
Venue: Interspeech 2025 (main)

Abstract: The analysis of speech in individuals with amyotrophic lateral sclerosis is a powerful tool to support clinicians in the assessment of bulbar dysfunction. However, current methods used in clinical practice consist of subjective evaluations or expensive instrumentation. This study investigates different approaches combining audio-visual analysis and machine learning to predict the speech impairment evaluation performed by clinicians. Using a small dataset of acoustic and kinematic features extracted from audio and video recordings of speech tasks, we trained and tested some regression models. The best performance was achieved using the extreme boosting machine regressor with multimodal features, which resulted in a root mean squared error of 0.93 on a scale ranging from 5 to 25. Results suggest that integrating audio-video analysis enhances speech impairment assessment, providing an objective tool for early detection and monitoring of bulbar dysfunction, also in home settings.
Development and Validation of a Wav2Vec 2.0-Based Cross-Language Methodology for Measurement of Articulatory Precision
Authors: Tanya Talkar; Kan Kawabata; Connor Higgins; Sean Tobyne
Venue: Interspeech 2025 (main)

Abstract: Degraded speech intelligibility due to dysarthrias can result from changes inherent to diseases such as Amyotrophic Lateral Sclerosis (ALS). Tracking this degradation longitudinally can highlight treatment effects. Various approaches have been developed to automatically measure speech intelligibility, often focusing on output from automatic speech recognition systems. In this paper, we extract phonetic outputs from the wav2vec 2.0 model and compare them to expected phonetic outputs across 12 languages to derive Articulatory Precision (ArtP). The strongest correlations between ArtP and Azure Pronunciation Assessment were 0.93 for English, 0.85 in German, and 0.66 for Swedish. We additionally find that ArtP has a correlation of 0.77 with the ALSFRS-R speech subscore and is sensitive to perceptual measures of speech intelligibility. The measure shows promise as a reliable and clinically relevant tool that can be used in multiple languages and disorders to assess speech intelligibility.
J-j-j-just Stutter: Benchmarking Whisper's Performance Disparities on Different Stuttering Patterns
Authors: Charan Sridhar; Shaomei Wu
Venue: Interspeech 2025 (main)

Abstract: Despite their prevalence in everyday technologies, automated speech recognition (ASR) systems often struggle with disfluent speech. To diagnose and address these technical challenges, we evaluate OpenAI's Whisper, a state-of-the-art ASR model, using speech samples from podcasts with people who stutter. Our results show significant disparities in Whisper's performance between fluent and stuttered speech. Within disfluent speech, Whisper performs significantly worse on speech with sound repetitions - a disfluency more unique to stuttering. Notably, sound repetitions not only lead to transcription mistakes but also trigger Whisper to hallucinate over 20% of the time. Conducted by researchers who stutter, this study brings new insights on ASR biases against disfluent speech and highlights the value of disability-led research in addressing technological inequities affecting people with disabilities.
MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing
Authors: Junjie Zheng; Zihao Chen; Chaofan Ding; Yunming Liang; Yihan Fan; Huan Yang; Lei Xie; Xinhan Di
Venue: Interspeech 2025 (main)

Abstract: Current movie dubbing technology can produce the desired speech using a reference voice and input video, maintaining perfect synchronization with the visuals while effectively conveying the intended emotions. However, crucial aspects of movie dubbing, including adaptation to various dubbing styles, effective handling of dialogue, narration, and monologues, as well as consideration of subtle details such as speaker age and gender, remain insufficiently explored. To tackle these challenges, we introduce a multi-modal generative framework. First, it utilizes a multi-modal large vision-language model (VLM) to analyze visual inputs, enabling the recognition of dubbing types and fine-grained attributes. Second, it produces high-quality dubbing using large speech generation models, guided by multimodal inputs. Additionally, a movie dubbing dataset with annotations for dubbing types and subtle details is constructed to enhance movie understanding and improve dubbing quality for the proposed multi-modal framework. Experimental results across multiple benchmark datasets show superior performance compared to state-of-the-art (SOTA) methods. In details, the LSED, SPK-SIM, EMO-SIM, and MCD exhibit improvements of up to 1.09%, 8.80%, 19.08%, and 18.74%, respectively.
Learning Phonetic Context-Dependent Viseme for Enhancing Speech-Driven 3D Facial Animation
Authors: Hyung Kyu Kim; Hak Gu Kim
Venue: Interspeech 2025 (main)

Abstract: Speech-driven 3D facial animation aims to generate realistic facial movements synchronized with audio. Traditional methods primarily minimize reconstruction loss by aligning each frame with ground-truth. However, this frame-wise approach often fails to capture the continuity of facial motion, leading to jittery and unnatural outputs due to coarticulation. To address this, we propose a novel phonetic context-aware loss, which explicitly models the influence of phonetic context on viseme transitions. By incorporating a viseme coarticulation weight, we assign adaptive importance to facial movements based on their dynamic changes over time, ensuring smoother and perceptually consistent animations. Extensive experiments demonstrate that replacing the conventional reconstruction loss with ours improves both quantitative metrics and visual quality. It highlights the importance of explicitly modeling phonetic context-dependent visemes in synthesizing natural speech-driven 3D facial animation.
Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation
Authors: Fang Kang; Yin Cao; Haoyu Chen
Venue: Interspeech 2025 (main)

Abstract: Recent studies in speech-driven talking face generation achieve promising results, but their reliance on fixed-driven speech limits further applications (e.g., face-voice mismatch). Thus, we extend the task to a more challenging setting: given a face image and text to speak, generating both talking face animation and its corresponding speeches. Accordingly, we propose a novel framework, Face2VoiceSync, with several novel contributions: 1) Voice-Face Alignment, ensuring generated voices match facial appearance; 2) Diversity & Manipulation, enabling generated voice control over paralinguistic features space; 3) Efficient Training, using a lightweight VAE to bridge visual and audio large-pretrained models, with significantly fewer trainable parameters than existing methods; 4) New Evaluation Metric, fairly assessing the diversity and identity consistency. Experiments show Face2VoiceSync achieves both visual and audio state-of-the-art performances on a single 40GB GPU.
Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis
Authors: Minsu Kim; Pingchuan Ma; Honglie Chen; Stavros Petridis; Maja Pantic
Venue: Interspeech 2025 (main)

Abstract: This paper explores multi-modal controllable Text-to-Speech Synthesis (TTS) where the voice can be generated from face image, and the characteristics of output speech (e.g., pace, noise level, distance, tone, place) can be controllable with natural text description. Specifically, we aim to mitigate the following three challenges in face-driven TTS systems. 1) To overcome the limited audio quality of audio-visual speech corpora, we propose a training method that additionally utilizes high-quality audio-only speech corpora. 2) To generate voices not only from real human faces but also from artistic portraits, we propose augmenting the input face image with stylization. 3) To consider one-to-many possibilities in face-to-voice mapping and ensure consistent voice generation at the same time, we propose to first employ sampling-based decoding and then use prompting with generated speech samples. Experimental results validate the proposed model's effectiveness in face-driven voice synthesis.
VisualSpeech: Enhancing Prosody Modeling in TTS Using Video
Authors: Shumin Que; Anton Ragni
Venue: Interspeech 2025 (main)

Abstract: Text-to-Speech (TTS) synthesis faces the inherent challenge of producing multiple speech outputs with varying prosody given a single text input. While previous research has addressed this by predicting prosodic information from both text and speech, additional contextual information, such as video, remains under-utilized despite being available in many applications. This paper investigates the potential of integrating visual context to enhance prosody prediction. We propose a novel model, VisualSpeech, which incorporates visual and textual information for improving prosody generation in TTS. Empirical results indicate that incorporating visual features improves prosodic modeling, enhancing the expressiveness of the synthesized speech.
LightL2S: Ultra-Low Complexity Lip-to-Speech Synthesis for Multi-Speaker Scenarios
Authors: Yifan Liang; Kang Yang; Fangkun Liu; Andong Li; Xiaodong Li; Chengshi Zheng
Venue: Interspeech 2025 (main)

Abstract: Lip-to-speech synthesis in the wild remains challenging due to the limited visual information. While self-supervised models have shown promising results in relatively high-quality lip-to-speech synthesis, their computational demands make them impractical for edge devices. To address this issue, we introduce LightL2S, a novel multi-speaker lip-to-speech system designed to achieve ultra-low complexity for edge deployment. To reduce the computational cost, we adopt a much more efficient architecture MoViNet for visual encoder instead of using the conventional ResNet-18. Furthermore, we introduce Zipformer blocks to efficiently learn prosodic information and quantized self-supervised audio representations from the output features generated by MoViNet. Finally, we employ the differentiable digital signal processing vocoder to synthesize speech. Experimental results demonstrate that LightL2S can generate reasonable speech even with a computational complexity of only 0.8 GMacs.
Processing of grammatical information in cochlear implant simulated speech by German adult listeners
Authors: Atty Schouwenaars; Esther Ruigendijk
Venue: Interspeech 2025 (main)

Abstract: This study investigates how German which-questions are processed in cochlear implant (CI) simulated speech compared to normal speech. In addition, it explores the role of working memory capacity (WM). Using eye-tracking, accuracy and gaze patterns for subject, object and passive questions were studied. Questions were disambiguated with grammatical information, specifically morphosyntactic cues: case and/or subject-verb agreement. Results show that only object question accuracy was affected by CI simulation. Similarly, eye movements showed weaker interpretation preferences for object questions in CI simulation. Participants with high WM had higher accuracy and their eye gaze showed quicker revisions of garden path effects compared to those with medium or low WM capacity. These findings show that morphosyntactic cues support correct interpretation of object questions in CI simulation, but processing these questions is slower and influenced by working memory capacity.
A Gradient Effect of Hand Beat Timing on Spoken Word Recognition
Authors: Chengjia Ye; James M. McQueen; Hans Rutger Bosker
Venue: Interspeech 2025 (main)

Abstract: Visual cues play a key role in speech perception. Beat gestures (i.e., simple up-and-down hand movements) usually co-occur with prominence in speech. Previous studies found that hand beat timing can indicate word stress. The present study further examines whether hand beat timing influences spoken word recognition in a gradient fashion. On watching videos of a native speaker of Dutch uttering a disyllabic word voornaam while making a hand beat, 40 participants needed to decide if they heard the word with initial (VOORnaam, "first name") or final stress (voorNAAM, "respectable"). Crucially, nine beat apex timings were equally distributed between the pitch peaks of the two syllables. Results exhibited a gradient effect of hand beat timing on stress perception, which appeared not to be susceptible to brief pretest feedback implying that visual cues should be ignored. Our findings provide novel evidence for audiovisual interaction and can inform gesture generation in conversational agents.
The Effect of Word Predictability on Spoken Cross-Language Intelligibility
Authors: Wei Xue; Iuliia Zaitova; Bernd MÃ¶bius
Venue: Interspeech 2025 (main)

Abstract: Cross-language intelligibility refers to how well speakers of language A understand language B without prior learning. While the impact of linguistic and extra-linguistic factors on cross-language intelligibility has been widely studied, the effect of word predictability, known to impact comprehension and speech perception, remains underexplored. This study examines this effect by comparing German and English native speakers translating Dutch words presented in Dutch spoken sentential utterances with varying word predictability. We also investigate whether additional written context would aid cross-language intelligibility. Our results showed that word predictability significantly influences cross-language intelligibility, with German speakers experiencing even stronger effects, whereas only English speakers benefit from the additional written context. These findings suggest that word predictability dynamically shapes cross-language intelligibility, tending to be language-specific.
Sentence-Final Particles in Mandarin Child-Directed Speech: Frequency and Impact on Speech Rate
Authors: Yizhi Liu; Luyuan Geng; Yan Gu; Mengru Han
Venue: Interspeech 2025 (main)

Abstract: This study examined the semi-spontaneous speech produced by 40 Mandarin-speaking mothers during a storytelling task, focusing on the frequency of sentence-final particles (SFPs) and their impact on speech rate. Results showed that although the overall usage of SFPs was higher in the child-directed speech (CDS) than in adult-directed speech (ADS), this increase was limited to declaratives; in contrast, interrogatives showed the opposite pattern. Further analysis revealed that the higher frequency of SFPs in CDS was driven by a higher proportion of interrogatives. For speech rate, CDS utterances with SFPs were spoken faster than those without, a pattern not observed in ADS. Moreover, mothers lengthened the duration of SFPs in both CDS and ADS, resulting in a slower rate for SFPs compared to the rest utterance parts. These findings enhanced our understanding of caregivers' speech strategies when communicating with children and provided new perspectives in the research of Mandarin SFPs.
Bilingual Speakers Exhibit Cognitive Fatigue: A Speech Disfluencies Case Study on Research Talks
Authors: Ashwin Ram; Marisol MuÃ±oz; Zoi Gkalitsiou; Alexandros G. Dimakis
Venue: Interspeech 2025 (main)

Abstract: Speech disfluencies are vital for understanding cognitive processes and improving speech recognition systems. We curate a dataset with annotated text and labeled speech disfluencies from more than 20 hours of speech from monolingual and bilingual speakers. Furthermore, we illustrate a large-scale validation of a bilingual cognitive fatigue phenomenon that seems to be independent of the two spoken languages of the speakers. That is, after navigating a lexically complex word, bilingual speakers tend to use a disfluency, such as a filled pause or repair, followed by a phonetically simpler word in order to possibly regain momentum for subsequent utterance segments. We conclude by exploring how our research can help speech pathologists by revealing distinct bilingual cognitive strategies and how they manifest in speaker disfluencies.
Boosting StoRM Convergence with Metric Guidance and Non-uniform State-Sampling for Optimal Dereverberation
Authors: Chandra Mohan Sharma; Arnab Kumar Roy; Anupam Mandal; Prasanta Kumar Ghosh; Prasanna Kumar Kr
Venue: Interspeech 2025 (main)

Abstract: This paper proposes a novel approach to address late reverberation, which degrades speech intelligibility by convolving clean speech with room impulse response. Our method combines metric-guided training and non-uniform state sampling within the Stochastic Regeneration Model (StoRM) diffusion architecture, enabling better diffusion variability modeling while maintaining computational efficiency. Key metrics such as STFT loss, spectral convergence loss, Mel Frequency Cepstral Coefficient (MFCC) loss and log-magnitude loss guide the regeneration process, improving convergence by reducing training epochs by ~19.6% with slight improvements in dereverberation. Meanwhile, the non-linear state sampling approach enhances training convergence by ~27.2% with practically similar perceptual performance. We evaluate the impact of these modifications on automatic speech recognition and clean speech distortion relative to the baseline, demonstrating optimal speech-quality-aware performance.
Unified Variational and Physics-aware Model for Room Impulse Response Estimation
Authors: Louis Lalay; Mathieu Fontaine; Roland Badeau
Venue: Interspeech 2025 (main)

Abstract: Room impulse response estimation is essential for tasks like speech dereverberation, which improves automatic speech recognition. Most existing methods rely on either statistical signal processing or deep neural networks designed to replicate signal processing principles. However, combining statistical and physical modeling for RIR estimation remains largely unexplored. This paper proposes a novel approach integrating both aspects through a theoretically grounded model. The RIR is decomposed into interpretable parameters: white Gaussian noise filtered by a frequency-dependent exponential decay (e.g. modeling wall absorption) and an autoregressive filter (e.g. modeling microphone response). A variational free-energy cost function enables practical parameter estimation. As a proof of concept, we show that given dry and reverberant speech signals, the proposed method outperforms classical deconvolution in noisy environments, as validated by objective metrics.