GenAI4ELab/papercli-papers huggingface dataset:
Text Entry for All: Towards Speech-based Multimodal Interaction for Inclusion, Accessibility and the Preservation of the Worldâs Linguistic Heritage
Authors: Julian Zapata; Lara Hanna
Venue: Interspeech 2025 (main)

Abstract: This paper describes an emerging project that aims at better understanding how speakers of different languages, with diverse needs, produce texts today, and rethinking how they should do so in the future. Text entry (TE), the process of inputting text into a computer, is a crucial part of our interaction with technology. However, research has shown that the conventional TE method--typing on a keyboard--is not the most efficient or appropriate way of inputting text in various contexts, for diverse users. Now, how can we design more effective, accessible and inclusive TE methods that consider the uniqueness of different languages and diverse users and use cases? This fascinating question is likely to motivate speech technology, writing-process and human-computer interaction researchers alike, particularly in the generative AI age. But we are also living in the age of speech and multimodal interactions, which offer unprecedented opportunities to reinvent how we learn, write and communicate.
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach
Authors: Umberto Cappellazzo; Minsu Kim; Stavros Petridis; Daniele Falavigna; Alessio Brutti
Venue: Interspeech 2025 (main)

Abstract: Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy environments by integrating visual cues. While recent advances integrate Large Language Models (LLMs) into AVSR, their high computational cost hinders deployment in resource-constrained settings. To address this, we propose Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of Projectors (SMoP) module to scale model capacity without increasing inference costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors, Llama-SMoP enables the use of smaller LLMs while maintaining strong performance. We explore three SMoP configurations and show that Llama-SMoP DEDR (Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation studies confirm its effectiveness in expert activation, scalability, and noise robustness.
Cocktail-Party Audio-Visual Speech Recognition
Authors: Thai-Binh Nguyen; Ngoc-Quan Pham; Alexander Waibel
Venue: Interspeech 2025 (main)

Abstract: Audio-Visual Speech Recognition (AVSR) offers a robust solution for speech recognition in challenging environments, such as cocktail-party scenarios, where relying solely on audio proves insufficient. However, current AVSR models are often optimized for idealized scenarios with consistently active speakers, overlooking the complexities of real-world settings that include both speaking and silent facial segments. This study addresses this gap by introducing a novel audio-visual cocktail-party dataset designed to benchmark current AVSR systems and highlight the limitations of prior approaches in realistic noisy conditions. Additionally, we contribute a 1526-hour AVSR dataset comprising both talking-face and silent-face segments, enabling significant performance gains in cocktail-party environments. Our approach reduces WER by 67% relative to the state-of-the-art, reducing WER from 119% to 39.2% in extreme noise, without relying on explicit segmentation cues.
Efficient Noise-Robust Hybrid Audiovisual Encoder with Joint Distillation and Pruning for Audiovisual Speech Recognition
Authors: Zhengyang Li; Pascal Reichert; Thomas Graave; Patrick Blumenberg; Tim Fingscheidt
Venue: Interspeech 2025 (main)

Abstract: Powered by self-supervised learning (SSL) on vast amounts of unlabeled data, a computationally intensive audiovisual encoder -a hybrid architecture combining ResNet and transformer in series- achieves state-of-the-art performance in audiovisual speech recognition (AV-ASR). In this work, we are the first to apply joint distillation and pruning (DP) with a teacher-student model for an efficient and noise-robust audiovisual encoder. First, we compress the transformer of the AV encoder. Second, we extend joint DP to both the ResNet and transformer of the hybrid AV encoder. In addition, we provide analyses on the teacher and the final student, respectively. With a similar number of parameters, our proposed student outperforms the previous state-of-the-art in clean condition (word error rate of 3.1% vs. 4.6%) and across all noisy conditions, while at the same time reducing computational complexity by 31.8%. Our code is at GitHub.
Unified Audio-Visual Modeling for Recognizing Which Face Spoke When and What in Multi-Talker Overlapped Speech and Video
Authors: Naoki Makishima; Naotaka Kawata; Taiga Yamane; Mana Ihori; Tomohiro Tanaka; Satoshi Suzuki; Shota Orihashi; Ryo Masumura
Venue: Interspeech 2025 (main)

Abstract: We have developed a model that jointly recognizes which face spoke âwhenâ and âwhatâ from multi-talker overlapped speech and video of multiple speakers. For understanding of videos in which multiple speakers are speaking, it is important to recognize which face spoke âwhenâ and âwhatâ from the overlapped speech and multiple speakersâ videos. Conventional methods have to combine speech separation, active speaker detection, and audio-visual speech recognition to address this task. However, the combined system makes the system complex and suboptimal. To address this problem, our idea is to serialize âwhich face spoke when and whatâ of multiple speakers into a single token sequence, which is recursively estimated with the proposed unified model using multiple speakersâ videos and overlapped speech as input. Experimental results demonstrate the validity of the proposed method.
Leveraging LLM for Stuttering Speech: A Unified Architecture Bridging Recognition and Event Detection
Authors: Shangkun Huang; Jing Deng; Jintao Kang; Rong Zheng
Venue: Interspeech 2025 (main)

Abstract: The performance bottleneck of Automatic Speech Recognition (ASR) in stuttering speech scenarios has limited its applicability in domains such as speech rehabilitation. This paper proposed an LLM-driven ASR-SED multi-task learning framework that jointly optimized the ASR and Stuttering Event Detection (SED) tasks. We proposed a dynamic interaction mechanism where the ASR branch leveraged CTC-generated soft prompts to assist LLM context modeling, while the SED branch output stutter embeddings to enhance LLM comprehension of stuttered speech. We incorporated contrastive learning to strengthen the discriminative power of stuttering acoustic features and applied Focal Loss to mitigate the long-tailed distribution in stuttering event categories. Evaluations on the AS-70 Mandarin stuttering dataset demonstrated that our framework reduced the ASR character error rate (CER) to 5.45% (-37.71% relative reduction) and achieved an average SED F1-score of 73.63% (+46.58% relative improvement).
Seamless Dysfluent Speech Text Alignment for Disordered Speech Analysis
Authors: Zongli Ye; Jiachen Lian; Xuanru Zhou; Jinming Zhang; Haodong Li; Shuhe Li; Chenxu Guo; Anaisha Das; Peter Park; Zoe Ezzes; Jet Vonk; Brittany Morin; Rian Bogley; Lisa Wauters; Zachary Miller; Maria Gorno-Tempini; Gopala Anumanchipalli
Venue: Interspeech 2025 (main)

Abstract: Accurate alignment of dysfluent speech with intended text is crucial for automating the diagnosis of neurodegenerative speech disorders. Traditional methods often fail to model phoneme similarities effectively, limiting their performance. In this work, we propose Neural LCS, a novel approach for dysfluent text-text and speech-text alignment. Neural LCS addresses key challenges, including partial alignment and context-aware similarity mapping, by leveraging robust phoneme-level modeling. We evaluate our method on a large-scale simulated dataset, generated using advanced data simulation techniques, and real PPA data. Neural LCS significantly outperforms state-of-the-art models in both alignment accuracy and dysfluent speech segmentation. Our results demonstrate the potential of Neural LCS to enhance automated systems for diagnosing and analyzing speech disorders, offering a more accurate and linguistically grounded solution for dysfluent speech alignment.
Analysis and Evaluation of Synthetic Data Generation in Speech Dysfluency Detection
Authors: Jinming Zhang; Xuanru Zhou; Jiachen Lian; Shuhe Li; William Li; Zoe Ezzes; Rian Bogley; Lisa Wauters; Zachary Miller; Jet Vonk; Brittany Morin; Maria Gorno-Tempini; Gopala Anumanchipalli
Venue: Interspeech 2025 (main)

Abstract: Speech dysfluency detection is crucial for clinical diagnosis and language assessment, but existing methods are limited by the scarcity of high-quality annotated data. Although recent advances in TTS model have enabled synthetic dysfluency generation, existing synthetic datasets suffer from unnatural prosody and limited contextual diversity. To address these limitations, we propose LLM-Dys - the most comprehensive dysfluent speech corpus with LLM-enhanced dysfluency simulation. This dataset captures 11 dysfluency categories spanning both word and phoneme levels. Building upon this resource, we improve an end-to-end dysfluency detection framework. Experimental validation demonstrates state-of-the-art performance.
Fine-tuning Strategies for Automatic Speech Recognition of Low-Resource Speech with Autism Spectrum Disorder
Authors: Yeseul Park; Bowon Lee
Venue: Interspeech 2025 (main)

Abstract: Individuals with autism spectrum disorder (ASD) exhibit unique speech patterns that challenge conventional automatic speech recognition (ASR) systems. However, research on ASD-adapted ASR models remains limited. This study explores fine-tuning strategies for ASD-specific ASR models using Whisper, comparing full fine-tuning, selective fine-tuning, adapter tuning, and LoRA-based fine-tuning. Experiments using a small-scale Korean ASD speech dataset demonstrate that adapter tuning and LoRA significantly reduce the character error rate (CER) while reducing trainable parameters. In case of Whisper-small, adapter tuning and LoRA improve the CER by 7.22% and 10.14% over full fine-tuning, respectively. Furthermore, LoRA improved CER by 10.35% and 10.15% with Whisper-base and Whisper-large-v2 models compared to full fine-tuning. These results demonstrate the adaptation efficiency and effectiveness of LoRA for low resource ASD speech dataset.
Identification of Pathological Pronunciation Profiles in ASR Transcription Errors
Authors: Margot Masson; Isabelle FerranÃ©; Julie Mauclair
Venue: Interspeech 2025 (main)

Abstract: Despite recent developments, ASR systems still struggle to handle atypical speech. This difference in performance has been leveraged to use ASR to automatically assess speech quality in the context of speech disorders. In the first part of this paper, we confirm the correlation between ASR sensitivity and speech quality. At the same time, ASR systems remain black-box and difficult to interpret. Therefore, we explore ASR performance for Head and Neck Cancer and Parkinson's disease speech disorders in the second part of this paper. We hypothesise that ASR errors are representative of pronunciation patterns caused by these disorders. We build pronunciation profiles from ASR transcription errors for each category and assess the representativeness of these profiles by synthesising speech variants including the variations identified in the profiles. Analysis of these variants shows that ASR transcription errors are characteristic of pronunciation patterns caused by speech disorders.
A simple method for predicting Clinical Scores in Huntingtonâs Disease by leveraging ASR's uncertainty on spontaneous speech
Authors: Hadrien Titeux; Quang Tuan RÃ©my Nguyen; Andres Gil-Salcedo; Anne-Catherine Bachoud-Levi; Emmanuel Dupoux
Venue: Interspeech 2025 (main)

Abstract: Recent automatic speech recognition (ASR) models struggle to correctly transcribe pathological speech but implicitly capture phonetic, syntactic, and semantic properties. Unlike state-of-the-art methods that rely on speech features based on time-consuming handcrafted speech transcription, we propose a simple and fully automated approach using ASR log-probabilities to quantify intelligibility in spontaneous speech of patients with Huntingtonâs disease. By linking this measure to clinical scores, we explore its potential as a scalable, lightweight biomarker for disease progression. Our findings suggest that ASR-derived uncertainty offers a novel, efficient, and noninvasive alternative for clinical assessment.
Introducing EMOPARKNZ: the Emotional Speech Database from New Zealand English Speakers with Parkinsonâs Disease
Authors: Itay Ben-Dom; Catherine I. Watson; Clare M. McCann
Venue: Interspeech 2025 (main)

Abstract: This paper introduces EMOPARKNZ, a new speech database designed to study emotional speech production in New Zealand English speakers with Parkinson's disease. The database features 13 speakers (7 male, 6 female) and 1,950 speech recordings expressing five primary emotions: excited, angry, happy, sad, and neutral. The design considerations, participant selection, and recording procedures are detailed. Acoustic analysis of the database revealed significant differences in fundamental frequency, intensity, and speech rate across emotional states. A preliminary perception test with 22 New Zealand English listeners achieved 63% classification accuracy across the five emotions, aligning with the performance reported for similar emotional speech databases. The results indicate EMOPARKNZ is a valuable resource for advancing research in emotion communication for individuals with Parkinson's disease and speech emotion classification.
Revisiting WFST-based Hybrid Japanese Speech Recognition System for Individuals with Organic Speech Disorders
Authors: Naoki Hojo; Ryoichi Takashima; Chihiro Sugiyama; Nobukazu Tanaka; Kanji Nohara; Kazunori Nozaki; Tetsuya Takiguchi
Venue: Interspeech 2025 (main)

Abstract: End-to-end automatic speech recognition (ASR) technology has advanced significantly, yet it remains ineffective for individuals with speech disorders. In particular, Japanese ASR faces unique challenges due to its diverse character set, including kanji (Chinese characters), hiragana, and katakana (Japanese phonetic syllabary). Adapting an end-to-end ASR model to this task requires extensive training data. In this paper, we revisit a WFST-based hybrid ASR system that decomposes the system into an acoustic model, a pronunciation dictionary, and a language model. This approach is effective when speech data is limited because it uses speech data only to train the acoustic model, while the other components learn only from text data. In addition, to enhance the acoustic model, we introduce multi-step model adaptation using synthetic speech. Experimental results with speakers with organic speech disorders demonstrated that the proposed system outperformed Whisper.
Pseudo Labels-based Neural Speech Enhancement for the AVSR Task in the MISP-Meeting Challenge
Authors: Longjie Luo; Shenghui Lu; Lin Li; Qingyang Hong
Venue: Interspeech 2025 (main)

Abstract: This paper presents our system for the MISP-Meeting Challenge Track 2. The primary difficulty lies in the dataset, which contains strong background noise, reverberation, overlapping speech, and diverse meeting topics. To address these issues, we (a) designed G-SpatialNet, a speech enhancement (SE) model to improve Guided Source Separation (GSS) signals; (b) proposed TLS, a framework comprising time alignment, level alignment, and signal-to-noise ratio filtering, to generate signal-level pseudo labels for real-recorded far-field audio data, thereby facilitating SE models' training; and (c) explored fine-tuning strategies, data augmentation, and multimodal information to enhance the performance of pre-trained Automatic Speech Recognition (ASR) models in meeting scenarios. Finally, our system achieved character error rates (CERs) of 5.44% and 9.52% on the Dev and Eval sets, respectively, with relative improvements of 64.8% and 52.6% over the baseline, securing second place.
The Multimodal Information Based Speech Processing (MISP) 2025 Challenge: Audio-Visual Diarization and Recognition
Authors: Ming Gao; Shilong Wu; Hang Chen; Jun Du; Chin-Hui Lee; Shinji Watanabe; Jingdong Chen; Sabato Marco Siniscalchi; Odette Scharenborg
Venue: Interspeech 2025 (main)

Abstract: Meetings are a valuable yet challenging scenario for speech applications due to complex acoustic conditions. This paper summarizes the outcomes of the MISP 2025 Challenge, hosted at Interspeech 2025, which focuses on multi-modal, multi-device meeting transcription by incorporating video modality alongside audio. The tasks include Audio-Visual Speaker Diarization (AVSD), Audio-Visual Speech Recognition (AVSR), and Audio-Visual Diarization and Recognition (AVDR). We present the challengeâs objectives, tasks, dataset, baseline systems, and solutions proposed by participants. The best-performing systems achieved significant improvements over the baseline: the top AVSD model achieved a Diarization Error Rate (DER) of 8.09%, improving by 7.43%; the top AVSR system achieved a Character Error Rate (CER) of 9.48%, improving by 10.62%; and the best AVDR system achieved a concatenated minimum-permutation Character Error Rate (cpCER) of 11.56%, improving by 72.49%.
Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge
Authors: Zhaoyang Li; Haodong Zhou; Longjie Luo; XiaoXiao Li; Yongxin Chen; Lin Li; Qingyang Hong
Venue: Interspeech 2025 (main)

Abstract: This paper presents the system developed for Task 1 of the Multi-modal Information-based Speech Processing (MISP) 2025 Challenge. We introduce CASA-Net, an embedding fusion method designed for end-to-end audio-visual speaker diarization (AVSD) systems. CASA-Net incorporates a cross-attention (CA) module to effectively capture cross-modal interactions in audio-visual signals and employs a self-attention (SA) module to learn contextual relationships among audio-visual frames. To further enhance performance, we adopt a training strategy that integrates pseudo-label refinement and retraining, improving the accuracy of timestamp predictions. Additionally, median filtering and overlap averaging are applied as post-processing techniques to eliminate outliers and smooth prediction labels. Our system achieved a diarization error rate (DER) of 8.18% on the evaluation set, representing a relative improvement of 47.3% over the baseline DER of 15.52%.
Multi-Channel Sequence-to-Sequence Neural Diarization: Experimental Results for The MISP 2025 Challenge
Authors: Ming Cheng; Fei Su; Cancan Li; Juan Liu; Ming Li
Venue: Interspeech 2025 (main)

Abstract: This paper describes the speaker diarization system developed for the Multimodal Information-Based Speech Processing (MISP) 2025 Challenge. First, we utilize the Sequence-to-Sequence Neural Diarization (S2SND) framework to generate initial predictions using single-channel audio. Then, we extend the original S2SND framework to create a new version, Multi-Channel Sequence-to-Sequence Neural Diarization (MC-S2SND), which refines the initial results using multi-channel audio. The final system achieves a diarization error rate (DER) of 8.09% on the evaluation set of the competition database, ranking first place in the speaker diarization task of the MISP 2025 Challenge.
Leveraging Self-Supervised Learning Based Speaker Diarization for MISP 2025 AVSD Challenge
Authors: Zeyan Song; Tianchi Sun; Ronghui Hu; Kai Chen; Jing Lu
Venue: Interspeech 2025 (main)

Abstract: This paper presents the submission of our team to the audio-visual speaker diarization (AVSD) track of the Multimodal Information Based Speech Processing (MISP) 2025 Challenge. The submitted system is adapted from the DiariZen pipeline, with a primary focus on optimizing it for the challenge dataset. The pipeline consists of a WavLM based local end-to-end neural diarization module followed by two different clustering methods. To further refine the results, DOVER-Lap is employed to integrate results across different input channels and clustering methods. Our final submission system achieves a diarization error rate (DER) of 8.33% on the evaluation set, representing a relative improvement of 46.3% compared to the baseline and ranking 3rd in the AVSD track of this challenge.
Overlap-Adaptive Hybrid Speaker Diarization and ASR-Aware Observation Addition for MISP 2025 Challenge
Authors: Shangkun Huang; Yuxuan Du; Jingwen Yang; Dejun Zhang; Xupeng Jia; Jing Deng; Jintao Kang; Rong Zheng
Venue: Interspeech 2025 (main)

Abstract: This paper presents the system developed to address the MISP 2025 Challenge. For the diarization system, we proposed a hybrid approach combining a WavLM end-to-end segmentation method with a traditional multi-module clustering technique to adaptively select the appropriate model for handling varying degrees of overlapping speech. For the automatic speech recognition (ASR) system, we proposed an ASR-aware observation addition method that compensates for the performance limitations of Guided Source Separation (GSS) under low signal-to-noise ratio conditions. Finally, we integrated the speaker diarization and ASR systems in a cascaded architecture to address Track 3. Our system achieved character error rates (CER) of 9.48% on Track 2 and concatenated minimum permutation character error rate (cpCER) of 11.56% on Track 3, ultimately securing first place in both tracks and thereby demonstrating the effectiveness of the proposed methods in real-world meeting scenarios.
Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling
Authors: Md Asif Jalal; Luca Remaggi; Vasileios Moschopoulos; Thanasis Kotsiopoulos; Vandana Rajan; Karthikeyan Saravanan; Anastasis Drosou; Junho Heo; Hyuk Oh; Seokyeong Jeong
Venue: Interspeech 2025 (main)

Abstract: Traditional speech separation and speaker diarization approaches rely on prior knowledge of target speakers or a predetermined number of participants in audio signals. To address these limitations, recent advances focus on developing enrollment-free methods capable of identifying targets without explicit speaker labelling. This work introduces a new approach to train simultaneous speech separation and diarization using automatic identification of target speaker embeddings within mixtures. Our proposed model employs a dual-stage training pipeline designed to learn robust speaker representation features that are resilient to background noise interference. Furthermore, we present an overlapping spectral loss function specifically tailored for enhancing diarization accuracy during overlapped speech frames. Experimental results show significant performance gains compared to the current SOTA baseline, achieving 71% relative reduction in DER and 69% in cpWER.
Inter-Speaker Relative Cues for Text-Guided Target Speech Extraction
Authors: Wang Dai; Archontis Politis; Tuomas Virtanen
Venue: Interspeech 2025 (main)

Abstract: We propose a novel approach that utilize inter-speaker relative cues for distinguishing target speakers and extracting their voices from mixtures. Continuous cues (e.g., temporal order, age, pitch level) are grouped by relative differences, while discrete cues (e.g., language, gender, emotion) retain their categories. Relative cues offers greater flexibility than fixed speech attribute classification, facilitating much easier expansion of text-guided target speech extraction datasets. Our experiments show that combining all relative cues yields better performance than random subsets, with gender and temporal order being the most robust across languages and reverberant conditions. Additional cues like pitch level, loudness, distance, speaking duration, language, and pitch range also demonstrate notable benefit in complex scenarios. Fine-tuning pre-trained WavLM Base+ CNN encoders improves overall performance over the baseline of using only a Conv1d encoder.