GenAI4ELab/papercli-papers huggingface dataset:
Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning
Authors: Long Mai; Julie Carson-Berndsen
Venue: Interspeech 2025 (main)

Abstract: While Large Language Models (LLMs) have made significant strides in replicating human-like abilities, there are concerns about a reduction in the linguistic diversity of their outputs. This results in the homogenization of viewpoints and perspectives, as well as the underrepresentation of specific demographic groups. Although several fine-tuning and prompting techniques have been suggested to tackle the issue, they are often tailored to specific tasks or come with a substantial increase in computational cost and latency. This makes them challenging to apply to applications that demand very low latency, such as spoken chatbots or virtual assistants. We propose Possibility Exploration Fine-Tuning (PoExFT), a task-agnostic framework that enhances response diversity of LLMs without increasing latency. Given the same prompt, models fine-tuned with PoExFT can simultaneously generate multiple diverse responses. Experiments on dialogue and story generation tasks show that PoExFT significantly enhances the diversity of LLMs, as evidenced by lower similarity between candidate responses. As PoExFT emphasizes semantic diversity over lexical diversity, it can also notably reduce demographic bias in dialogue systems.
OpusLM: A Family of Open Unified Speech Language Models
Authors: Jinchuan Tian; William Chen; Yifan Peng; Jiatong Shi; Siddhant Arora; Shikhar Bharadwaj; Takashi Maekaku; Yusuke Shinohara; Keita Goto; Xiang Yue; Huck Yang; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: This paper presents Open Unified Speech Language Models (OpusLMs), a family of open foundational speech language models (SpeechLMs) up to 7B. Initialized from decoder-only text language models, the OpusLMs are continuously pre-trained on 213K hours of speech-text pairs and 292B text-only tokens. We demonstrate our OpusLMs achieve comparable (or even superior) performance with existing SpeechLMs in speech recognition, speech synthesis, and text-only capabilities. Technically, this paper articulates our SpeechLM designs on tokenization, multi-stream language models, and multi-stage training strategies. We experimentally demonstrate the importance of model size scaling and the effect of annealing data selection. The OpusLMs are all built from publicly available materials and are fully transparent models. We release our code, data, checkpoints, and training logs to facilitate open SpeechLM research.
CAPR: Confidence-Aware Prompt Refinement in Large Language Models
Authors: Jen-Tzung Chien; Po-Chun Huang
Venue: Interspeech 2025 (main)

Abstract: Mitigating hallucination in large language models (LLMs) is crucial to ensure trustworthy generation. A meaningful solution to tackle this issue involves eliciting a reliable confidence score in the generation. However, the previous methods only focused on using white-box models for short-form generation. The solutions to long-form sentences with black-box models remain very limited. This study proposes a reliable black-box confidence elicitation for long-form generation by leveraging the external knowledge, inspired by retrieval augmented generation. In particular, this paper proposes the confidence-aware prompt refinement (CAPR) to modify user prompts to ensure trustworthy responses. The prompt refiner is trained by maximizing the rewards based on the confidence and the accuracy of generated sentences. Experiments show that CAPR effectively elicits reliable confidence scores. The refined prompts enable LLMs to produce reliable responses with calibrated confidence.
The Interspeech 2025 Speech Accessibility Project Challenge
Authors: Xiuwen Zheng; Bornali Phukon; Jonghwan Na; Ed Cutrell; Kyu J. Han; Mark Hasegawa-Johnson; Pan-Pan Jiang; Aadhrik Kuila; Colin Lea; Bob MacDonald; Gautam Mantena; Venkatesh Ravichandran; Leda Sari; Katrin Tomanek; Chang D. Yoo; Chris Zwilling
Venue: Interspeech 2025 (main)

Abstract: While the last decade has witnessed significant advancements in Automatic Speech Recognition (ASR) systems, performance of these systems for individuals with speech disabilities remains inadequate, partly due to limited public training data. To bridge this gap, the 2025 Interspeech Speech Accessibility Project (SAP) Challenge was launched, utilizing over 400 hours of SAP data collected and transcribed from more than 500 individuals with diverse speech disabilities. Hosted on EvalAI and leveraging the remote evaluation pipeline, the SAP Challenge evaluates submissions based on Word Error Rate and Semantic Score. Consequently, 12 out of 22 valid teams outperformed the whisper-large-v2 baseline in terms of WER, while 17 teams surpassed the baseline on SemScore. Notably, the top team achieved the lowest WER of 8.11%, and the highest SemScore of 88.44% at the same time, setting new benchmarks for future ASR systems in recognizing impaired speech.
Towards Inclusive and Fair ASR: Insights from the SAPC Challenge for Optimizing Disordered Speech Recognition
Authors: Nada Gohider; Otman Basir
Venue: Interspeech 2025 (main)

Abstract: ASR has advanced significantly, yet remains limited for impaired speakers due to data scarcity. In response to this gap, the Speech Accessibility Project (SAP) represents a significant initiative in data collection on impaired speech. This paper reports our participation in the SAPC challenge, where we leveraged SAP data to improve ASR performance for disordered speech. Our system ranked fourth in terms of Word Error Rate, recording values of 10.06% and 11.8% WER for Test1 and Test2 subsets, respectively, on the challenge leaderboard. In particular, our research examines the power of SOTA ASR models to capture contextual information in the presence of disordered speech disfluencies. We focused on two ASR architectures, ContextNet and Parakeet, based on their documented ability to efficiently and effectively handle contextual information for typical speech, utilizing distinct mechanisms. Our experiments demonstrated that Parakeet slightly outperformed ContextNet, as evidenced by WER.
Robust fine-tuning of speech recognition models via model merging: application to disordered speech
Authors: Alexandre Ducorroy; Rachid Riad
Venue: Interspeech 2025 (main)

Abstract: Automatic Speech Recognition (ASR) has advanced with Speech Foundation Models (SFMs), yet performance degrades on dysarthric speech due to variability and limited data. This study as part of the submission to the Speech Accessibility challenge, explored model merging to improve ASR generalization using Whisper as the base SFM. We compared fine-tuning with single-trajectory merging, combining models from one fine-tuning path, and multi-run merging, merging independently trained models.  Our best multi-run merging approach achieved a 12% relative decrease of WER over classic fine-tuning, and a 16.2% relative decrease on long-form audios, a major loss contributor in dysarthric ASR. Merging more and more models led to continuous gains, remained effective in low-data regimes, and generalized across model architectures. These results highlight model merging as an easily replicable adaptation method that consistently improves ASR without additional inference cost or hyperparameter tuning.
Exploring Generative Error Correction for Dysarthric Speech Recognition
Authors: Moreno La Quatra; Alkis Koudounas; Valerio Mario Salerno; Sabato Marco Siniscalchi
Venue: Interspeech 2025 (main)

Abstract: Despite the remarkable progress in end-to-end Automatic Speech Recognition (ASR) engines, accurately transcribing dysarthric speech remains a major challenge. In this work, we proposed a two-stage framework for the Speech Accessibility Project Challenge at INTERSPEECH 2025, which combines cutting-edge speech recognition models with LLM-based generative error correction (GER). We assess different configurations of model scales and training strategies, incorporating specific hypothesis selection to improve transcription accuracy. Experiments on the Speech Accessibility Project dataset demonstrate the strength of our approach on structured and spontaneous speech, while highlighting challenges in single-word recognition. Through comprehensive analysis, we provide insights into the complementary roles of acoustic and linguistic modeling in dysarthric speech recognition.
Pathology-Aware Speech Encoding and Data Augmentation for Dysarthric Speech Recognition
Authors: Ilja Baumann; Dominik Wagner; Korbinian Riedhammer; Tobias Bocklet
Venue: Interspeech 2025 (main)

Abstract: Automatic speech recognition (ASR) for pathologic speech remains a major challenge due to high variability in articulation, phonation, and prosody distortions. In this work, we propose a pathology-aware speech encoder based on BEST-RQ pre-training, which incorporates 46k hours of speech, including pathologic and atypical speech. We continue pre-training for domain adaptation and experiment with etiology-specific codebooks. We achieve a 13.2% relative word error rate (WER) improvement using the pathology-aware speech encoder with etiology-specific continued pre-training. Additionally, we examine the impact of incorporating synthetic and out-of-domain (OOD) data to further enhance ASR performance. Synthetic data reduces WER by up to 8.7%, while OOD data improves WER by 12.2%. Finally, we introduce a semantic similarity-based data augmentation technique to optimize data selection, achieving a WER improvement of up to 9.7% while minimizing the need for additional training data.
Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition
Authors: Dominik Wagner; Ilja Baumann; Natalie Engert; Seanie Lee; Elmar NÃ¶th; Korbinian Riedhammer; Tobias Bocklet
Venue: Interspeech 2025 (main)

Abstract: In this work, we present our submission to the Speech Accessibility Project challenge for dysarthric speech recognition. We integrate parameter-efficient fine-tuning with latent audio representations to improve an encoder-decoder ASR system. Synthetic training data is generated by fine-tuning Parler-TTS to mimic dysarthric speech, using LLM-generated prompts for corpus-consistent target transcripts. Personalization with x-vectors consistently reduces word error rates (WERs) over non-personalized fine-tuning. AdaLoRA adapters outperform full fine-tuning and standard low-rank adaptation, achieving relative WER reductions of ~23% and ~22%, respectively. Further improvements (~5% WER reduction) come from incorporating wav2vec 2.0-based audio representations. Training with synthetic dysarthric speech yields up to ~7% relative WER improvement over personalized fine-tuning alone.
A Self-Training Approach for Whisper to Enhance Long Dysarthric Speech Recognition
Authors: Shiyao Wang; Jiaming Zhou; Shiwan Zhao; Yong Qin
Venue: Interspeech 2025 (main)

Abstract: Dysarthric speech recognition (DSR) enhances the accessibility of smart devices for dysarthric speakers with limited mobility. Previously, DSR research was constrained by the fact that existing datasets typically consisted of isolated words, command phrases, and a limited number of sentences spoken by a few individuals. This constrained research to command-interaction systems and speaker adaptation. The Speech Accessibility Project (SAP) changed this by releasing a large and diverse English dysarthric dataset, leading to the SAP Challenge to build speaker- and text-independent DSR systems. We enhanced the Whisper model's performance on long dysarthric speech via a novel self-training method. This method increased training data and adapted the model to handle potentially incomplete speech segments encountered during inference. Our system achieved second place in both Word Error Rate and Semantic Score in the SAP Challenge.
Fine-tuning Parakeet-TDT for Dysarthric Speech Recognition in the Speech Accessibility Project Challenge
Authors: Kaito Takahashi; Keigo Hojo; Toshimitsu Sakai; Yukoh Wakabayashi; Norihide Kitaoka
Venue: Interspeech 2025 (main)

Abstract: We present our dysarthric speech recognition system submitted to the Interspeech 2025 Speech Accessibility Project Challenge. This challenge is a competition aimed at improving the recognition accuracy of dysarthric speech. In this challenge, we submitted a speech recognition system with high accuracy for dysarthric speech and achieved first place. In dysarthric speech recognition, models based on self-supervised learning are commonly used. However, we hypothesized that fine-tuning pre-trained model with inherently high recognition accuracy would achieve the best performance. To this end, we developed a model that combines various techniques, including data preprocessing to expand training datasets, data augmentation to enhance generalization during fine-tuning, and decoding acceleration to optimize inference speed. As a result, our speech recognizer greatly improved accuracy, reducing WER to 8.11 from the baseline Whisper large v2's 17.82 provided by the organizers.
CBA-Whisper: Curriculum Learning-Based AdaLoRA Fine-Tuning on Whisper for Low-Resource Dysarthric Speech Recognition
Authors: Tianyi Tan; Xinan Chen; Xiaohuai Le; Wenzhi Fan; Xianjun Xia; Chuanzeng Huang; Jing Lu
Venue: Interspeech 2025 (main)

Abstract: Whisper is a powerful automatic speech recognition (ASR) model. However, its zero-shot performance on low-resource speech requires further improvement, especially in dysarthric speech recognition (DSR). This paper addresses the Interspeech 2025 Speech Accessibility Project Challenge (SAPC) by fine-tuning Whisper large-v2 on SAP 2024-04-30, UA-Speech, and TORGO datasets using adaptive low-rank adaptation (AdaLoRA). We incorporate improved WhisperX processing and rule-based postprocessing designed for stuttering and machine hallucination. In addition, we employ curriculum learning (CL) with adaptively optimized data filtering to progressively enhance the performance of our model. Using less than 1/5 of the official training data, our final system ranked 2nd in this challenge, with a WER of 10.51% and a SemScore of 85.50% on Test 2 Split, reducing WER by 41.02% and improving SemScore by 12.72% over the baseline (WER 17.82%, SemScore 75.85%). Code and models are publicly available.
Cross-lingual Data Selection Using Clip-level Acoustic Similarity for Enhancing Low-resource Automatic Speech Recognition
Authors: Shunsuke Mitsumori; Sara Kashiwagi; Keitaro Tanaka; Shigeo Morishima
Venue: Interspeech 2025 (main)

Abstract: This paper presents a novel donor data selection method to enhance low-resource automatic speech recognition (ASR). While ASR performs well in high-resource languages, its accuracy declines in low-resource settings due to limited training data. A common solution is to leverage multilingual self-supervised learning (SSL) models with donor languages. However, existing methods rely on language-level similarity, overlooking clip-level variations. To address this limitation, we propose clip-wise acoustic token distribution similarity (CATDS), a fine-grained selection method that identifies acoustically relevant donor clips for better alignment with the target language. Unlike existing clip-level selection methods, our method aligns with the representation of SSL models and offers more challenging yet valuable samples. Experimental results show that CATDS outperforms traditional selection methods and can even utilize donor languages previously considered detrimental.
Spot and Merge: A Hybrid Context Biasing Approach for Rare Word and Out of Vocabulary Recognition
Authors: Jatin Agrawal; Bramhendra Koilakuntla; Srikanth Konjeti
Venue: Interspeech 2025 (main)

Abstract: Deep context biasing improves rare word recognition in automatic speech recognition (ASR) but often struggles with larger biasing lists. Performance on Out-of-Vocabulary (OOV) words remains limited as the ASR model must learn to generate unseen token sequences. These limitations become more pronounced in contact center applications with prevalent business-specific terminologies. Additionally, using an existing ASR model is challenging, as deep biasing requires full joint training of the ASR model and the biasing module. To address these issues, we introduce 'spot and merge' (SAM), a novel Low-Rank Adapter (LoRA) based system that spots bias words in the cross-attention weights of the biasing module and merges them with ASR output. Unlike existing methods, our approach maintains strong performance even with larger biasing lists, achieving a 1.0% absolute word error rate (WER) reduction on LibriSpeech. It also demonstrates robust OOV recognition on an in-house contact center dataset.
Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR
Authors: Martin Ratajczak; Jean-Philippe Robichaud; Jennifer Drexler Fox
Venue: Interspeech 2025 (main)

Abstract: Long-form speech recognition is an application area of increasing research focus. ASR models based on multi-head attention (MHA) are ill-suited to long-form ASR because of their quadratic complexity in sequence length. We build on recent work that has investigated linear complexity recurrent attention (RA) layers for ASR. We find that bidirectional RA layers can match the accuracy of MHA for both short-and long-form applications. We present a strong limited-context attention (LCA) baseline, and show that RA layers are just as accurate while being more efficient. We develop a long-form training paradigm which further improves RA performance, leading to better accuracy than LCA with 44% higher throughput. We also present Direction Dropout, a novel regularization method that improves accuracy, provides fine-grained control of the accuracy/throughput trade-off of bidirectional RA, and enables a new alternating directions decoding mode with even higher throughput.
Improving Cross-Attention based on Positional Alignment during Inference for Robust Long-form Speech Recognition
Authors: Changhan Oh; Kiyoung Park; Jeomja Kang; Woo Yong Choi; Hwa Jeon Song
Venue: Interspeech 2025 (main)

Abstract: End-to-end (E2E) models have significantly advanced automatic speech recognition (ASR), with hybrid architectures that combine Connectionist Temporal Classification (CTC) and attention-based encoder-decoder (AED) models demonstrating superior performance. However, AED architectures, particularly Conformer, face notable challenges with long-form speech, with performance degradation becoming evident for audio exceeding 25 second. In this study, we propose improving the Conformerâs robustness for long-form ASR by applying Gaussian masking to the cross-attention mechanism of the Transformer decoder during inference, using the aligned positions obtained from the CTC prefix score. The proposed method achieves an error reduction rate (ERR) of 88.41% (from 26.41% to 3.06%) for audio longer than 20 seconds on a LibriSpeech evaluation set constructed by concatenating three utterances. Moreover, the method remains effective with either a Transformer or an E-Branchformer encoder.
Improving End-to-end Mixed-case ASR with Knowledge Distillation and Integration of Voice Activity Cues
Authors: Sashi Novitasari; Takashi Fukuda; Gakuto Kurata
Venue: Interspeech 2025 (main)

Abstract: E2E mixed-case (MC) ASR is a more challenging task than unicase (UC) ASR because of the necessity of capitalizing and punctuating the decoded outputs simultaneously. MC models that are simply trained on formatted transcriptions often suffer from various negative impacts, notably a degradation in case-and-punctuation-insensitive performance due to the increased learning complexity. In this paper, we describe novel techniques for training E2E MC ASR models and use them to improve casing-and-punctuation sensitive and insensitive performance. Our approach incorporates knowledge distillation from UC teacher to MC student models not only to improve capitalization and punctuation accuracy but also to maximize phone classification capability in MC ASR. Furthermore, we attempt to integrate voice activity cues into MC ASR to support text formatting tasks. Our method significantly reduces errors by up to 9.2% relative to baseline models that operate at a similar decoding cost.
Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASR
Authors: Xugang Lu; Peng Shen; Yu Tsao; Hisashi Kawai
Venue: Interspeech 2025 (main)

Abstract: Transferring linguistic knowledge from a pretrained language model (PLM) to acoustic feature learning has proven effective in enhancing end-to-end automatic speech recognition (E2E-ASR). However, aligning representations between linguistic and acoustic modalities remains a challenge due to inherent modality gaps. Optimal transport (OT) has shown promise in mitigating these gaps by minimizing the Wasserstein distance (WD) between linguistic and acoustic feature distributions. However, previous OT-based methods overlook structural relationships, treating feature vectors as unordered sets. To address this, we propose GraphMatching Optimal Transport (GM-OT), which models linguistic and acoustic sequences as structured graphs. Nodes represent feature embeddings, while edges capture temporal and sequential relationships. GM-OT minimizes both WD (between nodes) and Gromov-Wasserstein distance (GWD) (between edges), leading to a fused Gromov Wasserstein distance (FGWD) formulation. This enables structured alignment and more efficient knowledge transfer compared to existing OT-based approaches. Theoretical analysis further shows that prior OT-based method in linguistic knowledge transfer can be viewed as a special case within our GMOT framework. We evaluate GM-OT on Mandarin ASR using a CTC-based E2E-ASR system with a PLM for knowledge transfer. Experimental results demonstrate significant performance gains over state-of-the-art models, validating the effectiveness of our approach.
Age-related changes in multisensory integration of emotions in an audiovisual face-prosody-semantics Stroop task
Authors: Yi Lin; Shumeng Ni; Yangfan Lu
Venue: Interspeech 2025 (main)

Abstract: The present study examined age-related changes in multisensory integration of emotions across facial, prosodic, and semantic channels. Older and younger adults performed an audiovisual Stroop task, in which happy or sad emotions were expressed simultaneously through the three channels. They selectively attended to one of the emotional channels while ignoring congruent or incongruent emotions from other channels. Results indicated that older adults showed an overall decline in emotion integration with greater deficits in the nonverbal channels (particularly prosody) compared to verbal semantics. These channel-specific patterns of age differences were more prominent when information was incongruent across channels compared to the cross-channel congruent condition. To sum up, age differences in multisensory emotion perception are shaped by the complex interactions between channel asymmetry and information congruity, which may be linked to age-related declines in cognitive control.