GenAI4ELab/papercli-papers huggingface dataset:
EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer
Authors: Jiarui Hai; Yong Xu; Hao Zhang; Chenxing Li; Helin Wang; Mounya Elhilali; Dong Yu
Venue: Interspeech 2025 (main)

Abstract: We introduce EzAudio, a text-to-audio (T2A) generation framework designed to produce high-quality, natural-sounding sound effects. Core designs include: (1) We propose EzAudio-DiT, an optimized Diffusion Transformer (DiT) designed for audio latent representations, improving convergence speed, as well as parameter and memory efficiency. (2) We apply a classifier-free guidance (CFG) rescaling technique to mitigate fidelity loss at higher CFG scores and enhancing prompt adherence without compromising audio quality. (3) We propose a synthetic caption generation strategy leveraging recent advances in audio understanding and LLMs to enhance T2A pretraining. We show that EzAudio, with its computationally efficient architecture and fast convergence, is a competitive open-source model that excels in both objective and subjective evaluations by delivering highly realistic listening experiences.
You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks
Authors: Ãnal Ege Gaznepoglu; Anna Leschanowsky; Ahmad Aloradi; Prachi Singh; Daniel Tenbrinck; EmanuÃ«l A. P. Habets; Nils Peters
Venue: Interspeech 2025 (main)

Abstract: Speaker anonymization systems hide the identity of speakers while preserving other information such as linguistic content and emotions. To evaluate their privacy benefits, attacks in the form of automatic speaker verification (ASV) systems are employed. In this study, we assess the impact of intra-speaker linguistic content similarity in the attacker training and evaluation datasets, by adapting BERT, a language model, as an ASV system. On the VoicePrivacy Attacker Challenge datasets, our method achieves a mean equal error rate (EER) of 35%, with certain speakers attaining EERs as low as 2%, based solely on the textual content of their utterances. Our explainability study reveals that the system decisions are linked to semantically similar keywords within utterances, stemming from how LibriSpeech is curated. Our study suggests reworking the VoicePrivacy datasets to ensure a fair and unbiased evaluation and challenge the reliance on global EER for privacy evaluations.
Recognizing Every Voice: Towards Inclusive ASR for Rural Bhojpuri Women
Authors: Sakshi Joshi; Eldho Ittan George; Tahir Javed; Kaushal Bhogale; Nikhil Narasimhan; Mitesh M. Khapra
Venue: Interspeech 2025 (main)

Abstract: Digital inclusion remains a challenge for marginalized communities, especially rural women in low-resource language regions like Bhojpuri. Voice-based access to agricultural services, financial transactions, government schemes, and healthcare is vital for their empowerment, yet existing ASR systems for this group remain largely untested. To address this gap, we create SRUTI, a benchmark consisting of rural Bhojpuri women speakers. Evaluation of current ASR models on SRUTI shows poor performance due to data scarcity, which is difficult to overcome due to social and cultural barriers that hinder large-scale data collection. To overcome this, we propose generating synthetic speech using just 25â30 seconds of audio per speaker from approximately 100 rural women. Augmenting existing datasets with this synthetic data achieves an improvement of 4.7 WER, providing a scalable, minimally intrusive solution to enhance ASR and promote digital inclusion in low-resource language.
Augment Mandarin to Cantonese Speech Databases via Retrieval-Augmented Generation and Speech Synthesis
Authors: Fan Liu; Cheng Gong; Boyu Zhu; Ruihao Jing; Chunyu Qiang; Tianrui Wang; Xiao-Lei Zhang; Xuelong Li
Venue: Interspeech 2025 (main)

Abstract: Using large-scale training data has significantly driven recent advances in speech recognition models. However, the lack of corpus for some low-resource languages (e.g., Cantonese) is still a bottleneck for speech processing. With the continuous development of large language models (LLMs) and speech synthesis technologies, it has become possible to expand Cantonese corpora using automatically generated text and speech. In this paper, we propose using LLMs and text-to-speech (TTS) techniques to augment Mandarin data to the Mandarin-Cantonese parallel speech database (MCPSD). We propose a novel retrieval-augmented generation (RAG) method that incorporates a Cantonese knowledge base to enhance both the diversity and the accuracy of text generation. We implement TTS models to generate speech and design a data filter to ensure quality. Experimental results show that the generated parallel database is effective in fine-tuning models in Cantonese speech recognition and translation.
An Exploratory Framework for LLM-assisted Human Annotation of Speech Datasets
Authors: Alexander Johnson; Harsh Deshpande; Emmy Phung; Ahmad Emami
Venue: Interspeech 2025 (main)

Abstract: We introduce a framework for LLM-based human-in-the-loop ASR designed to enhance the quality of ASR transcripts, with a particular focus on accurately capturing named entities. A key contribution of this work is demonstrating that when LLMs are provided with high-quality, human-annotated transcript examples, even a small set can significantly improve WER and entity recall rates. Our framework drastically reduces the costly need for human annotation to just 5% of the entire call. Our framework outperforms all baselines, including out-of-the-box Whisper and Whisper with a zero-shot GPT corrector. In this work, we derive insights on how a chain-of-thought framework can effectively utilize LLM prompts and human input to improve speech data annotation quality. Using our framework, we achieve a 10% or greater relative improvement in WER and entity F1 score over the baseline with a minimal amount of human effort.
Automatic Labeling and Correction of Noisy Labels for Robust Self-Supervised Speaker Verification
Authors: Abderrahim Fathan; Jahangir Alam
Venue: Interspeech 2025 (main)

Abstract: Supervised speaker verification relies on large labeled datasets, which are costly and labor-intensive to create. However, both manual and clustering-based labeling methods introduce label noise, degrading model generalization. To leverage unlabeled speech data, we propose a framework that automatically generates and refines pseudo speaker labels. It first generates pseudo-labels using a clustering algorithm, then trains a speaker verification system to boost the quality of pseudo-labeled data using self-supervised learning and a neural embedding extractor optimized with refined loss function. This function integrates a dynamic and adaptive label noise cleansing method, termed AdaptiveDropSC, which tracks dominant sub-centers via a dictionary table for better label correction. Experiments on VoxCeleb corpus show that our method improves pseudo-labeling accuracy across different clustering techniques, achieving state-of-the-art performance in self-supervised speaker verification.
Auto-Landmark: Acoustic Landmark Dataset and Open-Source Toolkit for Landmark Extraction
Authors: Xiangyu Zhang; Daijiao Liu; Tianyi Xiao; Cihan Xiao; TÃ¼nde Szalay; Mostafa Shahin; Beena Ahmed; Julien Epps
Venue: Interspeech 2025 (main)

Abstract: In the speech signal, acoustic landmarks identify times when the acoustic manifestations of the linguistically motivated distinctive features are most salient. Acoustic landmarks have been widely applied in various domains, including speech recognition, speech depression detection, clinical analysis of speech abnormalities, and the detection of disordered speech. However, there is currently no dataset available that provides precise timing information for landmarks, which has been proven to be crucial for downstream applications involving landmarks. In this paper, we selected the most useful acoustic landmarks based on previous research and annotated the TIMIT dataset with them, based on a combination of phoneme boundary information and manual inspection. Moreover, previous landmark extraction tools were not open source or benchmarked, so to address this, we developed an open source Python-based landmark extraction tool and established a series of landmark detection baselines. The first of their kinds, the dataset with landmark precise timing information, landmark extraction tool and baselines are designed to support a wide variety of future research
AusKidTalk: Using Strategic Data Collection and Out-of-Domain Tools to Semi-Automate Novel Corpora Annotation
Authors: TÃ¼nde Szalay; Mostafa Shahin; Tharmakulasingam Sirojan; Zheng Nan; Renata Huang; Kirrie Ballard; Beena Ahmed
Venue: Interspeech 2025 (main)

Abstract: Annotating speech corpora for novel populations presents a circular problem: eliminating costly manual transcription requires automatic speech recognition (ASR) tools not yet developed; but developing ASR tools requires annotated speech corpora not available. Manual transcription burden was reduced for AusKidTalk, a novel population due to speaker age and accent, by strategic data collection protocol combined with out-of-domain ASR tools for semi-automatic annotation. The data collection protocol inserted tones and timestamps to automatically segment the recordings. Automatic annotation was conducted by out-of-domain tools for diarisation (NeMo) and orthographic transcription (UNSW ASR). Transcription accuracy with 17% word error rate (WER) for single words and 23% WER for continuous speech allowed for hand-correction instead of transcription, reducing annotation burden. The workflow can be adapted for other corpora and updated with new ASR tools as they become available.
ASR-based segmentation for the analysis of larger child-speech datasets: Performance evaluation on vowels from Australian-English speaking children aged 4 to 11 years
Authors: Rui Cai; Titia Benders
Venue: Interspeech 2025 (main)

Abstract: Annotation of segment boundaries in child speech presents a persistent challenge, particularly with large-scale datasets. While auto-segmentation methods have been developed for adult speech, little attention has been paid to evaluating their performance on child speech. This study evaluates factors contributing to performance of automatic segmentation by analyzing the annotations of two studies into Australian-English-speaking children. The first study assesses human-human reliability in 3- and 12-year-olds. The second compares manual and the Montreal Forced Aligner (MFA) segmentation in 4- to 11-year-olds. The results indicate that the MFA falls short of the human annotator, though discrepancies decrease as children grow older. Systematic discrepancies between the human annotator and the MFA suggest different criteria for placing segment boundaries. These results shed light on how to incorporate adult-based automatic aligners into semi-automatic (and thus: still partially manual) acoustic analysis of child speech.
A semi-automatic pipeline for transcribing and segmenting child speech
Authors: Polychronia Christodoulidou; James Tanner; Jane Stuart-Smith; Michael McAuliffe; Mridhula Murali; Amy Smith; Lauren Taylor; Joanne Cleland; Anja Kuschmann
Venue: Interspeech 2025 (main)

Abstract: This study evaluates both automated transcription (WhisperX) and forced alignment (MFA) in developing a semi-automated pipeline for obtaining  acoustic vowel measures from field recordings from 275 children speaking a non-standard, English dialect, Scottish English. As expected, manual correction of speech transcriptions before forced alignment improves the quality of acoustic vowel measures with respect to manually-annotated data, though speech style and recording environment present some challenges for both tools. Adaptation of the MFA pre-trained english_us_arpa acoustic model towards the children's speech also improves the quality of acoustic measures, though greater improvement was not found by increasing training sample size.
Hybrid Data Sampling for ASR: Integrating Acoustic Diversity and Transcription Uncertainty
Authors: Komei Hiruta; Yosuke Yamano; Hideaki Tamori
Venue: Interspeech 2025 (main)

Abstract: Efficiently selecting training data is crucial for improving automatic speech recognition (ASR) models while minimizing annotation costs. This research extends TypiClust, a data sampling method originally validated for images, to ASR by jointly considering acoustic diversity and transcription uncertainty. Our method clusters speech embeddings from Wav2Vec2 and prioritizes typical and low transcription-confidence samples within each cluster, ensuring selection of representative and hard-to-train samples. We evaluate our method on Japanese speech datasets, CSJ and ReazonSpeech, demonstrating that it achieves lower recognition error rates than random or single-criterion-based data selection. These results indicate that selecting data based on both diversity and uncertainty enhances speech recognition model performance while reducing annotation costs.
Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification
Authors: William Ravenscroft; George Close; Kit Bower-Morris; Jamie Stacey; Dmitry Sityaev; Kris Y. Hong
Venue: Interspeech 2025 (main)

Abstract: Large-scale in-the-wild speech datasets have become more prevalent in recent years due to increased interest in models that can learn useful features from unlabelled data for tasks such as speech recognition or synthesis. These datasets often contain undesirable features, such as multiple speakers, non-target languages, and music, which may impact model learning. The Whilter model is proposed as a multitask solution to identify these undesirable samples. Whilter uses a Whisper encoder with an attention-based classifier to solve five diverse classification problems at once. In addition, an annotated dataset is published for a subset of two popular in-the-wild corpora. Whilter achieves F1 scores above 85% and equal error rates of 6.5% to 7.8% for three of five subtasks, outperforming a state-of-the-art BEATs classifier on speech-specific classes, with a notable decrease in processing time compared to a combination of single-task alternatives.
Adapting Whisper for low-resource Hindi-English Code-Mix speech with on-the-fly Augmentation & LLM-Synthesised Data
Authors: Astik Biswas; Oleg Shevelev; Amine Abdaoui; Vivek Tyagi; Abdelmoumene Boumadane
Venue: Interspeech 2025 (main)

Abstract: Code-switching (CS) automatic speech recognition (ASR) faces challenges due to language confusion from accents, acoustic similarities, and seamless transitions. In multilingual India, CS is prevalent, yet adapting pre-trained Whisper for low-resource Indian CS-ASR remains under-explored. This study explores strategies for adapting Whisper with limited data. First, we propose language prompts for fine-tuning and on-the-fly code-mixed data simulation to handle language switches. Second, we use Llama for few-shot code-switching (CS) text generation, coupled with audio synthesis, to create synthetic data for fine-tuning the Whisper model. Experiments on a Hindi-English CS dataset show promising results, demonstrating the techniques effectiveness. These findings are applicable to other multilingual contexts, aiding Whisperâs adaptation to new domains.
Optimizing ASR for Catalan-Spanish Code-Switching: A Comparative Analysis of Methodologies
Authors: Carlos Mena; Pol Serra; Jacobo Romero; Abir Messaoudi; Jose Giraldo; Carme Armentano-Oller; Rodolfo Zevallos; Ivan Meza; Javier Hernando
Venue: Interspeech 2025 (main)

Abstract: Code-switching (CS), the alternating use of two or more languages, challenges automatic speech recognition (ASR) due to scarce training data and linguistic similarities. The lack of dedicated CS datasets limits ASR performance, as most models rely on monolingual or mixed-language corpora that fail to reflect real-world CS patterns. This issue is critical in multilingual societies where CS occurs in informal and formal settings. A key example is Catalan-Spanish CS, widely used in media and parliamentary speeches. In this work, we improve ASR for Catalan-Spanish CS by exploring three strategies: (1) generating synthetic CS data, (2) concatenating monolingual audio, and (3) leveraging real CS data with language tokens. We extract CS data from Catalan speech corpora and fine-tune OpenAIâs Whisper models, making them available on Hugging Face. Results show that combining a modest amount of synthetic CS data with the dominant language token yields the best transcription performance.
From Scarcity to Sufficiency: Speech Recognition Pipeline for Zero-resource Language
Authors: Nikolay Karpov; Sofia Kostandian; Nune Tadevosyan; Alexan Ayrapetyan; Andrei Andrusenko; Ara Yeroyan; Mher Yerznkanyan; Vitaly Lavrukhin
Venue: Interspeech 2025 (main)

Abstract: The quality of Automatic Speech Recognition (ASR) systems largely depends on the availability of training data, which is predominantly accessible for either high-resource or low-resource languages. In contrast, languages such as Armenian face significant challenges due to the almost zero availability of public speech and text corpora. In this paper, we introduce a comprehensive framework that elevates data availability for a zero-resource language to a new level, thereby enabling the development of a fully operational online ASR model. Our approach involves data collection and processing through diverse resources, including audiobooks, paid crowdsourcing, and leveraging the volunteer platform to assemble a labeled dataset totaling 149 hours. This data made it possible to apply pseudo-labeling techniques on additional 145 hours of public audio data, achieving a new state-of-the-art Word Error Rate (WER) of 9.90% on Common Voice test. All datasets and ASR models are open-sourced.
MIKU-PAL: An Automated and Standardized Multimodal Method for Speech Paralinguistic and Affect Labeling
Authors: Yifan Cheng; Ruoyi Zhang; Jiatong Shi
Venue: Interspeech 2025 (main)

Abstract: Acquiring large-scale emotional speech data with strong consistency remains a challenge for speech synthesis. This paper presents MIKU-PAL, a fully automated multimodal pipeline for extracting high-consistency emotional speech from unlabeled video data. Leveraging face detection and tracking algorithms, we developed an automatic emotion analysis system using a multimodal large language model (MLLM). Our results demonstrate that MIKU-PAL can achieve human-level accuracy (68.5% on MELD) and superior consistency (0.93 Fleiss Îº score) while being much cheaper and faster than human annotation. With the high-quality, flexible, and consistent annotation from MIKU-PAL, we can annotate fine-grained speech emotion categories of up to 26 types, validated by human annotators with 83% rationality ratings. Based on our proposed system, we further released a fine-grained emotional speech dataset MIKU-EmoBench(131.2 hours) as a new benchmark for emotional text-to-speech and visual voice cloning.
Multimodal Fusion with Semi-Supervised Learning Minimizes Annotation Quantity for Modeling Videoconference Conversation Experience
Authors: Andrew Chang; Chenkai Hu; Ji Qi; Zhuojian Wei; Kexin Zhang; Viswadruth Akkaraju; David Poeppel; Dustin Freeman
Venue: Interspeech 2025 (main)

Abstract: Group conversations over videoconferencing are a complex social behavior. However, the subjective moments of negative experience, where the conversation loses fluidity or enjoyment remain understudied. These moments are infrequent in naturalistic data, and thus training a supervised learning (SL) model requires costly manual data annotation. We applied semi-supervised learning (SSL) to leverage targeted labeled and unlabeled clips for training multimodal (audio, facial, text) deep features to predict non-fluid or unenjoyable moments in holdout videoconference sessions. The modality-fused co-training SSL achieved an ROC-AUC of 0.9 and an F1 score of 0.6, outperforming SL models by up to 4% with the same amount of labeled data. Remarkably, the best SSL model with just 8% labeled data matched 96% of the SL model's full-data performance. This shows an annotation-efficient framework for modeling videoconference experience.
Clinical Annotations for Automatic Stuttering Severity Assessment
Authors: Ana Valente; Rufael Marew; Hawau Toyin; Hamdan Al-Ali; Anelise Bohnen; Inma Becerra; Elsa Soares; GonÃ§alo Leal; Hanan Aldarmaki
Venue: Interspeech 2025 (main)

Abstract: Stuttering is a complex disorder that requires specialized expertise for effective assessment and treatment. This paper presents an effort to enhance the FluencyBank dataset with a new stuttering annotation scheme based on established clinical standards. To achieve high-quality annotations, we hired expert clinicians to label the data, ensuring that the resulting annotations mirror real-world clinical expertise. The annotations are multi-modal, incorporating audiovisual features for the detection and classification of stuttering moments, secondary behaviors, and tension scores. In addition to individual annotations, we additionally provide a test set with highly reliable annotations based on expert consensus for assessing individual annotators and machine learning models. Our experiments and analysis illustrate the complexity of this task that necessitates extensive clinical expertise for valid training and evaluation of stuttering assessment models.
Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech
Authors: Wonjune Kang; Junteng Jia; Chunyang Wu; Wei Zhou; Egor Lakomkin; Yashesh Gaur; Leda Sari; Suyoun Kim; Ke Li; Jay Mahadeokar; Ozlem Kalinli
Venue: Interspeech 2025 (main)

Abstract: This work studies the capabilities of a large language model (LLM) to understand paralinguistic aspects of speech without fine-tuning its weights. We utilize an end-to-end system with a speech encoder, which is trained to produce token embeddings such that the LLM's response to an expressive speech prompt is aligned with its response to a semantically matching text prompt that has also been conditioned on the user's speaking style. This framework enables the encoder to generate tokens that capture both linguistic and paralinguistic information and effectively convey them to the LLM, even when the LLM's weights remain completely frozen. To the best of our knowledge, our work is the first to explore how to induce a frozen LLM to understand more than just linguistic content from speech inputs in a general interaction setting. Experiments demonstrate that our system is able to produce higher quality and more empathetic responses to expressive speech prompts compared to several baselines.
AA-SLLM: An Acoustically Augmented Speech Large Language Model for Speech Emotion Recognition
Authors: Jialong Mai; Xiaofen Xing; Weidong Chen; Yuanbo Fang; Xiangmin Xu
Venue: Interspeech 2025 (main)

Abstract: Recently, the rapid advancements in Speech Large Language Models (SpeechLLMs) have greatly accelerated progress in the Speech Emotion Recognition (SER) field. However, SpeechLLMs rely on powerful semantic encoders and acoustically irrelevant pre-training data, granting limited attention to acoustic information, which is closely related to the emotion in speech. In this paper, we leverage acoustic properties correlated with emotions to automatically generate acoustic descriptions. These descriptions are combined with the semantic representations as inputs to the LLM, enhancing emotion recognition capabilities. Accordingly, we propose AA-SLLM, an acoustically augmented SpeechLLM adopting instruction fine-tuning via Low-Rank Adaptation (LoRA). Experimental results indicate that AA-SLLM effectively alleviates the class imbalance problem while improving overall performance. Furthermore, AA-SLLM achieves state-of-the-art results on IEMOCAP, MELD, and LSSED datasets.