GenAI4ELab/papercli-papers huggingface dataset:
REAL-T: Real Conversational Mixtures for Target Speaker Extraction
Authors: Shaole Li; Shuai Wang; Jiangyu Han; Ke Zhang; Wupeng Wang; Haizhou Li
Venue: Interspeech 2025 (main)

Abstract: Current target speaker extraction (TSE) systems achieve remarkable performance on synthetic datasets like LibriMix and WSJMix. However, their effectiveness in real conversational scenarios, where the cocktail party problem is most prevalent, remains largely unexplored. In this paper, we conduct a comprehensive analysis of several speaker diarization datasets and introduce REAL-T, the first conversation-centric dataset specifically designed for TSE in real-world conditions. Our evaluations reveal significant performance degradation of existing TSE models on this dataset, highlighting the unaddressed complexity of real-world speech extraction. To facilitate controlled benchmarking, we define two subsets: BASE and PRIMARY, ensuring more manageable yet challenging evaluation settings.
Online Audio-Visual Autoregressive Speaker Extraction
Authors: Zexu Pan; Wupeng Wang; Shengkui Zhao; Chong Zhang; Kun Zhou; Yukun Ma; Bin Ma
Venue: Interspeech 2025 (main)

Abstract: This paper proposes a novel online audio-visual speaker extraction model. In the streaming regime, most studies optimize the audio network only, leaving the visual frontend less explored. We first propose a lightweight visual frontend based on depth-wise separable convolution. Then, we propose a lightweight autoregressive acoustic encoder to serve as the second cue, to actively explore the information in the separated speech signal from past steps. Scenario-wise, for the first time, we study how the algorithm performs when there is a change in focus of attention, i.e., the target speaker. Experimental results on LRS3 datasets show that our visual frontend performs comparably to the previous state-of-the-art on both SkiM and ConvTasNet audio backbones with only 0.1 million network parameters and 2.1 MACs per second of processing. The autoregressive acoustic encoder provides an additional 0.9 dB gain in terms of SI-SNRi, and its momentum is robust against the change in attention.
Plug-and-Play Co-Occurring Face Attention for Robust Audio-Visual Speaker Extraction
Authors: Zexu Pan; Shengkui Zhao; Tingting Wang; Kun Zhou; Yukun Ma; Chong Zhang; Bin Ma
Venue: Interspeech 2025 (main)

Abstract: Audio-visual speaker extraction isolates a target speaker's speech from a mixture speech signal conditioned on a visual cue, typically using the target speaker's face recording. However, in real-world scenarios, other co-occurring faces are often present on-screen, providing valuable speaker activity cues in the scene. In this work, we introduce a plug-and-play inter-speaker attention module to process these flexible numbers of co-occurring faces, allowing for more accurate speaker extraction in complex multi-person environments. We integrate our module into two prominent models: the AV-DPRNN and the state-of-the-art AV-TFGridNet. Extensive experiments on diverse datasets, including the highly overlapped VoxCeleb2 and sparsely overlapped MISP, demonstrate that our approach consistently outperforms baselines. Furthermore, cross-dataset evaluations on LRS2 and LRS3 confirm the robustness and generalizability of our method.
SardinianVoxes: A Speech Recognition Dataset for the Sardinian Languages
Authors: Salvatore Carta; Alessandro Giuliani; Marco Manolo Manca; Mirko Marras; Leonardo Piano
Venue: Interspeech 2025 (main)

Abstract: Low-resource languages inherently lack the extensive data typically required to train effective intelligent systems, leading to a substantial gap in technological support. In this paper, we address this challenge by focusing on the language varieties spoken in Sardinia, which are marked by limited and fragmented linguistic resources. First, we present the design and implementation of a reproducible pipeline that led to the preparation of SardinianVoxes, an audio-text dataset comprising approximately 170 hours of transcribed speech, carefully annotated to reflect the internal linguistic diversity of Sardinian. Second, we introduce a tailored evaluation protocol and benchmarking suite aimed at assessing the performance of state-of-the-art speech-to-text models, and their fine-tuned counterparts, across Sardinian varieties. We expect that our contributions will support the development of intelligent speech technologies for Sardinian.
Prompting Whisper for Improved Verbatim Transcription and End-to-end Miscue Detection
Authors: Griffin Smith; Dianna Yee; Jennifer King Chen; Leah Findlater
Venue: Interspeech 2025 (main)

Abstract: Identifying mistakes (i.e., miscues) made while reading aloud is commonly approached post-hoc by comparing automatic speech recognition (ASR) transcriptions to the target reading text. However, post-hoc methods perform poorly when ASR inaccurately transcribes verbatim speech. To improve on current methods for reading error annotation, we propose a novel end-to-end architecture that incorporates the target reading text via prompting and is trained for both improved verbatim transcription and direct miscue detection. Our contributions include: first, demonstrating that incorporating reading text through prompting benefits verbatim transcription performance over fine-tuning, and second, showing that it is feasible to augment speech recognition tasks for end-to-end miscue detection. We conducted two case studiesâchildrenâs read-aloud and adult atypical speechâand found that our proposed strategies improve verbatim transcription and miscue detection compared to current state-of-the-art.
Automated evaluation of children's speech fluency for low-resource languages
Authors: Bowen Zhang; Nur Afiqah Abdul Latiff; Justin Kan; Rong Tong; Donny Soh; Xiaoxiao Miao; Ian McLoughlin
Venue: Interspeech 2025 (main)

Abstract: Assessment of children's speaking fluency in education is well researched for majority languages, but remains highly challenging for low resource languages. This paper proposes a system to automatically assess fluency by combining a fine-tuned multilingual ASR model, an objective metrics extraction stage, and a generative pre-trained transformer (GPT) network. The objective metrics include phonetic and word error rates, speech speed, and speech-pause duration ratio. These are interpreted by a GPT-based classifier guided by a small set of human-evaluated ground truth examples, to score fluency. We evaluate the proposed system on a dataset of children's speech in two low-resource languages, Tamil and Malay and compare the classification performance against Random Forest and XGBoost, as well as using ChatGPT-4o to predict fluency directly from speech input. Results demonstrate that the proposed approach achieves significantly higher accuracy than multimodal GPT or other methods.
Cantonese Punctuation Restoration using LLM Annotated Data
Authors: King Yiu Suen; Rudolf Chow; Albert Y.S. Lam
Venue: Interspeech 2025 (main)

Abstract: One of the main challenges for punctuation restoration for a low-resource language such as Cantonese is data scarcity. While its spoken and written forms are very different, current Cantonese datasets are mostly from formal written text. Naturally spoken data are very scarce. To address this gap, we leverage LLM to annotate naturally spoken Cantonese transcripts sourced from YouTube. Then, we fine-tune pre-trained language models for punctuation restoration using the LLM-annotated transcripts. Our experiments show that models trained on LLM-annotated transcripts outperform those trained solely on formal written text, despite the smaller dataset size. Our best-performing model achieves performance on par with the strongest LLM evaluated on a benchmark dataset, while being significantly smaller. These findings highlight the potential of LLM-generated data for improving NLP tasks in low-resource languages. Our data and code are publicly available.
Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody
Authors: David Sasu; Benedict Quartey; Kweku Andoh Yamoah; Natalie Schluter
Venue: Interspeech 2025 (main)

Abstract: Enabling robots to accurately interpret and execute spoken language instructions is essential for effective human-robot collaboration. Traditional methods rely on speech recognition to transcribe speech into text, often discarding crucial prosodic cues needed for disambiguating intent. We propose a novel approach that directly leverages speech prosody to infer and resolve instruction intent. Predicted intents are integrated into large language models via in-context learning to disambiguate and select appropriate task plans. Additionally, we present the first ambiguous speech dataset for robotics, designed to advance research in speech disambiguation. Our method achieves 95.79% accuracy in detecting referent intents within an utterance and determines the intended task plan of ambiguous instructions with 71.96% accuracy, demonstrating its potential to significantly improve human-robot communication.
Beyond Traditional Speech Modifications : Utilizing Self Supervised Features for Enhanced Zero-Shot Children ASR
Authors: Abhijit Sinha; Hemant Kumar Kathania; Mikko Kurimo
Venue: Interspeech 2025 (main)

Abstract: Zero-shot automatic speech recognition (ASR) for children is challenging due to pronounced acoustic and linguistic mismatches, speaker variability and limited annotated data. This work utilizes self-supervised learning (SSL) features to address these challenges without requiring child specific data. We perform a layer-wise analysis of SSL models, Wav2Vec2, HuBERT, and Data2Vec to identify optimal representations for zero-shot children ASR. Our results show that features from specific layers (e.g., layer 22 of Wav2Vec2) capture robust, speaker-invariant phonetic information, significantly improving recognition accuracy by reducing the word error rate (WER) from 10.65% to 5.15%, a 51.64% relative improvement over Wav2Vec2 baseline. Additionally, while conventional acoustic modifications (pitch, speaking rate, formant) enhance performance in traditional systems, they yield minimal gains for SSL-based models, highlighting the intrinsic speaker invariance of SSL representations.
Spoken Language Modeling with Duration-Penalized Self-Supervised Units
Authors: Nicol Visser; Herman Kamper
Venue: Interspeech 2025 (main)

Abstract: Spoken language models (SLMs) operate on acoustic units obtained by discretizing self-supervised speech representations. Although the characteristics of these units directly affect performance, the interaction between codebook size and unit coarseness (i.e., duration) remains unexplored. We investigate SLM performance as we vary codebook size and unit coarseness using the simple duration-penalized dynamic programming (DPDP) method. New analyses are performed across different linguistic levels. At the phone and word levels, coarseness provides little benefit, as long as the codebook size is chosen appropriately. However, when producing whole sentences in a resynthesis task, SLMs perform better with coarser units. In lexical and syntactic language modeling tasks, coarser units also give higher accuracies at lower bitrates. We therefore show that coarser units aren't always better, but that DPDP is a simple and efficient way to obtain coarser units for the tasks where they are beneficial.
Towards One-bit ASR: Extremely Low-bit Conformer Quantization Using Co-training and Stochastic Precision
Authors: Zhaoqing Li; Haoning Xu; Zengrui Jin; Lingwei Meng; Tianzi Wang; Huimeng Wang; Youjun Chen; Mingyu Cui; Shujie Hu; Xunying Liu
Venue: Interspeech 2025 (main)

Abstract: Model compression has become an emerging need as the sizes of modern speech systems rapidly increase. In this paper, we study model weight quantization, which directly reduces the memory footprint to accommodate computationally resource-constrained applications. We propose novel approaches to perform extremely low-bit (i.e., 2-bit and 1-bit) quantization of Conformer automatic speech recognition systems using multiple precision model co-training, stochastic precision, and tensor-wise learnable scaling factors to alleviate quantization incurred performance loss. The proposed methods can achieve performance-lossless 2-bit and 1-bit quantization of Conformer ASR systems trained with the 300-hr Switchboard and 960-hr LibriSpeech corpus. Maximum overall performance-lossless compression ratios of 16.2 and 16.6 times are achieved without a statistically significant increase in the word error rate (WER) over the full precision baseline systems, respectively.
Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models
Authors: Zhaoqing Li; Haoning Xu; Xurong Xie; Zengrui Jin; Tianzi Wang; Xunying Liu
Venue: Interspeech 2025 (main)

Abstract: This paper presents a novel memory-efficient model compression approach for Conformer ASR and speech foundation systems. Our approach features a unique "small-to-large" design. A compact "seed" model containing a few Conformer or Transformer blocks is trained and unfolded many times to emulate the performance of larger uncompressed models with different logical depths. The seed model and many unfolded paths are jointly trained within a single unfolding cycle. The KL-divergence between the largest unfolded and smallest seed models is used in a self-distillation process to minimize their performance disparity. Experimental results show that our foldable model produces ASR performance comparable to individually constructed Conformer and wav2vec2/HuBERT speech foundation models under various depth configurations, while requiring only minimal memory and storage. Conformer and wav2vec2 models with a reduction of 35% and 30% parameters are obtained without loss of performance, respectively.
Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates
Authors: Haoning Xu; Zhaoqing Li; Youjun Chen; Huimeng Wang; Guinan Li; Mengzhe Geng; Chengxi Deng; Xunying Liu
Venue: Interspeech 2025 (main)

Abstract: This paper presents a novel approach for speech foundation models compression that tightly integrates model pruning and parameter update into a single stage. Highly compact layer-level tied self-pinching gates each containing only a single learnable threshold are jointly trained with uncompressed models and used in fine-grained neuron level pruning. Experiments conducted on the LibriSpeech-100hr corpus suggest that our approach reduces the number of parameters of wav2vec2.0-base and HuBERT-large models by 65% and 60% respectively, while incurring no statistically significant word error rate (WER) increase on the test-clean dataset. Compared to previously published methods on the same task, our approach not only achieves the lowest WER of 7.05% on the test-clean dataset under a comparable model compression ratio of 4.26x, but also operates with at least 25% less model compression time.
Ultra-Low Bit Post-Training Quantization of Large Speech Models via K-Means Clustering and Mixed Precision Allocation
Authors: Tianteng Gu; Bei Liu; Haoyu Wang; Yanmin Qian
Venue: Interspeech 2025 (main)

Abstract: Large speech foundation models like Whisper face significant deployment challenges due to their massive storage requirements. While post-training quantization (PTQ) offers a practical compression solution, existing methods suffer severe performance degradation below 8 bits, particularly for transformer-based architectures with prevalent weight outliers. We propose an ultra-low bit PTQ framework combining three key innovations: 1) K-means clustering for distribution-aware nonlinear quantization, 2) Mixed-precision allocation based on columnwise outlier density, and 3) Selective retention of critical outliers in sparse FP32 format. Evaluated on Whisper-Large-V3 (1.5B parameters), our method achieves 2.12-bit quantization with only 0.17% absolute WER increase on LibriSpeech test-clean. The approach also maintains whisper's robust capabilities, showing less than 1% WER degradation across multiple dataset.
Context-Driven Dynamic Pruning for Large Speech Foundation Models
Authors: Masao Someki; Shikhar Bharadwaj; Atharva Anand Joshi; Chyi-Jiunn Lin; Jinchuan Tian; Jee-weon Jung; Markus MÃ¼ller; Nathan Susanj; Jing Liu; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: Speech foundation models achieve strong generalization across languages and acoustic conditions, but require significant computational resources for inference. In the context of speech foundation models, pruning techniques have been studied that dynamically optimize model structures based on the target audio leveraging external context. In this work, we extend this line of research and propose context-driven dynamic pruning, a technique that optimizes the model computation depending on the context between different input frames and additional context during inference. We employ the Open Whisper-style Speech Model (OWSM) and incorporate speaker embeddings, acoustic event embeddings, and language information as additional context. By incorporating the speaker embedding, our method achieves a reduction of 56.7 GFLOPs while improving BLEU scores by a relative 25.7% compared to the fully fine-tuned OWSM model.
Analyzing the Importance of Blank for CTC-Based Knowledge Distillation
Authors: Benedikt Hilmes; Nick Rossenbach; Ralf SchlÃ¼ter
Venue: Interspeech 2025 (main)

Abstract: With the rise of large pre-trained foundation models for automatic speech recognition new challenges appear. While the performance of these models is good, runtime and cost of inference increases. One approach to make use of their strength while retaining efficiency is to distill their knowledge to smaller models during training. In this work, we explore different CTC-based distillation variants, focusing on blank token handling. We show that common approaches like blank elimination do not always work off the shelf. We explore new blank selection patterns as a potential sweet spot between standard knowledge distillation and blank elimination mechanisms. Through the introduction of a symmetric selection method, we are able to remove the CTC loss during knowledge distillation with minimal to no performance degradation. With this, we make the training independent from target labels, potentially allowing for distillation on untranscribed audio data.
Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages
Authors: Seraphina Fong; Marco Matassoni; Alessio Brutti
Venue: Interspeech 2025 (main)

Abstract: Large language models (LLMs) have demonstrated potential in handling spoken inputs for high-resource languages, reaching state-of-the-art performance in various tasks. However, their applicability is still less explored in low-resource settings. This work investigates the use of Speech LLMs for low-resource Automatic Speech Recognition using the SLAM-ASR framework, where a trainable lightweight projector connects a speech encoder and a LLM. Firstly, we assess training data volume requirements to match Whisper-only performance, re-emphasizing the challenges of limited data. Secondly, we show that leveraging mono- or multilingual projectors pretrained on high-resource languages reduces the impact of data scarcity, especially with small training sets. Using multilingual LLMs (EuroLLM, Salamandra) with whisper-large-v3-turbo, we evaluate performance on several public benchmarks, providing insights for future research on optimizing Speech LLMs for low-resource languages and multilinguality.
A Chinese Heart Failure Status Speech Database with Universal and Personalised Classification
Authors: Yue Pan; Liwei Liu; Changxin Li; Xingyao Wang; Yili Xia; Hanyue Zhang; Ming Chu
Venue: Interspeech 2025 (main)

Abstract: Speech is a cost-effective and non-intrusive data source for identifying acute and chronic heart failure (HF). However, there is a lack of research on whether Chinese syllables contain HF-related information, as observed in other well-studied languages. This study presents the first Chinese speech database of HF patients, featuring paired recordings taken before and after hospitalisation. The findings confirm the effectiveness of the Chinese language in HF detection using both standard 'patient-wise' and personalised 'pair-wise' classification approaches, with the latter serving as an ideal speaker-decoupled baseline for future research. Statistical tests and classification results highlight individual differences as key contributors to inaccuracy. Additionally, an adaptive frequency filter (AFF) is proposed for frequency importance analysis.
Heart Rate as a Proxy Measure to Assess Human Confidence in Spoken Speech
Authors: Harish Battula; Gauri Deshpande; Yagna Gudipalli; Sachin Patel
Venue: Interspeech 2025 (main)

Abstract: Human confidence reflects a positive self-perception and balanced autonomic nervous system response. In this paper, we present a three stage approach to detect human confidence level by computing the heart rate from speech signals. First stage involves extraction of breathing patterns from speech using a pre-trained model followed by stage 2, where the heart rate is extracted using Independent Component Analysis (ICA) on the breathing patterns. Finally in stage 3, an analysis of heart rate values indicating human confidence levels is done. To the best of our knowledge, this is the first time ever, with our experiments, empirically it is found that heart rate extracted from speech carries information related to the confidence of the candidate. We observe that, on an average, confident speakers have 10 beats per minute lower heart rate as compared to non confident speakers.
Foundation Model Hidden Representations for Heart Rate Estimation from Auscultation
Authors: Jingping Nie; Tien Dung Tran; Karan Thakkar; Vasudha Kowtha; Jon Huang; Carlos Avendano; Erdrin Azemi; Vikramjit Mitra
Venue: Interspeech 2025 (main)

Abstract: Auscultation, particularly heart sound, is a non-invasive technique that provides essential vital sign information. Recently, self-supervised acoustic representation foundation models (FMs) have been proposed to offer insights into acoustics-based vital signs. However, there has been little exploration of the extent to which auscultation is encoded in these pre-trained FM representations. In this work, using a publicly available phonocardiogram (PCG) dataset and a heart rate (HR) estimation model, we conduct a layer-wise investigation of six acoustic representation FMs: HuBERT, wav2vec2, wavLM, Whisper, Contrastive Language-Audio Pretraining (CLAP), and an in-house CLAP model against acoustic features as baseline. Notably, HR estimation using the representations from the audio encoder of the in-house CLAP model outperforms the results obtained from the baseline, achieving a lower mean absolute error (MAE) across various train/validation/test splits despite the domain mismatch.