GenAI4ELab/papercli-papers huggingface dataset:
Backchannel prediction for natural spoken dialog systems using general speaker and listener information
Authors: Yoshinori Fukunaga; Ryota Nishimura; Kengo Ohta; Norihide Kitaoka
Venue: Interspeech 2025 (main)

Abstract: Backchannel responses are a crucial component of conversations enabling more effective communication through listener feedback. Current backchannel prediction models classify these responses into just three categories, using speech, text, and listener IDs. These IDs, which contain detailed personal information, cannot be applied in real-world dialog systems however, and three-category classification limits response generation capabilities. Therefore, we propose a model for predicting a backchannel's 'surface form' using only general speaker and listener embeddings. Our experiments show a 1.3% improvement in prediction accuracy when performing 3-category classification, and a 0.9% improvement when performing 11-category classification, compared to conventional ID embeddings, demonstrating an enhancement in performance that is deployable in real-world systems.
Rapport-Building Dialogue Strategies for Deeper Connection: Integrating Proactive Behavior, Personalization, and Aizuchi Backchannels
Authors: Muhammad Yeza Baihaqi; Angel GarcÃ­a Contreras; Seiya Kawano; Koichiro Yoshino
Venue: Interspeech 2025 (main)

Abstract: Proactive behavior, dialogue personalization, and backchannels are essential elements for enhancing rapport, fostering engagement, preventing conversation stalls, and supporting a natural dialogue flow. This study integrates these behaviors into a rapport-building dialogue strategy using the CO-STAR and few-shot frameworks to prompt large language models (LLMs) within a human-robot interaction. The success of the integration was assessed through metrics such as conversation stall frequency, dialogue similarity, and backchannel usage by the robot. We then observed the effect of integrating these elements on participant-observed behavioral outcomes, including participant backchannel usage, self-disclosure, and subjective questionnaire responses. Experimental results indicated that incorporating these behaviors into the dialogue strategy positively impacts human-robot rapport both behaviorally and subjectively.
Does effortful speech production indicate communication difficulty caused by noise and hearing aid support?
Authors: Lena-Marie Huttner; Jeppe H. Christensen; Gitte Keidser; Tobias May; Torsten Dau; Sergi Rotger-Griful
Venue: Interspeech 2025 (main)

Abstract: Hearing impairment affects a personâs ability to communicate with others. Communication difficulty remains a little understood concept in research. In this study we assess how parameters of speech affected by noise, hearing aid support, and hearing status reflect a participantâs experience of communication difficulty. We paired 44 participants into dyads consisting of one normal-hearing (NH) and one hearing-impaired (HI) participant. Participants engaged in task-based conversation in quiet as well as in 70dB background noise. HI participants completed the task both with and without hearing aid. After each conversation participants indicated their communication difficulty on a questionnaire. We then analyze how F1, vocal level, and turn taking variability predict communication difficulty. For HI participants, increased vocal level as well as higher variability in turn taking are associated with greater difficulty. For NH participants, increased F1 is associated with greater difficulty.
``Dyadosyncrasy'', Idiosyncrasy and Demographic Factors in Turn-Taking
Authors: Julio Cesar Cavalcanti; Gabriel Skantze
Venue: Interspeech 2025 (main)

Abstract: Turn-taking in dialogue follows universal constraints but also varies significantly. This study examines how demographic (sex, age, education) and individual factors shape turn-taking using a large dataset of US English conversations (Fisher). We analyze Transition Floor Offset (TFO) and find notable interspeaker variation. Sex and age have small but significant effects â female speakers and older individuals exhibit slightly shorter offsets â while education shows no effect. Lighter topics correlate with shorter TFOs. However, individual differences have a greater impact, driven by a strong idiosyncratic and an even stronger "dyadosyncratic" component â speakers in a dyad resemble each other more than they resemble themselves in different dyads. This suggests that the dyadic relationship and joint activity are the strongest determinants of TFO, outweighing demographic influences.
SSPS: Self-Supervised Positive Sampling for Robust Self-Supervised Speaker Verification
Authors: Theo Lepage; Reda Dehak
Venue: Interspeech 2025 (main)

Abstract: Self-Supervised Learning (SSL) has led to considerable progress in Speaker Verification (SV). The standard framework uses same-utterance positive sampling and data-augmentation to generate anchor-positive pairs of the same speaker. This is a major limitation, as this strategy primarily encodes channel information from the recording condition, shared by the anchor and positive. We propose a new positive sampling technique to address this bottleneck: Self-Supervised Positive Sampling (SSPS). For a given anchor, SSPS aims to find an appropriate positive, i.e., of the same speaker identity but a different recording condition, in the latent space using clustering assignments and a memory queue of positive embeddings. SSPS improves SV performance for both SimCLR and DINO, reaching 2.57% and 2.53% EER, outperforming SOTA SSL methods on VoxCeleb1-O. In particular, SimCLR-SSPS achieves a 58% EER reduction by lowering intra-speaker variance, providing comparable performance to DINO-SSPS.
ParaNoise-SV: Integrated Approach for Noise-Robust Speaker Verification with Parallel Joint Learning of Speech Enhancement and Noise Extraction
Authors: Minu Kim; Kangwook Jang; Hoirin Kim
Venue: Interspeech 2025 (main)

Abstract: Noise-robust speaker verification leverages joint learning of speech enhancement (SE) and speaker verification (SV) to improve robustness. However, prevailing approaches rely on implicit noise suppression, which struggles to separate noise from speaker characteristics as they do not explicitly distinguish noise from speech during training. Although integrating SE and SV helps, it remains limited in handling noise effectively. Meanwhile, recent SE studies suggest that explicitly modeling noise, rather than merely suppressing it, enhances noise resilience. Reflecting this, we propose ParaNoise-SV, with dual U-Nets combining a noise extraction (NE) network and a speech enhancement (SE) network. The NE U-Net explicitly models noise, while the SE U-Net refines speech with guidance from NE through parallel connections, preserving speaker-relevant features. Experimental results show that ParaNoise-SV achieves a relatively 8.4% lower equal error rate (EER) than previous joint SE-SV models.
Disentangling Speaker and Content in Pre-trained Speech Models with Latent Diffusion for Robust Speaker Verification
Authors: Zhe Li; Man-Wai Mak; Jen-Tzung Chien; Mert Pilanci; Zezhong Jin; Helen Meng
Venue: Interspeech 2025 (main)

Abstract: Disentangled speech representation learning for speaker verification aims to separate spoken content and speaker timbre into distinct representations. However, existing variational autoencoder (VAE)--based methods for speech disentanglement rely on latent variables that lack semantic meaning, limiting their effectiveness for speaker verification. To address this limitation, we propose a diffusion-based method that disentangles and separates speaker features and speech content in the latent space. Building upon the VAE framework, we employ a speaker encoder to learn latent variables representing speaker features while using frame-specific latent variables to capture content. Unlike previous sequential VAE approaches, our method utilizes a conditional diffusion model in the latent space to derive speaker-aware representations. Experiments on the VoxCeleb datasets demonstrate that our method effectively isolates speaker features from speech content using pre-trained speech
Evaluating Deep Speaker Embedding Robustness to Domain, Sampling Rate, and Codec Variations
Authors: Alexandre Ferro Filho; Diogo Fernandes Costa Silva; Pedro Elias Engelberg Silva Borges; Arlindo Rodrigues GalvÃ£o Filho
Venue: Interspeech 2025 (main)

Abstract: Speaker verification systems based on deep speaker embeddings perform well under matched training and evaluation conditions, but performance degrades under domain shifts. This work evaluates the sensitivity of four speaker embedding modelsâECAPA-TDNN, TitaNet, ECAPA2, and ReDimNetâto variations in acoustic domains, sampling rates, and audio codecs. Experiments on datasets with far-field speech, noise, and music interference demonstrate that all models degrade under mismatched conditions. ReDimNet demonstrates the smallest performance degradation compared to the other models but is still affected in certain cases. Downsampling and low-bitrate compression further degrade performance, revealing the reliance of models on high-frequency information and their sensitivity to compression artifacts.
Towards Robust Speaker Recognition against Intrinsic Variation with Foundation Model Few-shot Tuning and Effective Speech Synthesis
Authors: Zhiyong Chen; Shuhang Wu; Xinnuo Li; Zhiqi Ai; Shugong Xu
Venue: Interspeech 2025 (main)

Abstract: Speaker recognition is essential for secure authentication and personalized voice assistants in smart home settings, but it faces challenges due to intrinsic speaker variability, such as aging and emotional fluctuations. Existing methods often rely on pretraining and require extensive data. To address these challenges, we propose a framework for time-varying and emotion-robust open-set identification (OSI) for smart home environments, utilizing few-shot foundation enrollment-time tuning and style-rich zero-shot text-to-speech (TTS) systems. We explore best practices for synthetic data selection and suitable open-set outlier-focused loss functions. Our proposed method improves handling emotional and aging variations in target speakers, enhancing robustness to intrinsic variability while maintaining resilience to unknown outliers. Experiments demonstrate strong generalization across multiple time-varying and emotionally rich benchmarks.
Bayesian Learning for Domain-Invariant Speaker Verification and Anti-Spoofing
Authors: Jin Li; Man-Wai Mak; Johan Rohdin; Kong Aik Lee; Hynek Hermansky
Venue: Interspeech 2025 (main)

Abstract: The performance of automatic speaker verification (ASV) and anti-spoofing drops seriously under real-world domain mismatch conditions. The relaxed instance frequency-wise normalization (RFN), which normalizes the frequency components based on the feature statistics along the time and channel axes, is a promising approach to reducing the domain dependence in the feature maps of a speaker embedding network. We advocate that the different frequencies should receive different weights and that the weights' uncertainty due to domain shift should be accounted for. To these ends, we propose leveraging variational inference to model the posterior distribution of the weights, which results in Bayesian weighted RFN (BWRFN). This approach overcomes the limitations of fixed-weight RFN, making it more effective under domain mismatch conditions. Extensive experiments on cross-dataset ASV, cross-TTS anti-spoofing, and spoofing-robust ASV show that BWRFN is significantly better than WRFN and RFN.
Switch Conformer with Universal Phonetic Experts for Multilingual ASR
Authors: Masato Mimura; Jaeyoung Lee; Tatsuya Kawahara
Venue: Interspeech 2025 (main)

Abstract: Multilingual end-to-end ASR presents significant challenges due to the need to accommodate diverse writing systems, lexicons, and grammatical structures. Existing methods often rely on large models with high computational costs for adequate cross-language performance. To address this, we propose the switch Conformer, which enhances model capacity while maintaining nearly the same inference cost as a standard Conformer. Our approach replaces the FFN module in each Conformer block with a sparse mixture of independent experts, activating only one expert per input to enable efficient language-specific feature learning. In addition, a shared expert trained with phonetic supervision captures language-universal speech characteristics. Experiments on streaming ASR using the CommonVoice dataset demonstrate that these experts work synergistically to achieve better performance than the baseline Conformer, with minimal additional active parameters.
Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR
Authors: Hongli Yang; Sheng Li; Hao Huang; Ayiduosi Tuohan; Yizhou Peng
Venue: Interspeech 2025 (main)

Abstract: Recent advancements in multilingual automatic speech recognition (ASR) have been driven by large-scale end-to-end models like Whisper. However, challenges such as language interference and expanding to unseen languages (language expansion) without degrading performance persist. This paper addresses these with three contributions: 1) Entire Soft Prompt Tuning (Entire SPT), which applies soft prompts to both the encoder and decoder, enhancing feature extraction and decoding; 2) Language-Aware Prompt Tuning (LAPT), using cross-lingual similarities to encode shared and language-specific features in lightweight prompt matrices; 3) SPT-Whisper, a toolkit integrating SPT into Whisper, enabling efficient continual learning. Experiments across three languages from FLEURS demonstrate that Entire SPT and LAPT outperform Decoder SPT by 5.0% and 16.0% in language expansion tasks, respectively, providing an efficient solution for dynamic, multilingual ASR models with minimal computational overhead.
Efficient Multilingual ASR Finetuning via LoRA Language Experts
Authors: Jiahong Li; Yiwen Shao; Jianheng Zhuo; Chenda Li; Liliang Tang; Dong Yu; Yanmin Qian
Venue: Interspeech 2025 (main)

Abstract: Recent advancements in deep learning have significantly enhanced multilingual automatic speech recognition (ASR) due to the development of advanced model architectures and available large-scale multilingual datasets. Despite that, multilingual ASR still suffers from the curse of multilinguality in that different languages tend to interfere with each other, making it difficult for the ASR model to identify multiple languages effectively while sharing model capacity across them. This paper proposes an efficient finetuning framework for customized multilingual ASR via prepared LoRA language experts based on Whisper. Through LoRA expert fusion or knowledge distillation, our approach achieves better recognition performance on target languages than standard fine-tuning methods. Experimental results demonstrate that the proposed models yield approximately 10% and 15% relative performance gains in language-aware and language-agnostic scenarios, respectively.
Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognition
Authors: RaphaÃ«l Bagat; Irina Illina; Emmanuel Vincent
Venue: Interspeech 2025 (main)

Abstract: We aim to improve the robustness of Automatic Speech Recognition (ASR) systems against non-native speech, particularly in low-resourced multi-accent settings. We introduce Mixture of Accent-Specific LoRAs (MAS-LoRA), a fine-tuning method that leverages a mixture of Low-Rank Adaptation (LoRA) experts, each specialized in a specific accent. This method can be used when the accent is known or unknown at inference time, without the need to fine-tune the model again. Our experiments, conducted using Whisper on the L2-ARCTIC corpus, demonstrate significant improvements in Word Error Rate compared to regular LoRA and full fine-tuning when the accent is unknown. When the accent is known, the results further improve. Furthermore, MAS-LoRA shows less catastrophic forgetting than the other fine-tuning methods. To the best of our knowledge, this is the first use of a mixture of LoRA experts for non-native multi-accent ASR.
Effects of Speaker Count, Duration, and Accent Diversity on Zero-Shot Accent Robustness in Low-Resource ASR
Authors: Zheng Xin Yong; Vineel Pratap; Michael Auli; Jean Maillard
Venue: Interspeech 2025 (main)

Abstract: To build an automatic speech recognition (ASR) system that can serve everyone in the world, the ASR needs to be robust to a wide range of accents including unseen accents. We systematically study how three different variables in training data - the number of speakers, the audio duration per each individual speaker, and the diversity of accents - affect ASR robustness towards unseen accents in a low-resource training regime. We observe that for a fixed number of ASR training hours, it is more beneficial to increase the number of speakers (which means each speaker contributes less) than the number of hours contributed per speaker. We also observe that more speakers enables ASR performance gains from scaling number of hours. Surprisingly, we observe minimal benefits to prioritizing speakers with different accents when the number of speakers is controlled. Our work suggests that practitioners should prioritize increasing the speaker count in ASR training data composition for new languages.
Leveraging Geographic Metadata for Dialect-Aware Speech Recognition
Authors: Pouya Mehralian; Hugo Van hamme
Venue: Interspeech 2025 (main)

Abstract: Automatic Speech Recognition (ASR) systems struggle with dialectal variations due to significant phonetic, morphosyntactic, and lexical differences across regions. Traditional approaches rely on dialect-specific models, per-dialect fine-tuning, or unified models that struggle to generalize across diverse dialects. In this work, we propose novel methods for integrating geographical metadata into ASR models to enhance dialect adaptation. By leveraging latitude and longitude as continuous inputs, our model learns smooth transitions across dialectal variations, allowing it to interpolate between known dialects and generalize to unseen regions. Our experiments demonstrate that these approaches significantly outperform both dialect-specific models and state-of-the-art Whisper models while reducing computational and maintenance costs. These findings suggest that location-based adaptation is a promising direction for improving ASR performance in dialect-rich environments.
Overcoming Data Scarcity in Multi-Dialectal Arabic ASR via Whisper Fine-Tuning
Authors: Ãmer Tarik Ãzyilmaz; Matt Coler; Matias Valdenegro-Toro
Venue: Interspeech 2025 (main)

Abstract: Although commercial Arabic automatic speech recognition (ASR) systems support Modern Standard Arabic (MSA), they struggle with dialectal speech. We investigate the effect of fine-tuning OpenAI's Whisper on five major Arabic dialects (Gulf, Levantine, Iraqi, Egyptian, Maghrebi) using Mozilla Common Voice for MSA and the MASC dataset for dialectal speech. We evaluate MSA training size effects, benefits of pre-training on MSA data, and dialect-specific versus dialect-pooled models. We find that small amounts of MSA fine-tuning data yield substantial improvements for smaller models, matching larger non-fine-tuned models. While MSA pre-training shows minimal benefit, suggesting limited shared features between MSA and dialects, our dialect-pooled models perform comparably to dialect-specific ones. This indicates that pooling dialectal data, when properly balanced, can help address data scarcity in low-resource ASR without significant performance loss.
VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining
Authors: Jianheng Zhuo; Yifan Yang; Yiwen Shao; Yong Xu; Dong Yu; Kai Yu; Xie Chen
Venue: Interspeech 2025 (main)

Abstract: Automatic speech recognition (ASR) has made remarkable progress but heavily relies on large-scale labeled data, which is scarce for low-resource languages like Vietnamese. While existing systems such as Whisper, USM, and MMS achieve promising performance, their efficacy remains inadequate in terms of training costs, latency, and accessibility. To address these issues, we propose VietASR, a novel ASR training pipeline that leverages vast amounts of unlabeled data and a small set of labeled data. Through multi-iteration ASR-biased self-supervised learning on a large-scale unlabeled dataset, VietASR offers a cost-effective and practical solution for enhancing ASR performance. Experiments demonstrate that pre-training on 70,000-hour unlabeled data and fine-tuning on merely 50-hour labeled data yield a lightweight but powerful ASR model. It outperforms Whisper Large-v3 and commercial ASR systems on real-world data. Our code and models will be open-sourced to facilitate research in low-resource ASR.
Open Universal Arabic ASR Leaderboard
Authors: Yingzhi Wang; Anas Alhmoud; Muhammad Alqurishi
Venue: Interspeech 2025 (main)

Abstract: In recent years, the enhanced capabilities of ASR models and the emergence of multi-dialect datasets have increasingly pushed Arabic ASR model development toward an all-dialect-in-one direction. This trend highlights the need for benchmarking studies that evaluate model performance in a multi-dialect setting, providing the community with insights into models' generalization capabilities.In this paper, we introduce Open Universal Arabic ASR Leaderboard, a continuous benchmark project for open-source universal Arabic ASR models across various multi-dialect datasets. We also provide a comprehensive analysis of the model's robustness, speaker adaptation, inference efficiency, and memory consumption. This work aims to offer the Arabic ASR community a reference for models' general performance and also establish a common evaluation framework for multi-dialectal Arabic ASR models.
Mel-McNet: A Mel-Scale Framework for Online Multichannel Speech Enhancement
Authors: Yujie Yang; Bing Yang; Xiaofei Li
Venue: Interspeech 2025 (main)

Abstract: Online multichannel speech enhancement has been intensively studied recently. Though Mel-scale frequency is more matched with human auditory perception and computationally efficient than linear frequency, few works are implemented in a Mel-frequency domain. To this end, this work proposes a Mel-scale framework (namely Mel-McNet). It processes spectral and spatial information with two key components: an effective STFT-to-Mel module compressing multi-channel STFT features into Mel-frequency representations, and a modified McNet backbone directly operating in the Mel domain to generate enhanced LogMel spectra. The spectra can be directly fed to vocoders for waveform reconstruction or ASR systems for transcription. Experiments on CHiME-3 show that Mel-McNet can reduce computational complexity by 60% while maintaining comparable enhancement and ASR performance to the original McNet. Mel-McNet also outperforms other SOTA methods, verifying the potential of Mel-scale speech enhancement.
A Lightweight Hybrid Dual Channel Speech Enhancement System under Low-SNR Conditions
Authors: Zheng Wang; Xiaobin Rong; Yu Sun; Tianchi Sun; Zhibin Lin; Jing Lu
Venue: Interspeech 2025 (main)

Abstract: Although deep learning based multi-channel speech enhancement has achieved significant advancements, its practical deployment is often limited by constrained computational resources, particularly in low signal-to-noise ratio (SNR) conditions. In this paper, we propose a lightweight hybrid dual-channel speech enhancement system that combines independent vector analysis (IVA) with a modified version of the dual-channel grouped temporal convolutional recurrent network (GTCRN). IVA functions as a coarse estimator, providing auxiliary information for both speech and noise, while the modified GTCRN further refines the speech quality. We investigate several modifications to ensure the comprehensive utilization of both original and auxiliary information. Experimental results demonstrate the effectiveness of the proposed system, achieving enhanced speech with minimal parameters and low computational complexity.