GenAI4ELab/papercli-papers huggingface dataset:
DiffStereo: End-to-End Mono-to-Stereo Audio Generation with Diffusion Transformer
Authors: Suqi Zhang; Zheqi Dai; Yongyi Zang; Yin Cao; Qiuqiang Kong
Venue: Interspeech 2025 (main)

Abstract: Mono-to-stereo audio generation is a task that converts mono audio into two-channel stereo audio. Stereo audio generation plays a crucial role in enhancing spatial perception and auditory immersion. Traditional mono-to-stereo methods include rule-based, simulation-based, and various deep learning-based approaches. These methods require expert knowledge or explicit positional information to generate specific stereo effects, limiting their scalability and generalization. To address these challenges, we propose DiffStereo, an end-to-end diffusion transformer-based model that generates stereo audio conditioned on mono audio. The contributions of DiffStereo are as follows: First, DiffStereo directly synthesizes stereo audio from a mono waveform input in an end-to-end fashion, requiring no human intervention or prior knowledge. Second, DiffStereo achieves competitive objective ratings and consistently better subjective ratings, validating the effectiveness of our end-to-end approach.
RELATE: Subjective evaluation dataset for automatic evaluation of relevance between text and audio
Authors: Yusuke Kanamori; Yuki Okamoto; Taisei Takano; Shinnosuke Takamichi; Yuki Saito; Hiroshi Saruwatari
Venue: Interspeech 2025 (main)

Abstract: In text-to-audio (TTA) research, the relevance between input text and output audio is an important evaluation aspect. Traditionally, it has been evaluated from both subjective and objective perspectives. However, subjective evaluation is costly in terms of money and time, and objective evaluation is unclear regarding the correlation to subjective evaluation scores. In this study, we construct RELATE, an open-sourced dataset that subjectively evaluates the relevance. Also, we benchmark a model for automatically predicting the subjective evaluation score from synthesized audio. Our model outperforms a conventional CLAPScore model, and that trend extends to many sound categories.
Crowdsourcing MUSHRA Tests in the Age of Generative Speech Technologies: A Comparative Analysis of Subjective and Objective Testing Methods
Authors: Laura Lechler; Chamran Moradi; Ivana Balic
Venue: Interspeech 2025 (main)

Abstract: The MUSHRA framework is widely used for detecting subtle audio quality differences but traditionally relies on expert listeners in controlled environments, making it costly and impractical for model development. As a result, objective metrics are often used during development, with expert evaluations conducted later. While effective for traditional DSP codecs, these metrics often fail to reliably evaluate generative models. This paper proposes adaptations for conducting MUSHRA tests with non-expert, crowdsourced listeners, focusing on generative speech codecs. We validate our approach by comparing results from MTurk and Prolific crowdsourcing platforms with expert listener data, assessing test-retest reliability and alignment. Additionally, we evaluate six objective metrics, showing that traditional metrics undervalue generative models. Our findings reveal platform-specific biases and emphasize codec-aware metrics, offering guidance for scalable perceptual testing of speech codecs.
SMARTMOS: Modeling Subjective Audio Quality Evaluation for Real-Time Applications
Authors: Sivakumar Balasubramanian; Jose Antonio Jimenez Amador; Kaustubh Kalgaonkar; King Wei Hor; Sriram Srinivasan
Venue: Interspeech 2025 (main)

Abstract: Evaluating audio quality is a crucial task, with subjective listening tests being the gold standard. However, these tests are time-consuming intrusive and expensive, making them impractical for real-time applications like telecommunications. Despite efforts to develop automatic methods that match human listener fidelity, current approaches have limitations that hinder their use in real-world scenarios. In this paper, we introduce SMARTMOS, a novel approach that addresses some of these gaps and enables accurate, real-time, non-intrusive, privacy aware audio quality assessment. We demonstrate the effectiveness of SMARTMOS through a case study on Noise Suppression (NS) and Packet Loss Concealment (PLC) modules. Furthermore, we show how semi-supervised learning techniques can be leveraged to build a joint MOS model that seamlessly covers both PLC and NS scenarios.
Effect of Loudspeaker Emitted Speech on ASR performance
Authors: Vikram C M; Sanjoy Pal; Nidhi Mantri; Gopal Kumar Agrawal
Venue: Interspeech 2025 (main)

Abstract: Speech signal played out from the loudspeaker is referred as loudspeaker emitted speech or loud speaker speech. Most of the automatic speech recognition (ASR) systems are trained on the natural speech signals, recorded directly from the human speakers and gives higher word error rate (WER) for the loudspeaker speech. In this paper, first, we analyzed the whisper-medium ASR performance on the loudspeaker emitted speech. Five different equalizer modes, i.e., normal, pop, rock, jazz, and classic along with the distances 0m, 3m, and 5m are considered for the study. Further, based on the spectral differences between natural and loudspeaker speech, an algorithm is proposed to generate the loudspeaker quality speech from natural speech recordings. This algorithm is used to augment the Librispeech data and used to fine-tune the whisper-medium. The fine-tuned ASR on simulated loudspeaker quality speech showed significant improvement when compared to baseline system.
Contextualized Automatic Speech Recognition with Dynamic Vocabulary Prediction and Activation
Authors: Zhennan Lin; Kaixun Huang; Wei Ren; Linju Yang; Lei Xie
Venue: Interspeech 2025 (main)

Abstract: Deep biasing improves automatic speech recognition (ASR) performance by incorporating contextual phrases. However, most existing methods enhance subwords in a contextual phrase as independent units, potentially compromising contextual phrase integrity, leading to accuracy reduction. In this paper, we propose an encoder-based phrase-level contextualized ASR method that leverages dynamic vocabulary prediction and activation. We introduce architectural optimizations and integrate a bias loss to extend phrase-level predictions based on frame-level outputs. We also introduce a confidence-activated decoding method that ensures the complete output of contextual phrases while suppressing incorrect bias. Experiments on Librispeech and Wenetspeech datasets demonstrate that our approach achieves relative WER reductions of 28.31% and 23.49% compared to baseline, with the WER on contextual phrases decreasing relatively by 72.04% and 75.69%.
Character Error Rate Estimation for Semi-Supervised Training of Speech Recognition for Arabic Dialects
Authors: Chanho Park; Oscar Saz
Venue: Interspeech 2025 (main)

Abstract: The use of semi-supervised data for Automatic Speech Recognition (ASR) is nowadays commonplace and is the basis of the most advanced ASR models. For low-resourced languages, where limited labelled data is available, it opens the possibility of using unlimited amounts of data without escalating costs. For this, an initial ASR system is required that can produce a pseudo-transcript of the untranscribed data, but in low-resourced languages, the accuracy of this initial system might not be sufficient to provide accurate pseudo-transcripts, so techniques for data selection become necessary. This paper explores the use of Character Error Rate (CER) estimation for automatically selecting the best segments from a set of nearly 4,000 hours of untranscribed Arabic in different dialects.
Unified Semi-Supervised Pipeline for Automatic Speech Recognition
Authors: Nune Tadevosyan; Nikolay Karpov; Andrei Andrusenko; Vitaly Lavrukhin; Ante Jukic
Venue: Interspeech 2025 (main)

Abstract: Automatic Speech Recognition has been a longstanding research area, with substantial efforts dedicated to integrating semi-supervised learning due to the scarcity of labeled datasets. However, most prior work has focused on improving learning algorithms using existing datasets, without providing a complete public framework for large-scale semi-supervised training across new datasets or languages. In this work, we introduce a fully open-source semi-supervised training framework encompassing the entire pipeline: from unlabeled data collection to pseudo-labeling and model training. Our approach enables scalable dataset creation for any language using publicly available speech data under Creative Commons licenses. We also propose a novel pseudo-labeling algorithm, TopIPL, and evaluate it in both low-resource (Portuguese, Armenian) and high-resource (Spanish) settings. Notably, TopIPL achieves relative WER improvements of 18-40% for Portuguese, 5-16% for Armenian, and 2-8% for Spanish.
Scaling Laws for Synthetic Speech for Model Training
Authors: Christoph Minixhofer; OndÅej Klejch; Peter Bell
Venue: Interspeech 2025 (main)

Abstract: We investigate how the scale of Text-to-Speech (TTS) models' training data influences Automatic Speech Recognition (ASR) performance when real training data is replaced entirely by synthetic speech. We propose an extension to established data scaling laws that incorporates an additional term capturing the mismatch between real and synthetic distributions in low-data regimes. We compare Mean Squared Error (MSE) and Denoising Diffusion Probabilistic Models (DDPMs) for TTS: MSE-based speech, though oversmoothed, provides stronger ASR results with smaller TTS datasets, while DDPM-based speech surpasses MSE once trained on enough data to better approximate the real distribution. Our findings also show that synthetic speech can only approximate or match real data performance if the TTS model itself is trained on a sufficiently large corpus, emphasizing that distribution coverage is crucial for fully synthetic ASR training.
R2S: Real-to-Synthetic Representation Learning for Training Speech Recognition Models on Synthetic Data
Authors: Minh Tran; Debjyoti Paul; Yutong Pang; Laxmi Pandey; Jinxi Guo; Ke Li; Shun Zhang; Xuedong Zhang; Xin Lei
Venue: Interspeech 2025 (main)

Abstract: We investigate the use of synthetic speech to enhance the performance of Automatic Speech Recognition (ASR) systems. While pre-trained ASR models have demonstrated impressive capabilities, their performance can still vary across different conditions and speakers. Conversely, text-to-speech technology allows for precise control over factors such as environmental noise and speaker accents, producing clean speech that poses fewer challenges for ASR systems. Building on this insight, we propose a novel method called R2S (Real-to-Synthetic), which aligns the representation spaces of real and synthetic speech. Our approach incorporates a Gradient Reversal Layer to promote invariant representations between real and synthetic speech, and a Residual-Vector Quantization module to generate pseudo-labels from synthetic speech, guiding the representations of real speech. Our experimental results on three datasets demonstrate that the proposed method can boost ASR performance by 4-5% and successfully align the representation space of real and synthetic speech. Our qualitative results further demonstrate that R2S can suppress speaker-dependent features thanks to the alignment with synthetic speech.
Context is all you need? Low-resource conversational ASR profits from context, coming from the same or from the other speaker
Authors: Julian Linke; Jana Winkler; Barbara Schuppler
Venue: Interspeech 2025 (main)

Abstract: Despite the rapid advancement of automatic speech recognition (ASR) systems, spontaneous conversations still pose a major challenge, which is even more of an obstacle for low-resourced languages, dialects or non-dominant varieties. What is more, lively turn-changes in conversational speech cause short utterances that have been found to be error prone for transformer-based ASR systems, requiring larger context. The question thus arises which type of context is useful: rather more from the same speaker, providing acoustically relevant context, or more from the conversation - mixing utterances from both speakers - providing semantically relevant context. Comparing seven ASR systems on conversational Austrian German, we find the best performance with a minimum of 20s of context, independent of whether it was from the same or from the other speaker. Systems fine-tuned with data from the same variety and speaking style require less context and perform overall better than zero-shot systems.
Automatic Speech Recognition Biases in Newcastle English: an Error Analysis
Authors: Dana Serditova; Kevin Tang; Jochen Steffens
Venue: Interspeech 2025 (main)

Abstract: Automatic Speech Recognition (ASR) systems struggle with regional dialects due to biased training which favours mainstream varieties. While previous research has identified racial, age, and gender biases in ASR, regional bias remains underexamined. This study investigates ASR performance on Newcastle English from 160 speakers, a well-documented regional dialect known to be challenging for ASR. A two-stage analysis was conducted: first, a manual error analysis on a subsample identified key phonological, lexical, and morphosyntactic errors behind ASR misrecognitions; second, a case study focused on the systematic analysis of ASR recognition of the regional pronouns âyousâ and âworâ. Results show that ASR errors directly correlate with regional dialectal features, while social factors play a lesser role in ASR mismatches. We advocate for greater dialectal diversity in ASR training data and highlight the value of sociolinguistic analysis in diagnosing and addressing regional biases.
Speech Unlearning
Authors: Jiali Cheng; Hadi Amiri
Venue: Interspeech 2025 (main)

Abstract: We introduce a challenging and under-explored task -- Machine Unlearning for Speech Tasks. Machine unlearning aims to remove influence of a subset of data from an already trained model. It has practical applications of removing private or sensitive information, and updating outdated knowledge. We propose two unlearning tasks: sample unlearning and class unlearning, and discuss their challenges. In addition, we propose to use subset performances and membership inference attack as evaluation metrics. Experiments on two different speech tasks -- keyword spotting and speaker identification -- demonstrate that unlearning on speech data is more challenging than unlearning on image and text data, which receives more attention. We discuss future steps to advance machine unlearning on speech data, including curriculum learning.
Unlearning LLM-Based Speech Recognition Models
Authors: Zhe Liu
Venue: Interspeech 2025 (main)

Abstract: Recent advances in Large Language Models (LLMs) have been extended to the field of Automatic Speech Recognition (ASR), leveraging the exceptional generative capabilities of LLMs for speech transcription generation. However, concerns about privacy risks have emerged since both LLMs and large ASR models may unintentionally memorize training examples. After deploying a model, practitioners might be asked to remove any personal data from the model. Re-training the original model every time of these requests is computationally expensive. In this work, we provide the first investigation into the measurement and detection of unintended memorization in LLM-based ASR models and present an efficient solution for unlearning memorized data upon requests. Experiments on the LibriSpeech dataset show that the proposed method is effective and achieves strong privacy-utility trade-offs.
EASY: Emotion-aware Speaker Anonymization via Factorized Distillation
Authors: Jixun Yao; Hexin Liu; Eng Siong Chng; Lei Xie
Venue: Interspeech 2025 (main)

Abstract: Emotion plays a significant role in speech interaction, conveyed through tone, pitch, and rhythm, enabling the expression of feelings and intentions beyond words to create a more personalized experience. However, most existing speaker anonymization systems employ parallel disentanglement methods, which only separate speech into linguistic content and speaker identity, often neglecting the preservation of the original emotional state. In this study, we introduce EASY, an emotion-aware speaker anonymization framework. EASY employs a novel sequential disentanglement process to disentangle speaker identity, linguistic content, and emotional representation, modeling each speech attribute in distinct subspaces through a factorized distillation approach. By independently constraining speaker identity and emotional representation, EASY minimizes information leakage, enhancing privacy protection while preserving original linguistic content and emotional state. Experimental results on the VoicePrivacy Challenge official datasets demonstrate that our proposed approach outperforms all baseline systems, effectively protecting speaker privacy while maintaining linguistic content and emotional state.
Private kNN-VC: Interpretable Anonymization of Converted Speech
Authors: Carlos Franzreb; Arnab Das; Tim Polzehl; Sebastian MÃ¶ller
Venue: Interspeech 2025 (main)

Abstract: Speaker anonymization seeks to conceal a speaker's identity while preserving the utility of their speech. The achieved privacy is commonly evaluated with a speaker recognition model trained on anonymized speech. Although this represents a strong attack, it is unclear which aspects of speech are exploited to identify the speakers. Our research sets out to unveil these aspects. It starts with kNN-VC, a powerful voice conversion model that performs poorly as an anonymization system, presumably because of prosody leakage. To test this hypothesis, we extend kNN-VC with two interpretable components that anonymize the duration and variation of phones. These components increase privacy significantly, proving that the studied prosodic factors encode speaker identity and are exploited by the privacy attack. Additionally, we show that changes in the target selection algorithm considerably influence the outcome of the privacy attack.
Legally validated evaluation framework for voice anonymization
Authors: Nathalie Vauquier; Brij Mohan Lal Srivastava; Seyed Ahmad Hosseini; Emmanuel Vincent
Venue: Interspeech 2025 (main)

Abstract: Classical speaker verification metrics used to evaluate voice anonymization systems, such as the equal error rate (EER), fail to properly quantify the residual re-identification risk. This paper introduces a new evaluation framework based on two metrics, Linkability and Singling Out, derived from the legal definitions in the Article 29 Working Party's Opinion 05/2014 on Anonymization Techniques endorsed by the European Data Protection Board (EDPB). Our framework translates these legal concepts into quantitative metrics for speech data. The proposed framework has been legally validated by the French Data Protection Authority. Experiments across various attack scenarios reveal that, while the EER remains stable, Linkability and Singling Out vary much more. This demonstrates that the residual privacy risk after anonymization is far more variable than indicated by the EER, underscoring the need for evaluation metrics that align with legal criteria.
Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
Authors: Chi-Yuan Hsiao; Ke-Han Lu; Kai-Wei Chang; Chih-Kai Yang; Wei-Chih Chen; Hung-yi Lee
Venue: Interspeech 2025 (main)

Abstract: End-to-end training of Spoken Language Models (SLMs) commonly involves adapting pre-trained text-based Large Language Models (LLMs) to the speech modality through multi-stage training on diverse tasks such as ASR, TTS and spoken question answering (SQA). Although this multi-stage continual learning equips LLMs with both speech understanding and generation capabilities, the substantial differences in task and data distributions across stages can lead to catastrophic forgetting, where previously acquired knowledge is lost. This paper investigates catastrophic forgetting and evaluates three mitigation strategiesâmodel merging, discounting the LoRA scaling factor, and experience replay to balance knowledge retention with new learning. Results show that experience replay is the most effective, with further gains achieved by combining it with other methods. These findings provide insights for developing more robust and efficient SLM training pipelines.
Speechless: Speech Instruction Training Without Speech for Low Resource Languages
Authors: Alan Dao; Dinh Bach Vu; Huy Hoang Ha; Tuan Le Duc Anh; Shreyas Gopal; Yue Heng Yeo; Warren Keng Hoong Low; Eng Siong Chng; Jia Qi Yip
Venue: Interspeech 2025 (main)

Abstract: The rapid growth of voice assistants powered by large language models (LLM) has highlighted a need for speech instruction data to train these systems. Despite the abundance of speech recognition data, there is a notable scarcity of speech instruction data, which is essential for fine-tuning models to understand and execute spoken commands. Generating high-quality synthetic speech requires a good text-to-speech (TTS) model, which may not be available to low resource languages. Our novel approach addresses this challenge by halting synthesis at the semantic representation level, bypassing the need for TTS. We achieve this by aligning synthetic semantic representations with the pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text instructions while maintaining the ability to understand spoken instructions during inference. This simplified training process is a promising approach to building voice assistant for low-resource languages.
LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs
Authors: Pooneh Mousavi; Shubham Gupta; Cem Subakan; Mirco Ravanelli
Venue: Interspeech 2025 (main)

Abstract: Foundation models based on large language models (LLMs) have shown great success in handling various tasks and modalities. However, adapting these models for general-purpose audio-language tasks is challenging due to differences in acoustic environments and task variations. In this work, we introduce LiSTEN (Learning Soft Token Embeddings for Neural Audio LLMs), a framework for adapting LLMs to speech and audio tasks. LiSTEN uses a dynamic prompt selection strategy with learnable key-value pairs, allowing the model to balance general and task-specific knowledge while avoiding overfitting in a multitask setting. Our approach reduces dependence on large-scale ASR or captioning datasets, achieves competitive performance with fewer trainable parameters, and simplifies training by using a single-stage process. Additionally, LiSTEN enhances interpretability by analyzing the diversity and overlap of selected prompts across different tasks.
Cryfish: On deep audio analysis with Large Language Models
Authors: Anton Mitrofanov; Sergey Novoselov; Tatiana Prisyach; Vladislav Marchevskiy; Arseniy Karelin; Nikita Khmelev; Dmitry Dutov; Stepan Malykh; Igor Agafonov; Aleksandr Nikitin; Oleg Petrov
Venue: Interspeech 2025 (main)

Abstract: The recent revolutionary progress in text-based large language models (LLMs) has contributed to the growth of interest in extending capabilities of such models to multimodal perception and understanding tasks. Hearing is an essential capability that is highly desired to be integrated into LLMs. However, effective integrating listening capabilities into LLMs is a significant challenge lying in generalizing complex auditory tasks across speech and sounds. To address these issues, we introduce Cryfish, our version of auditory-capable LLM. The model integrates WavLM audio-encoder features into Qwen2 model using a transformer-based connector. Cryfish is adapted to various auditory tasks through a specialized training strategy. We evaluate the model on the new Dynamic SUPERB Phase-2 comprehensive multitask benchmark specifically designed for auditory-capable models. The paper presents an in-depth analysis and detailed comparison of Cryfish with the publicly available models.