GenAI4ELab/papercli-papers huggingface dataset:
Efficient Speech Enhancement via Embeddings from Pre-trained Generative Audioencoders
Authors: Xingwei Sun; Heinrich Dinkel; Yadong Niu; Linzhang Wang; Junbo Zhang; Jian Luan
Venue: Interspeech 2025 (main)

Abstract: Recent research has delved into speech enhancement (SE) approaches that leverage audio embeddings from pre-trained models, diverging from time-frequency masking or signal prediction techniques. This paper introduces an efficient and extensible SE method. Our approach involves initially extracting audio embeddings from noisy speech using a pre-trained audioencoder, which are then denoised by a compact encoder network. Subsequently, a vocoder synthesizes the clean speech from denoised embeddings. An ablation study substantiates the parameter efficiency of the denoise encoder with a pre-trained audioencoder and vocoder. Experimental results on both speech enhancement and speaker fidelity demonstrate that our generative audioencoder-based SE system outperforms models utilizing discriminative audioencoders. Furthermore, subjective listening tests validate that our proposed system surpasses an existing state-of-the-art SE model in terms of perceptual quality.
Towards Personalised Audio Visual Speech Enhancement
Authors: Mandar Gogate; Kia Dashtipour; Amir Hussain
Venue: Interspeech 2025 (main)

Abstract: Personalised speech enhancement (PSE) and audio-visual (AV) speech enhancement (SE) have emerged as promising approaches to improve speech quality and intelligibility in challenging acoustic environments. PSE leverages individual-specific vocal characteristics to address the label permutation problem, while AV SE incorporates visual cues, particularly lip movements, to complement auditory signals in noisy conditions where speech is degraded by competing noise sources. This paper presents a novel framework that unifies these two, advancing towards personalised AV SE. By integrating raw enrolment audio for adaptive target speaker representation with AV inputs the proposed system aims to achieve robust SE in real-world environments. Experimental results demonstrate significant improvements in speech intelligibility and noise suppression on the COG-MHEAR Audio-Visual Speech Enhancement Challenge dataset, outperforming state-of-the-art PSE and AV SE models.
FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching
Authors: Ziqian Wang; Zikai Liu; Xinfa Zhu; Yike Zhu; Mingshuai Liu; Jun Chen; Longshuai Xiao; Chao Weng; Lei Xie
Venue: Interspeech 2025 (main)

Abstract: Generative models have excelled in audio tasks using approaches such as language models, diffusion, and flow matching. However, existing generative approaches for speech enhancement (SE) face notable challenges: language model-based methods suffer from quantization loss, leading to compromised speaker similarity and intelligibility, while diffusion models require complex training and high inference latency. To address these challenges, we propose FlowSE, a flow-matching-based model for SE. Flow matching learns a continuous transformation between noisy and clean speech distributions in a single pass, significantly reducing inference latency while maintaining high-quality reconstruction. Specifically, FlowSE trains on noisy mel spectrograms and optional character sequences, optimizing a condition flow matching loss with ground-truth mel spectrograms as supervision. It implicitly learns speechâs temporal spectral structure and text-speech alignment. During inference, FlowSE can operate with or without textual information, achieving impressive results in both scenarios, with further improvements when transcripts are available. Extensive experiments demonstrate that FlowSE significantly outperforms state-of-the-art generative methods, establishing a new paradigm for generative-based SE and demonstrating the potential of flow matching to advance the field. Our code, pre-trained checkpoints, and audio samples are available.
Speech Enhancement based on cascaded two flows
Authors: Seonggyu Lee; Sein Cheong; Sangwook Han; Kihyuk Kim; Jong Won Shin
Venue: Interspeech 2025 (main)

Abstract: Speech enhancement (SE) based on diffusion probabilistic models has exhibited impressive performance, while requiring a relatively high number of function evaluations (NFE). Recently, SE based on flow matching has been proposed, which showed competitive performance with a small NFE. Early approaches adopted the noisy speech as the only conditioning variable. There have been other approaches which utilize speech enhanced with a predictive model as another conditioning variable and to sample an initial value, but they require a separate predictive model on top of the generative SE model. In this work, we propose to employ an identical model based on flow matching for both SE and generating enhanced speech used as an initial starting point and a conditioning variable. Experimental results showed that the proposed method required the same or fewer NFEs even with two cascaded generative methods while achieving equivalent or better performances to the previous baselines.
X-ARES: A Comprehensive Framework for Assessing Audio Encoder Performance
Authors: Junbo Zhang; Heinrich Dinkel; Yadong Niu; Chenyu Liu; Si Cheng; Anbei Zhao; Jian Luan
Venue: Interspeech 2025 (main)

Abstract: We introduces X-ARES (eXtensive Audio Representation and Evaluation Suite), a novel open-source benchmark designed to systematically assess audio encoder performance across diverse domains. By encompassing tasks spanning speech, environmental sounds, and music, X-ARES provides two evaluation approaches for evaluating audio representations: linear fine-tuning and unparameterized evaluation. The framework includes 22 distinct tasks that cover essential aspects of audio processing, from speech recognition and emotion detection to sound event classification and music genre identification. Our extensive evaluation of state-of-the-art audio encoders reveals significant performance variations across different tasks and domains, highlighting the complexity of general audio representation learning.
WavShape: Information-Theoretic Speech Representation Learning for Fair and Privacy-Aware Audio Processing
Authors: Oguzhan Baser; Ahmet Ege Tanriverdi; Kaan Kale; Sandeep Chinchali; Sriram Vishwanath
Venue: Interspeech 2025 (main)

Abstract: Speech embeddings often retain sensitive attributes such as speaker identity, accent, or demographic information, posing risks in biased model training and privacy leakage. We propose WavShape, an information-theoretic speech representation learning framework that optimizes embeddings for fairness and privacy while preserving task-relevant information. We leverage mutual information (MI) estimation using the Donsker-Varadhan formulation to guide an MI-based encoder that systematically filters sensitive attributes while maintaining speech content essential for downstream tasks. Experimental results on three known datasets show that WavShape reduces MI between embeddings and sensitive attributes by up to 81% while retaining 97% of task-relevant information. By integrating information theory with self-supervised speech models, this work advances the development of fair, privacy-aware, and resource-efficient speech systems.
FreeCodec: A Disentangled Neural Speech Codec with Fewer Tokens
Authors: Youqiang Zheng; Weiping Tu; Yueteng Kang; Jie Chen; Yike Zhang; Li Xiao; Yuhong Yang; Long Ma
Venue: Interspeech 2025 (main)

Abstract: Neural speech codec is a crucial component in generative tasks such as speech resynthesis and zero-shot TTS. However, most works exhibit degraded performance with fewer tokens due to low coding efficiency in modeling complex coupled information. In this paper, we propose a self-supervised disentangled neural speech codec named FreeCodec. It employs distinct frame-level encoders to decompose intrinsic speech properties into separate components and adopts enhanced decoders to reconstruct speech signals. By encoding and quantizing the different frame-level information with dedicated quantizers, FreeCodec gets higher encoding efficiency with 57 tokens. Furthermore, our proposed method can be applied flexibly in reconstruction and disentanglement scenarios with different training strategies. Subjective and objective experimental results demonstrate that our framework outperforms existing methods in both reconstruction and disentanglement tasks.
DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation
Authors: Jiaqi Li; Xiaolong Lin; Zhekai Li; Shixi Huang; Yuancheng Wang; Chaoren Wang; Zhenpeng Zhan; Zhizheng Wu
Venue: Interspeech 2025 (main)

Abstract: Neural audio codecs form the foundational building blocks for language model (LM)-based speech generation. Typically, there is a trade-off between frame rate and audio quality. This study introduces a low-frame-rate, semantically enhanced codec model. Existing approaches distill semantically rich self-supervised (SSL) representations into the first-layer codec tokens. This work proposes DualCodec, a dual-stream encoding approach that integrates SSL and waveform representations within an end-to-end codec framework. In this setting, DualCodec enhances the semantic information in the first-layer codec and enables the codec system to maintain high audio quality while operating at a low frame rate. Note that a low-frame-rate codec improves the efficiency of speech generation. Experimental results on audio codec and speech generation tasks confirm the effectiveness of the proposed DualCodec compared to state-of-the-art codec systems, such as Mimi Codec, SpeechTokenizer, DAC, and Encodec.
Comparative Analysis of Fast and High-Fidelity Neural Vocoders for Low-Latency Streaming Synthesis in Resource-Constrained Environments
Authors: Reo Yoneyama; Masaya Kawamura; Ryo Terashima; Ryuichi Yamamoto; Tomoki Toda
Venue: Interspeech 2025 (main)

Abstract: In real-time speech synthesis, neural vocoders often require low-latency synthesis through causal processing and streaming. However, streaming introduces inefficiencies absent in batch synthesis, such as limited parallelism, inter-frame dependency management, and parameter loading overhead. This paper proposes multi-stream Wavehax (MS-Wavehax), an efficient neural vocoder for low-latency streaming, by extending the aliasing-free neural vocoder Wavehax with multi-stream decomposition. We analyze the latency-throughput trade-off in a CPU-only environment and identify key bottlenecks in streaming neural vocoders. Our findings provide practical insights for optimizing chunk sizes and designing vocoders tailored to specific application demands and hardware constraints. Furthermore, our subjective evaluations show that MS-Wavehax delivers high speech quality under causal and non-causal conditions while being remarkably compact and easily deployable in resource-constrained environments.
Prosody-Adaptable Audio Codecs for Zero-Shot Voice Conversion via In-Context Learning
Authors: Junchuan Zhao; Xintong Wang; Ye Wang
Venue: Interspeech 2025 (main)

Abstract: Recent advances in discrete audio codecs have significantly improved speech representation modeling, while codec language models have enabled in-context learning for zero-shot speech synthesis. Inspired by this, we propose a voice conversion (VC) model within the VALLE-X framework, leveraging its strong in-context learning capabilities for speaker adaptation. To enhance prosody control, we introduce a prosody-aware audio codec encoder (PACE) module, which isolates and refines prosody from other sources, improving expressiveness and control. By integrating PACE into our VC model, we achieve greater flexibility in prosody manipulation while preserving speaker timbre. Experimental evaluation results demonstrate that our approach outperforms baseline VC systems in prosody preservation, timbre consistency, and overall naturalness, surpassing baseline VC systems.
Vocoder-Projected Feature Discriminator
Authors: Takuhiro Kaneko; Hirokazu Kameoka; Kou Tanaka; Yuto Kondo
Venue: Interspeech 2025 (main)

Abstract: In text-to-speech (TTS) and voice conversion (VC), acoustic features, such as mel spectrograms, are typically used as synthesis or conversion targets owing to their compactness and ease of learning. However, because the ultimate goal is to generate high-quality waveforms, employing a vocoder to convert these features into waveforms and applying adversarial training in the time domain is reasonable. Nevertheless, upsampling the waveform introduces significant time and memory overheads. To address this issue, we propose a vocoder-projected feature discriminator (VPFD), which uses vocoder features for adversarial training. Experiments on diffusion-based VC distillation demonstrated that a pretrained and frozen vocoder feature extractor with a single upsampling step is necessary and sufficient to achieve a VC performance comparable to that of waveform discriminators while reducing the training time and memory consumption by 9.6 and 11.4 times, respectively.
AF-Vocoder: Artifact-Free Neural Vocoder with Global Artifact Filter
Authors: Zhuangqi Chen; Xianjun Xia; Xiaohuai Le; Siyu Sun; Chuanzeng Huang
Venue: Interspeech 2025 (main)

Abstract: Recent studies have demonstrated the advantage of generative adversarial network (GAN)-based vocoders in high-fidelity speech synthesis and fast inference speed. However, they often suffer from audible artifacts such as aliasing and blurring. In this paper, we propose AF-Vocoder, a novel GAN-based vocoder that can synthesize high-fidelity speech with fewer artifacts. Specifically, we introduce a frequency-domain artifacts filter named GAFilter to achieve artifact removal. GAFilter incorporates a learnable frequency filter, which enforces a desired inductive bias of frequency control for artifact-free speech synthesis. Experimental results show that the proposed AF-Vocoder outperforms other GAN-based vocoders in speech reconstruction quality and artifact suppression on various datasets including out-of-domain speakers.
DS-Codec: Dual-Stage Training with Mirror-to-NonMirror Architecture Switching for Speech Codec
Authors: Peijie Chen; Wenhao Guan; Kaidi Wang; Weijie Wu; Hukai Huang; Qingyang Hong; Lin Li
Venue: Interspeech 2025 (main)

Abstract: Neural speech codecs are essential for advancing text-to-speech (TTS) systems. With the recent success of large language models in text generation, developing high-quality speech tokenizers has become increasingly important. This paper introduces DS-Codec, a novel neural speech codec featuring a dual-stage training framework with mirror and non-mirror architectures switching, designed to achieve superior speech reconstruction. We conduct extensive experiments and ablation studies to evaluate the effectiveness of our training strategy and compare the performance of the two architectures. Our results show that the mirrored structure significantly enhances the robustness of the learned codebooks, and the training strategy balances the advantages between mirrored and non-mirrored structures, leading to improved high-fidelity speech reconstruction.
PeriodCodec: A Pitch-Controllable Neural Audio Codec Using Periodic Signals for Singing Voice Synthesis
Authors: Masato Takagi; Miku Nishihara; Yukiya Hono; Kei Hashimoto; Yoshihiko Nankaku; Keiichi Tokuda
Venue: Interspeech 2025 (main)

Abstract: Neural audio codecs (NACs) have attracted considerable attention in the field of text-to-speech.  However, previous methods don't offer a mechanism for explicit controlling the fundamental frequency (F0), hence they are not suitable for singing voice synthesis. To overcome this limitation, we propose a NAC that can control F0 by introducing explicit periodic signals into the decoder. This architecture enables direct manipulation of F0 during the synthesis process. Experimental results show that our proposed method achieves F0 control and improves synthesis quality compared to previous methods. Furthermore, by including singing voices in the training data set, we showed that both F0 controllability and the quality of singing voices are improved, enabling the construction of a NAC suitable for singing voice synthesis tasks.
Enhancing Target-speaker Automatic Speech Recognition Using Multiple Speaker Embedding Extractors with Virtual Speaker Embedding
Authors: Ju-Seok Seong; Jeong-Hwan Choi; Ye-Rin Jeoung; Ilseok Kim; Joon-Hyuk Chang
Venue: Interspeech 2025 (main)

Abstract: Target-speaker automatic speech recognition (TS-ASR) utilizes speaker embeddings to identify a target speaker in multi-talker environments. While high-performance speaker embedding extractors provide discriminative embeddings, their computational demands limit practical deployment. In this study, we present two novel methods that effectively utilize lightweight extractors to enhance TS-ASR performance. First, we propose a multiple embeddings modulation that effectively transfers comprehensive speaker information to the ASR module, thereby improving overall performance and robustness against embedding variations. Second, we present a virtual speaker embedding augmentation technique that synthesizes embeddings of unseen speakers, reducing dependence on specific extractors while enhancing independent contributions from each extractor. Experimental results on the Libri2Mix dataset demonstrate that our proposed methods achieve significant WER reductions compared to the baseline model.
SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition
Authors: Yuta Hirano; Sakriani Sakti
Venue: Interspeech 2025 (main)

Abstract: We propose Speaker-Conditioned Serialized Output Training (SC-SOT), an enhanced SOT-based training for E2E multi-talker ASR. We first probe how SOT handles overlapped speech, and we found the decoder performs implicit speaker separation. We hypothesize this implicit separation is often insufficient due to ambiguous acoustic cues in overlapping regions. To address this, SC-SOT explicitly conditions the decoder on speaker information, providing detailed information about "who spoke when". Specifically, we enhance the decoder by incorporating: (1) speaker embeddings, which allow the model to focus on the acoustic characteristics of the target speaker, and (2) speaker activity information, which guides the model to suppress non-target speakers. The speaker embeddings are derived from a jointly trained E2E speaker diarization model, mitigating the need for speaker enrollment. Experimental results demonstrate the effectiveness of our conditioning approach on overlapped speech.
Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filtering
Authors: Pradeep Rangappa; AndrÃ©s Carofilis; Jeena Prakash; Shashi Kumar; Sergio Burdisso; Srikanth Madikeri; EsaÃº Villatoro-Tello; Bidisha Sharma; Petr Motlicek; Kadri Hacioglu; Shankar Venkatesan; Saurabh Vyas; Andreas Stolcke
Venue: Interspeech 2025 (main)

Abstract: Fine-tuning pretrained ASR models for specific domains is challenging for small organizations with limited labeled data and computational resources. Here we explore different data selection pipelines and propose a robust approach that improves ASR adaptation by filtering pseudo-labels generated using Whisper (encoder-decoder) and Zipformer (transducer) models. Our approach integrates multiple selection strategies-including word error rate (WER) prediction, named entity recognition (NER), and character error rate (CER) analysis-to extract high-quality training segments. We evaluate our method on Whisper and Zipformer using a 7500-hour baseline, comparing it to a CER-based approach relying on hypotheses from three ASR systems. Fine-tuning on 7500 hours of pseudo-labeled call center data achieves 12.3% WER, while our filtering reduces the dataset to 100 hours (1.4%) with similar performance; a similar trend is observed on Fisher English.
MOPSA: Mixture of Prompt-Experts Based Speaker Adaptation for Elderly Speech Recognition
Authors: Chengxi Deng; Xurong Xie; Shujie Hu; Mengzhe Geng; Yicong Jiang; Jiankun Zhao; Jiajun Deng; Guinan Li; Youjun Chen; Huimeng Wang; Haoning Xu; Mingyu Cui; Xunying Liu
Venue: Interspeech 2025 (main)

Abstract: This paper proposes a novel Mixture of Prompt-Experts based Speaker Adaptation approach (MOPSA) for elderly speech recognition. It allows zero-shot, real-time adaptation to unseen speakers, and leverages domain knowledge tailored to elderly speakers. Top-K most distinctive speaker prompt clusters derived using K-means serve as experts. A router network is trained to dynamically combine clustered prompt-experts. Acoustic and language level variability among elderly speakers are modelled using separate encoder and decoder prompts for Whisper. Experiments on the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets suggest that online MOPSA adaptation outperforms the speaker-independent (SI) model by statistically significant word error rate (WER) or character error rate (CER) reductions of 0.86% and 1.47% absolute (4.21% and 5.40% relative). Real-time factor (RTF) speed-up ratios of up to 16.12 times are obtained over offline batch-mode adaptation.
Visually-Adaptive Guided Robust Speech Recognition with Parameter-Efficient Adaptation
Authors: Zhao Yang; Rui Jiang; Yue Heng Yeo; Xiao Fu; Wei Xi; Jizhong Zhao
Venue: Interspeech 2025 (main)

Abstract: Recent developments in large-scale speech foundation models have further pushed the boundaries of automatic speech recognition (ASR) capabilities, making them excellent candidates for integration with multi-modality approaches. In this work, we propose AVWhisper-LoRA, an extension of the Whisper model that incorporates an auxiliary visual encoder to enable audio-visual speech recognition (AVSR) with lightweight trainable parameters. Our approach capitalizes on the existing attention mechanisms of the well-trained Whisper model, facilitating the integration of visual information through both self-attention and cross-attention interactions. Additionally, we introduce LoRA-based trainable lightweight adapters into the frozen Whisper model to enable effective adaptation to the multi-modality target domain during training. Experimental results on the LRS3-TED dataset demonstrate that our method consistently outperforms state-of-the-art methods, particularly in challenging speech environments.
Regularizing Learnable Feature Extraction for Automatic Speech Recognition
Authors: Peter Vieting; Maximilian Kannen; Benedikt Hilmes; Ralf SchlÃ¼ter; Hermann Ney
Venue: Interspeech 2025 (main)

Abstract: Neural front-ends are an appealing alternative to traditional, fixed feature extraction pipelines for automatic speech recognition (ASR) systems since they can be directly trained to fit the acoustic model. However, their performance often falls short compared to classical methods, which we show is largely due to their increased susceptibility to overfitting. This work therefore investigates regularization methods for training ASR models with learnable feature extraction front-ends. First, we examine audio perturbation methods and show that larger relative improvements can be obtained for learnable features. Additionally, we identify two limitations in the standard use of SpecAugment for these front-ends and propose masking in the short time Fourier transform (STFT)-domain as a simple but effective modification to address these challenges. Finally, integrating both regularization approaches effectively closes the performance gap between traditional and learnable features.