GenAI4ELab/papercli-papers huggingface dataset:
ARiSE: Auto-Regressive Multi-Channel Speech Enhancement
Authors: Pengjie Shen; Xueliang Zhang; Zhong-Qiu Wang
Venue: Interspeech 2025 (main)

Abstract: We propose ARiSE, an auto-regressive algorithm for multi-channel speech enhancement. ARiSE improves existing deep neural network (DNN) based frame-online multi-channel speech enhancement models by introducing auto-regressive connections, where the estimated target speech at previous frames is leveraged as extra input features to help the DNN estimate the target speech at the current frame. The extra input features can be derived from (a) the estimated target speech in previous frames; and (b) a beamformed mixture with the beamformer computed based on the previous estimated target speech. On the other hand, naively training the DNN in an auto-regressive manner is very slow. To deal with this, we propose a parallel training mechanism to speed up the training. Evaluation results in noisy-reverberant conditions show the effectiveness and potential of the proposed algorithms.
WTFormer: A Wavelet Conformer Network for MIMO Speech Enhancement with Spatial Cues Peservation
Authors: Lu Han; Junqi Zhao; Renhua Peng
Venue: Interspeech 2025 (main)

Abstract: Current multi-channel speech enhancement systems mainly adopt single-output architecture, which face significant challenges in preserving spatio-temporal signal integrity during multiple-input multiple-output (MIMO) processing. To address this limitation, we propose a novel neural network, termed WTFormer, for MIMO speech enhancement that leverages the multi-resolution characteristics of wavelet transform and multi-dimensional collaborative attention to effectively capture globally distributed spatial features, while using Conformer for time-frequency modeling. A multi task loss strategy accompanying MUSIC algorithm is further proposed for optimization training to protect spatial information to the greatest extent. Experimental results on the LibriSpeech dataset show that WTFormer can achieve comparable denoising performance to advanced systems while preserving more spatial information with only 0.98M parameters.
A Three-Stage Beamforming with Harmonic Guidance for Multi-Channel Speech Enhancement
Authors: Nurali Alip; Tianrui Wang; Rui Cao; Meng Ge; Jingru Lin; Longbiao Wang; Jianwu Dang
Venue: Interspeech 2025 (main)

Abstract: With the rapid advancement of multi-channel speech enhancement (MCSE) research, nonlinear spatial filtering methods integrating spatial and spectral processing are prevalent. However, many existing approaches overlook the explicit extraction of speech spectral structure information, leading to insufficient learning of joint spatial-spectral information,  limiting  performance under low signal-to-noise ratio (SNR) conditions. To address this, we propose a three-stage multi-channel speech enhancement framework. The first stage has an acoustic structure extraction module capturing speech spectral patterns from noisy inputs, enabling  spatial-spectral cue exploration and interaction. In the next two stages, the process combines full-band noise reduction with speech structure refinement by decoupling enhancement into coarse enhancement and spectral refinement. Experiments on LibriSpeech-based datasets demonstrate that the proposed method significantly outperforms the reference method.
Speech Enhancement with Dual-path Multi-Channel Linear Prediction Filter and Multi-norm Beamforming
Authors: Chengyuan Qin; Wenmeng Xiong; Jing Zhou; Maoshen Jia; Changchun Bao
Venue: Interspeech 2025 (main)

Abstract: In this paper, we propose a speech enhancement method using dual-path Multi-Channel Linear Prediction (MCLP) filters and multi-norm beamforming. Specifically, the MCLP part in the proposed method is designed with dual-path filters in both time and frequency dimensions. For the beamforming part, we minimize the power of the microphone array output as well as the l1 norm of the denoised signals while preserving source signals from the target directions. An efficient method to select the prediction orders in the dual-path filters is also proposed, which is robust for signals with different reverberation time (T60) values and can be applied to other MCLP-based methods. Evaluations demonstrate that our proposed method outperforms the baseline methods for speech enhancement, particularly in high reverberation scenarios.
Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR
Authors: Mingyu Cui; Yifan Yang; Jiajun Deng; Jiawen Kang; Shujie Hu; Tianzi Wang; Zhaoqing Li; Shiliang Zhang; Xie Chen; Xunying Liu
Venue: Interspeech 2025 (main)

Abstract: This paper investigates discrete tokens based cross-utterance speech contexts modelling for Zipformer-Transducer (Z-T) systems. Their efficacy and efficiency in modelling preceding, current and future speech utterance contexts using concatenation or pooling projection of Z-T encoder embeddings are extensively shown on the 1000-hr GigaSpeech-M and DementiaBank Pitt elderly speech datasets over comparable contextual Z-T baselines using filterbank or continuous WavLM features}. The best performing discrete tokens based contextual Z-T system outperforms the non-contextual baseline by statistically significant average WER reductions of 0.39% and 1.41% absolute (3.4% and 3.4% relative) on the two tasks, respectively. Model training time speedup ratios up to 4.36x is obtained over continuous WavLM feature-based contextual Z-T systems, while retaining up to 98.0% of their WER reductions over non-contextual baselines.
Self-supervised learning of speech representations with Dutch archival data
Authors: Nik Vaessen; Roeland Ordelman; David A. van Leeuwen
Venue: Interspeech 2025 (main)

Abstract: This paper explores the use of Dutch archival television broadcast data for self-supervised learning of speech foundation models, specifically wav2vec 2.0. We first study data quality assumptions for pre-training, and show how music, noise and speaker overlap affect SSL convergence and downstream fine-tuning performance. Secondly, we explore effectively pre-processing strategies to convert the noisy broadcast dataset into a qualitative dataset for pre-training, by using Whisper and WhisperX. Thirdly, we compare mono-lingual and multi-lingual pre-training with equivalent amounts of data, and show that mono-lingual pre-training is more robust to out-of-domain data. Lastly, we achieve a state-of-the-art LARGE wav2vec 2.0 model for the Dutch language, by a continuation of pre-training a wav2vec 2.0 XLS-R model checkpoint with our 55 k hour archival dataset.
GigaAM: Efficient Self-Supervised Learner for Speech Recognition
Authors: Aleksandr Kutsakov; Alexandr Maximenko; Georgii Gospodinov; Pavel Bogomolov; Fyodor Minkin
Venue: Interspeech 2025 (main)

Abstract: Self-Supervised Learning (SSL) has demonstrated strong performance in speech processing, particularly in automatic speech recognition. In this paper, we explore an SSL pretraining framework that leverages masked language modeling with targets derived from a speech recognition model. We also present chunkwise attention with dynamic chunk size sampling during pretraining to enable both full-context and streaming fine-tuning. Our experiments examine scaling with respect to model size and the amount of data. Using our method, we train the GigaAM family of models, including a state-of-the-art model for Russian speech recognition that outperforms Whisper-large-v3 by 50%. We have released our foundation and ASR models, along with the inference code, under the MIT license as open-source resources to the research community.
DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
Authors: Hyung-gun Chi; Zakaria Aldeneh; Tatiana Likhomanenko; Oggi Rudovic; Takuya Higuchi; Li-Wei Chen; Shinji Watanabe; Ahmed Hussen Abdelaziz
Venue: Interspeech 2025 (main)

Abstract: We introduce DiceHuBERT, a knowledge distillation framework for compressing HuBERT, a widely used self-supervised learning (SSL)-based speech foundation model. Unlike existing distillation methods that rely on layer-wise and feature-wise mapping between teacher and student models, DiceHuBERT leverages  HuBERT's iterative self-distillation mechanism by directly replacing the original model with a student model. This replacement allows the student to be trained using the same SSL objective used when pre-training HuBERT, eliminating the need for additional modules or architectural constraints. Experimental results on SUPERB show that DiceHuBERT consistently outperforms existing distillation methods, improving phoneme recognition performance by over 21% and ASR performance by more than 14%. Furthermore, DiceHuBERT demonstrates competitive performance across multiple tasks, highlighting its clear advantage.
Differentiable K-means for Fully-optimized Discrete Token-based ASR
Authors: Kentaro Onda; Yosuke Kashiwagi; Emiru Tsunoo; Hayato Futami; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: Recent studies have highlighted the potential of discrete tokens derived from self-supervised learning (SSL) models for various speech-related tasks. These tokens serve not only as substitutes for text in language modeling but also as intermediate representations for tasks such as automatic speech recognition (ASR). However, discrete tokens are typically obtained via k-means clustering of SSL features independently of downstream tasks, making them suboptimal for specific applications. This paper proposes the use of differentiable k-means, enabling the joint optimization of tokenization and downstream tasks. This approach enables the fine-tuning of the SSL parameters and learning weights for outputs from multiple SSL layers. Experiments were conducted with ASR as a downstream task. ASR accuracy successfully improved owing to the optimized tokens. The acquired tokens also exhibited greater purity of phonetic information, which were found to be useful even in speech resynthesis.
Towards Early Prediction of Self-Supervised Speech Model Performance
Authors: Ryan Whetten; Lucas Maison; Titouan Parcollet; Marco Dinarelli; Yannick EstÃ¨ve
Venue: Interspeech 2025 (main)

Abstract: In Self-Supervised Learning (SSL), pre-training and evaluation are resource intensive. In the speech domain, current indicators of the quality of SSL models during pre-training, such as the loss, do not correlate well with downstream performance. Consequently, it is often difficult to gauge the final downstream performance in a cost efficient manner during pre-training. In this work, we propose unsupervised efficient methods that give insights into the pre-training quality of SSL speech models, namely, measuring the cluster quality and rank of the embeddings produced by the SSL model. Results show that measures of cluster quality and rank correlate better with downstream performance than the pre-training loss, reducing the need for GPU hours and labeled data in SSL model evaluation.
VibE-SVC: Vibrato Extraction with High-frequency F0 Contour for Singing Voice Conversion
Authors: Joon-Seung Choi; Dong-Min Byun; Hyung-Seok Oh; Seong-Whan Lee
Venue: Interspeech 2025 (main)

Abstract: Controlling singing style is crucial for achieving an expressive and natural singing voice. Among the various style factors, vibrato plays a key role in conveying emotions and enhancing musical depth. However, modeling vibrato remains challenging due to its dynamic nature, making it difficult to control in singing voice conversion. To address this, we propose VibE-SVC, a controllable singing voice conversion model that explicitly extracts and manipulates vibrato using discrete wavelet transform. Unlike previous methods that model vibrato implicitly, our approach decomposes the F0 contour into frequency components, enabling precise transfer. This allows vibrato control for enhanced flexibility. Experimental results show that VibE-SVC effectively transforms singing styles while preserving speaker similarity. Both subjective and objective evaluations confirm high-quality conversion.
TVC-MusicGen: Time-Varying Structure Control for Background Music Generation via Self-Supervised Training
Authors: Chenyu Yang; Hangting Chen; Shuai Wang; Haina Zhu; Haizhou Li
Venue: Interspeech 2025 (main)

Abstract: Current text-to-music generation models typically are not involved in generating music with specific structures, thus not meeting some customized needs in practical applications. To address this limitation, we propose a self-supervised Time-Varying Control method (TVC-MusicGen). By providing the temporal boundary and text descriptions for each segment, it can effectively generate music adhering to the corresponding structures. TVC-MusicGen supports generation from both text (text-to-music) and existing music clips (music-to-music), enabling structure editing or local style transfer. Additionally, we propose a generation-based approach to bridge the gap between text and audio modalities in cross-modal models, which are typically used as feature extractors in text-to-music systems. Experiments on both language and diffusion-based models demonstrate that our approach achieves effective control without compromising overall quality.
Audiobox TTA-RAG: Improving Zero-Shot and Few-Shot Text-To-Audio with Retrieval-Augmented Generation
Authors: Mu Yang; Bowen Shi; Matthew Le; Wei-Ning Hsu; Andros Tjandra
Venue: Interspeech 2025 (main)

Abstract: This work focuses on improving Text-To-Audio (TTA) generation on zero-shot and few-shot settings (i.e. generating unseen or uncommon audio events). Inspired by the success of Retrieval-Augmented Generation (RAG) in Large Language Models, we propose Audiobox TTA-RAG, a novel retrieval-augmented TTA approach based on Audiobox, a flow-matching audio generation model. Unlike the vanilla Audiobox TTA solution that generates audio conditioned on text only, we extend the TTA process by augmenting the conditioning input with both text and retrieved audio samples. Our retrieval method does not require the external database to have labeled audio, offering more practical use cases. We show that the proposed model can effectively leverage the retrieved audio samples and significantly improve zero-shot and few-shot TTA performance, with large margins on multiple evaluation metrics, while maintaining the ability to generate semantically aligned audio for the in-domain setting.
Bridging Speech and Singing: Multi-stage Speech-Prompted Singing Voice Conversion with Speaker Embedding Adaptation
Authors: Mingda Liu; Jiatong Shi
Venue: Interspeech 2025 (main)

Abstract: Singing voice conversion (SVC) technology holds immense potential to transform the entertainment industry and advance human-computer interaction. Unlike previous SVC studies that primarily focus on the conversion between different singing voices, this work specifically investigates the conversion of speech timbre into singing. In this paper, we introduce a novel Singing Speech Alignment Network (SSAN) designed to align speaker and singer embeddings derived from input data. To effectively train the SVC model, we also propose a cycle training strategy that ensures the performance of SVC with speech prompts. Experimental results demonstrate that the proposed method (SSANSVC) achieves superior performance in both the naturalness of the synthesized singing and its similarity to the target speech.
Neurodyne: Neural Pitch Manipulation with Representation Learning and Cycle-Consistency GAN
Authors: Yicheng Gu; Chaoren Wang; Zhizheng Wu; Lauri Juvela
Venue: Interspeech 2025 (main)

Abstract: Pitch manipulation is the process of producers adjusting the pitch of an audio segment to a specific key and intonation, which is essential in music production. Neural-network-based pitch-manipulation systems have been popular in recent years due to their superior synthesis quality compared to classical DSP methods. However, their performance is still limited due to their inaccurate feature disentanglement using source-filter models and the lack of paired in- and out-of-tune training data. This work proposes Neurodyne to address these issues. Specifically, Neurodyne uses adversarial representation learning to learn a pitch-independent latent representation to avoid inaccurate disentanglement and cycle-consistency training to create paired training data implicitly. Experimental results on global-key and template-based pitch manipulation demonstrate the effectiveness of the proposed system, marking improved synthesis quality while maintaining the original singer identity.
VS-Singer: Vision-Guided Stereo Singing Voice Synthesis with Consistency SchrÃ¶dinger Bridge
Authors: Zijing Zhao; Kai Wang; Hao Huang; Ying Hu; Liang He; Jichen Yang
Venue: Interspeech 2025 (main)

Abstract: To explore the potential advantages of utilizing spatial cues from images for generating stereo singing voices with room reverberation, we introduce VS-Singer, a vision-guided model designed to produce stereo singing voices with room reverberation from scene images. VS-Singer comprises three modules: firstly, a modal interaction network integrates spatial features into text encoding to create a linguistic representation enriched with spatial information. Secondly, the decoder employs a consistency SchrÃ¶dinger bridge to facilitate one-step sample generation. Moreover, we utilize the SFE module to improve the consistency of audio-visual matching. To our knowledge, this study is the first to combine stereo singing voice synthesis with visual acoustic matching within a unified framework. Experimental results demonstrate that VS-Singer can effectively generate stereo singing voices that align with the scene perspective in a single step.
DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching
Authors: Wei Chen; Binzhu Sha; Dan Luo; Jing Yang; Zhuo Wang; Fan Fan; Zhiyong Wu
Venue: Interspeech 2025 (main)

Abstract: Singing Voice Conversion (SVC) transfers a source singer's timbre to a target while keeping melody and lyrics. The key challenge in any-to-any SVC is adapting unseen speaker timbres to source audio without quality degradation. Existing methods either face timbre leakage or fail to achieve satisfactory timbre similarity and quality in the generated audio. To address these challenges, we propose DAFMSVC, where the self-supervised learning (SSL) features from the source audio are replaced with the most similar SSL features from the target audio to prevent timbre leakage. It also incorporates a dual-cross-attention mechanism for the adaptive fusion of speaker embeddings, melody, and linguistic content. Additionally, we introduce a flow matching module for high-quality audio generation from the fused features. Experimental results show that DAFMSVC significantly enhances timbre similarity and naturalness, outperforming state-of-the-art methods in both subjective and objective evaluations.
Simple and Effective Content Encoder for Singing Voice Conversion via SSL-Embedding Dimension Reduction
Authors: Wangjin Zhou; Tianjiao Du; Chenglin Xu; Sheng Li; Yi Zhao; Tatsuya Kawahara
Venue: Interspeech 2025 (main)

Abstract: In any-to-any singing voice conversion (SVC), singing content can be encoded using either token-based or embedding-based approaches. Token-based methods often struggle with accurate content reconstruction, while embedding-based methods face significant timbre leakage. To address this trade-off, we propose a novel self-supervised learning (SSL)-based content representation method. By randomly selecting a subset of channels during training to serve as the new embedding and fixing them for subsequent SVC training, our approach achieves superior content modeling compared to token-based methods while mitigating timbre leakage typically observed in embedding-based approaches. We validate the effectiveness and generalizability of our method across SSL-based embeddings, SSL-based soft embeddings, and ContentVec.
Song Form-aware Full-Song Text-to-Lyrics Generation with Multi-Level Granularity Syllable Count Control
Authors: Yunkee Chae; Eunsik Shin; Suntae Hwang; Seungryeol Paik; Kyogu Lee
Venue: Interspeech 2025 (main)

Abstract: Lyrics generation presents unique challenges, particularly in achieving precise syllable control while adhering to song form structures such as verses and choruses. Conventional line-by-line approaches often lead to unnatural phrasing, underscoring the need for more granular syllable management. We propose a framework for lyrics generation that enables multi-level syllable control at the word, phrase, line, and paragraph levels, aware of song form. Our approach generates complete lyrics conditioned on input text and song form, ensuring alignment with specified syllable constraints.
Multitalker Babble in English Vowel Perception Training: A Comparison between Humans and Neural Models
Authors: Wenwei Dong; Alif Silpachai; Catia Cucchiarini; Helmer Strik
Venue: Interspeech 2025 (main)

Abstract: Perceptual training with multitalker babble can benefit first language listeners; however, it is unclear whether such training is beneficial for second language (L2) listeners and whether there is an optimal number of talkers for creating babble. Since perception experiments with humans are complex and time-consuming and neural models are inspired by the human brain, we explore the use of neural models and study how well their performance aligns with the results for human listeners. In this study, we first investigated how babble produced by 2 and 6 talkers affected the perceptual learning of English vowels by L2 listeners. We then fine-tuned Automatic Speech Recognition (ASR) models using the same babble datasets. The results showed that babble regardless of the number of talkers benefited listeners in speech-shaped noise, and the Wav2Vec2.0 model also improved the accuracy with babble training and exhibited trends more similarly to humans than the TDNN model.
Speech stimulus design to study the neural coding of speech and the impact of cochlear synaptopathy
Authors: Etienne Gaudrain; Sarah Verhulst; Deniz BaÅkent
Venue: Interspeech 2025 (main)

Abstract: Since auditory neural coding mechanisms have been uncovered using simple artificial stimuli, the nature of their involvement in speech perception has remained unclear. However, using analysis-resynthesis techniques, it is possible to design naturalistic speech stimuli that present the same appeal as artificial tones in terms of precise control and parametrisation. Here, speech stimuli were designed to assess the involvement of a specific coding mechanism: the coding of temporal fine structure through phase-locking. This mechanism is supposed to be impaired in cochlear synaptopathy, a form of hearing loss that remains undetected through pure tone audiometry. The design process described here seeks to accommodate all the constraints that can arise from large multi-centric studies covering different species, methodologies, and languages.
Prediction of listening effort ratings for habitual and clear-Lombard speech presented in noise
Authors: Esther Janse; Chen Shen; Martin Cooke
Venue: Interspeech 2025 (main)

Abstract: Earlier research developed the high-energy glimpse proportion metric (HEGP) to capture those speech-dominant spectro-temporal regions, or glimpses, that survive energetic masking when speech is presented in noise. Whereas the HEGP metric has been shown to correlate with intelligibility of speech presented in noise, this study investigated whether HEGP predicts listening effort (LE) ratings, and if so, whether it does so differentially for habitual and clear-Lombard speech. Secondly, the study investigated whether acoustic measures of spectral balance, median F0 and F0 range, and articulation rate, explained additional variance in LE. Results showed that HEGP predicted LE ratings equally strongly across speaking styles. Beyond HEGP, wider F0 range and lower articulation rate were also associated with decreased LE, particularly in the habitual speech condition. Thus, the HEGP release-from-masking metric, complemented by speaking-style aspects, can help explain LE ratings in noise.