GenAI4ELab/papercli-papers huggingface dataset:
A Perception-Based L2 Speech Intelligibility Indicator: Leveraging a Raterâs Shadowing and Sequence-to-sequence Voice Conversion
Authors: Haopeng Geng; Daisuke Saito; Nobuaki Minematsu
Venue: Interspeech 2025 (main)

Abstract: Evaluating L2 speech intelligibility is crucial for effective computer-assisted language learning (CALL). Conventional ASR-based methods often focus on native-likeness, which may fail to capture the actual intelligibility perceived by human listeners. In contrast, our work introduces a novel, perception-based L2 speech intelligibility indicator that leverages a native raterâs shadowing data within a sequence-to-sequence (seq2seq) voice conversion framework. By integrating an alignment mechanism and acoustic feature reconstruction, our approach simulates the auditory perception of native listeners, identifying segments in L2 speech that are likely to cause comprehension difficulties. Both objective and subjective evaluations indicate that our method aligns more closely with native judgments than traditional ASR-based metrics, offering a promising new direction for CALL systems in a global, multilingual contexts.
Multimodal and Multitask Learning for Predicting Multiple Scores in L2 English Speech
Authors: Sehyun Oh; Sunhee Kim; Minhwa Chung
Venue: Interspeech 2025 (main)

Abstract: This study presents a novel multimodal and multitask learning model for predicting five proficiency scores of L2 English speeches. The proposed approach integrates speech and text embeddings using multimodal transformer blocks with cross-modal attention to refine features dynamically between modalities, capturing complementary information. A joint loss function, combining MSE and a Trait-Aware (TA) loss, enhances the model by leveraging relationships among proficiency traits. Experiments with different combinations of four embeddings (MFCCs, GloVe, wav2vec 2.0, and BERT) revealed that the proposed model with wav2vec 2.0 and BERT embeddings achieved the best performance, with a mean PCC of 0.734 and a standard deviation of 0.0129 across five criteria. This approach significantly outperforms unimodal and baseline multimodal models, demonstrating the potential of advanced multimodal architectures and task-aware optimization in automated speech assessment systems.
Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving
Authors: Jingran Xie; Xiang Li; Hui Wang; Yue Yu; Yang Xiang; Xixin Wu; Zhiyong Wu
Venue: Interspeech 2025 (main)

Abstract: Large language models (LLMs) have shown remarkable generalization across tasks, leading to increased interest in integrating speech with LLMs. These speech LLMs (SLLMs) typically use supervised fine-tuning to align speech with text-based LLMs. However, the lack of annotated speech data across a wide range of tasks hinders alignment efficiency, resulting in poor generalization. To address these issues, we propose a novel multi-task 'behavior imitation' method with speech-text interleaving, called MTBI, which relies solely on paired speech and transcripts. By ensuring the LLM decoder generates equivalent responses to paired speech and text, we achieve a more generalized SLLM. Interleaving is used to further enhance alignment efficiency. We introduce a simple benchmark to evaluate prompt and task generalization across different models. Experimental results demonstrate that our MTBI outperforms state-of-the-art SLLMs on both prompt and task generalization, while requiring less supervised speech data.
Mispronunciation Detection Without L2 Pronunciation Dataset in Low-Resource Setting: A Case Study in Finland Swedish
Authors: Nhan Phan; Mikko Kuronen; Maria Kautonen; Riikka Ullakonoja; Anna von Zansen; Yaroslav Getman; Ekaterina Voskoboinik; TamÃ¡s GrÃ³sz; Mikko Kurimo
Venue: Interspeech 2025 (main)

Abstract: Mispronunciation detection (MD) models are the cornerstones of many language learning applications. Unfortunately, most systems are built for English and other major languages, while low-resourced language varieties, such as Finland Swedish (FS), lack such tools. In this paper, we introduce our MD model for FS, trained on 89 hours of first language (L1) speakers' spontaneous speech and tested on 33 minutes of L2 transcribed read-aloud speech. We trained a multilingual wav2vec 2.0 model with entropy regularization, followed by temperature scaling and top-k normalization after the inference to better adapt it for MD. The main novelty of our method lies in its simplicity, requiring minimal L2 data. The process is also language-independent, making it suitable for other low-resource languages. Our proposed algorithm allows us to balance Recall (43.2%) and Precision (29.8%), compared with the baseline model's Recall (77.5%) and Precision (17.6%).
RapFlow-TTS: Rapid and High-Fidelity Text-to-Speech with Improved Consistency Flow Matching
Authors: Hyun Joon Park; Jeongmin Liu; Jin Sob Kim; Jeong Yeol Yang; Sung Won Han; Eunwoo Song
Venue: Interspeech 2025 (main)

Abstract: We introduce RapFlow-TTS, a rapid and high-fidelity TTS acoustic model that leverages velocity consistency constraints in flow matching (FM) training. Although ordinary differential equation (ODE)-based TTS generation achieves natural-quality speech, it typically requires a large number of generation steps, resulting in a trade-off between quality and inference speed. To address this challenge, RapFlow-TTS enforces consistency in the velocity field along the FM-straightened ODE trajectory, enabling consistent synthetic quality with fewer generation steps. Additionally, we introduce techniques such as time interval scheduling and adversarial learning to further enhance the quality of the few-step synthesis. Experimental results show that RapFlow-TTS achieves high-fidelity speech synthesis with a 5- and 10-fold reduction in synthesis steps than the conventional FM- and score-based approaches, respectively.
Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling
Authors: Qixi Zheng; Yushen Chen; Zhikang Niu; Ziyang Ma; Xiaofei Wang; Kai Yu; Xie Chen
Venue: Interspeech 2025 (main)

Abstract: Flow-matching-based text-to-speech (TTS) models, such as Voicebox, E2 TTS, and F5-TTS, have attracted significant attention in recent years. These models require multiple sampling steps to reconstruct speech from noise, making inference speed a key challenge. Reducing the number of sampling steps can greatly improve inference efficiency. To this end, we introduce Fast F5-TTS, a training-free approach to accelerate the inference of flow-matching-based TTS models. By inspecting the sampling trajectory of F5-TTS, we identify redundant steps and propose Empirically Pruned Step Sampling (EPSS), a non-uniform time-step sampling strategy that effectively reduces the number of sampling steps. Our approach achieves a 7-step generation with an inference RTF of 0.030 on an NVIDIA RTX 3090 GPU, making it 4 times faster than the original F5-TTS while maintaining comparable performance. Furthermore, EPSS performs well on E2 TTS models, demonstrating its strong generalization ability.
Differentiable Reward Optimization for LLM based TTS system
Authors: Changfeng Gao; Zhihao Du; Shiliang Zhang
Venue: Interspeech 2025 (main)

Abstract: This paper proposes a novel Differentiable Reward Optimization (DiffRO) method aimed at enhancing the performance of neural codec language models based text-to-speech (TTS) systems. In contrast to conventional reinforcement learning from human feedback (RLHF) approaches applied to TTS, DiffRO directly compute the rewards based on neural codec tokens, rather than relying on synthesized audio. Furthermore, we employ the Gumbel-Softmax technique to render the reward function differentiable, thereby streamlining the RLHF training process. Additionally, we introduce a multi-task reward (MTR) model which can provide feedback from different perspectives and find that it can augment the system's capability to follow instructions effectively. Experimental results indicate that DiffRO significantly improves the pronunciation accuracy of the TTS system, achieving state-of-the-art (SOTA) WER results on the seed-tts-eval benchmark. Moreover, with the integration of the MTR model, we demonstrate the ability to control emotional and quality attributes in a zero-shot manner.
Long-Context Speech Synthesis with Context-Aware Memory
Authors: Zhipeng Li; Xiaofen Xing; Jingyuan Xing; Hangrui Hu; Heng Lu; Xiangmin Xu
Venue: Interspeech 2025 (main)

Abstract: In long-text speech synthesis, current approaches typically convert text to speech at the sentence-level and concatenate the results to form pseudo-paragraph-level speech. These methods overlook the contextual coherence of paragraphs, leading to reduced naturalness and inconsistencies in style and timbre across the long-form speech. To address these issues, we propose a Context-Aware Memory (CAM)-based long-context Text-to-Speech (TTS) model. The CAM block integrates and retrieves both long-term memory and local context details, enabling dynamic memory updates and transfers within long paragraphs to guide sentence-level speech synthesis. Furthermore, the prefix mask enhances the in-context learning ability by enabling bidirectional attention on prefix tokens while maintaining unidirectional generation. Experimental results demonstrate that the proposed method outperforms baseline and state-of-the-art long-context methods in terms of prosody expressiveness, coherence and context inference cost across paragraph-level speech.
Monotonic Attention for Robust Text-to-Speech Synthesis in Large Language Model Frameworks
Authors: Yike Zhang; Yiming Li; Jie Chen; Qinghua Wu; Songjun Cao; Long Ma
Venue: Interspeech 2025 (main)

Abstract: Text-to-speech (TTS) synthesis using large language models (LLMs) has demonstrated promising performance and has recently garnered significant attention. Despite their impressive naturalness, these methods often lack monotonic alignment constraints, resulting in issues such as repetition, omissions, and misalignment in the synthesized output. This paper introduces a stepwise monotonic attention algorithm tailored for LLM-based architectures, to enhance the robustness of TTS synthesis and effectively address these issues. Compared with the best existing model, VALL-E R, the proposed approach requires no additional forced aligners and exhibits greater robustness on out-of-domain test sets. Furthermore, experiments show that the proposed method scales well to large model sizes and large-scale training sets.
Improving Noise Robustness of LLM-based Zero-shot TTS via Discrete Acoustic Token Denoising
Authors: Ye-Xin Lu; Hui-Peng Du; Fei Liu; Yang Ai; Zhen-Hua Ling
Venue: Interspeech 2025 (main)

Abstract: Large language model (LLM) based zero-shot text-to-speech (TTS) methods tend to preserve the acoustic environment of the audio prompt, leading to degradation in synthesized speech quality when the audio prompt contains noise. In this paper, we propose a novel neural codec-based speech denoiser and integrate it with the advanced LLM-based TTS model, LauraTTS, to achieve noise-robust zero-shot TTS. The proposed codec denoiser consists of an audio codec, a token denoiser, and an embedding refiner. The token denoiser predicts the first two groups of clean acoustic tokens from the noisy ones, which can serve as the acoustic prompt for LauraTTS to synthesize high-quality personalized speech or be converted to clean speech waveforms through the embedding refiner and codec decoder. Experimental results show that our proposed codec denoiser outperforms state-of-the-art speech enhancement (SE) methods, and the proposed noise-robust LauraTTS surpasses the approach using additional SE models.
Bridging the TrainingâInference Gap in TTS: Training Strategies for Robust Generative Postprocessing for Low-Resource Speakers
Authors: Frank Zalkow; Paolo Sani; Kishor Kayyar Lakshminarayana; EmanuÃ«l A. P. Habets; Nicola Pia; Christian Dittmar
Venue: Interspeech 2025 (main)

Abstract: Modern text-to-speech synthesis systems usually consist of an acoustic model generating speech features, e.g., mel spectrograms, and a vocoder converting them into speech waveforms. The vocoder is typically trained with ground-truth features but receives features from the acoustic model during inference, leading to a mismatch between training and inference. To address this issue, previous work proposed employing generative postprocessing models to make the synthetic features appear more natural. While such systems can produce speech nearly indistinguishable from real speech when sufficient training data is available, their performance degrades with limited data. To mitigate this limitation, we propose a training data generation procedure using a subsampling strategy and multiple acoustic models. We evaluate it through listening tests, demonstrating consistent improvements in the naturalness of the synthetic speech across different postprocessing models and low-resource target speakers.
Robust Neural Codec Language Modeling with Phoneme Position Prediction for Zero-Shot TTS
Authors: Chunhui Lu; Xue Wen; Liming Song; Junkwang Oh
Venue: Interspeech 2025 (main)

Abstract: Though large language models (LLMs) based methods have achieved remarkable progress in zero-shot text-to-speech (TTS) synthesis, they suffer from robustness issues including mispronunciation, word skipping, and word repeating. To address these robustness challenges, we propose incorporating phoneme position prediction into the LLM-based TTS model. More concretely, given an input phoneme sequence as condition, our model autoregressively predicts acoustic codes and their corresponding phoneme positions within the input sequence synchronously. This mechanism ensures accurate and complete alignment between acoustic codes and phonemes. Experimental results demonstrate that our system significantly reduces phoneme skipping/repetition errors compared to strong baselines, achieving a 52.7% relative reduction in character error rate while maintaining comparable performance in zero-shot TTS evaluations.
SepVAC: Multitask Learning of Speaker Separation, Speaker Localization, Microphone Array Localization, and Room Acoustic Parameter Estimation in Various Acoustic Conditions
Authors: Roland Hartanto; Sakriani Sakti; Koichi Shinoda
Venue: Interspeech 2025 (main)

Abstract: This paper proposes a multitask learning method for speech separation, that Separates speech and estimates the recording conditions in Various Acoustic Conditions (SepVAC) jointly. Unlike the previous methods that aim to achieve robustness against the uncertainty caused by noise and reverberation, this method explicitly estimates speaker & microphone location and room acoustic parameters to disambiguate them from speech features. We introduce curriculum learning to learn the model parameters stably. In our evaluation using SMS-WSJ-Plus dataset, it outperforms the state-of-the-art SpatialNet baseline by 0.67 points in word error rate (WER).
TA-RIR: Topology-Aware Neural Modeling of Acoustic Propagation for Room Impulse Response Synthesis
Authors: Junhui Zhao; Hang Chen; Qing Wang; Jun Du; Yanhui Tu; Feng Ma
Venue: Interspeech 2025 (main)

Abstract: Accurate estimation of room impulse responses (RIRs) is crucial for applications like augmented reality and sound field modeling. Current methods either neglect the spatial relationships between the source and receiver or rely on computationally intensive volumetric grids or panoramic images to estimate RIRs. To address these challenges, we introduce TA-RIR, a topology-aware neural network that uses spatial coordinates of sources and receivers, along with reverberant speech, to learn compact embeddings encoding room geometry and acoustics. The topology-aware encoder captures structural relationships between spatial and acoustic features, integrated through a propagation-informed decoder to synthesize RIRs. Experimental results show that TA-RIR generates high-fidelity RIRs, accurately preserving target acoustic parameters such as reverberation time, while significantly reducing computational complexity compared to methods requiring detailed 3D models or room acoustic properties.
Spatially Weighted Contrastive Learning for Robust Sound Source Localization
Authors: Hyun-Soo Kim; Da-Hee Yang; Joon-Hyuk Chang
Venue: Interspeech 2025 (main)

Abstract: We propose a spatially weighted contrastive loss (SWeC loss) for sound source localization in real-world scenarios using multi-channel speech data. In multi-channel localization, phase differences between microphone channels provide critical cues for estimating the azimuth angle of incoming speech. To effectively extract azimuth information, we leverage contrastive learning and introduce a novel loss function that incorporates spatial relationships between azimuth classes. Specifically, our loss assigns weights to negative pairs based on their angular distance, penalizing high similarity between embeddings corresponding to distant angles. Furthermore, we propose a contrastive data generation method tailored to multi-channel localization, enhancing the effectiveness of contrastive learning. Experimental results demonstrate that the proposed loss function and data generation strategy significantly improve localization performance.
Efficient and Microphone-Fault-Tolerant 3D Sound Source Localization
Authors: Yiyuan Yang; Shitong Xu; Niki Trigoni; Andrew Markham
Venue: Interspeech 2025 (main)

Abstract: Sound source localization (SSL) is a critical technology for determining the position of sound sources in complex environments. However, existing methods face challenges such as high computational costs and precise calibration requirements, limiting their deployment in dynamic or resource-constrained environments. This paper introduces a novel 3D SSL framework, which uses sparse cross-attention, pretraining, and adaptive signal coherence metrics, to achieve accurate and computationally efficient localization with fewer input microphones. The framework is also fault-tolerant to unreliable or even unknown microphone position inputs, ensuring its applicability in real-world scenarios. Preliminary experiments demonstrate its scalability for multi-source localization without requiring additional hardware. This work advances SSL by balancing the model's performance and efficiency and improving its robustness for real-world scenarios.
Joint Reference Microphone Selection and Filter Order Determination in Multi-channel Active Noise Control
Authors: De Hu; Shuyao Liu; Yanrong He
Venue: Interspeech 2025 (main)

Abstract: Multi-channel active noise control (MCANC) systems have been widely investigated for acoustic noise attenuation over a spatial region. In MCANC systems, incorporating more reference microphones (RMs) can boost noise reduction performance but increases computational complexity, which may affect real-time processing capability. To improve computational efficiency, we propose a joint approach for RM selection and filter order (FO) determination in MCANC systems. Under the specified performance constraint, the optimal RM subset is selected by promoting the group sparsity in filter coefficients, and the optimal FO is determined by encouraging the consecutive highest-order filter coefficients to approach zero. By combining the above two criteria, we can simultaneously select the RM subset and  determine the FO. In addition, we present a re-weighting strategy to further reduce both the number of selected RMs and the FO value. Numerical simulations confirm the validity of the proposed method.
Direct-path Relative Harmonic Coefficients Detection for Multi-source Direction-of-Arrival Estimation in Reverberant Environments
Authors: Liang Tao; Maoshen Jia; Yonggang Hu
Venue: Interspeech 2025 (main)

Abstract: The identification of direct-path recordings for direction-of-arrival (DOA) estimation has been recognized as a crucial factor in enhancing the localization robustness under reverberant environments, especially for the challenging cases involving multiple acoustic sources. In particular, the recently introduced spatial feature denoted relative harmonic coefficients (RHC) has been extensively investigated for DOA estimation due to a strong association with spatial cues. Building upon this, this paper presents a multi-source localization method that utilizes higher-order RHC vector to detect direct-path dominant (DPD) bins in the time-frequency (TF) domain. Specifically, DPD bins are effectively identified by applying an orthogonal projection matrix (OPM) to the RHC vector. Moreover, we integrate the OPM with a closed-form single source DOA estimator, thus circumventing the exhaustive search over a dense spatial grid. Extensive experimental evaluations using both simulated and real-world recordings confirm the proposed algorithmâs effectiveness, and superiority over baseline methods.
D-GAT: Dual Graph Attention Network for Global HRTF Interpolation
Authors: Junsheng Hu; Shaojie Li; Qintuya Si; De Hu
Venue: Interspeech 2025 (main)

Abstract: To achieve 3D audio rendering, high-quality Head-Related Transfer Functions (HRTFs) are essential. As measuring HRTFs is time-consuming and tedious, spatial interpolation is often adopted to generate high-resolution HRTFs from low-resolution ones. In this paper, we propose a Dual-Graph Attention Network (D-GAT) for HRTF upsampling. Specifically, we first design a branch of GAT to learn the relationship among HRTFs from adjacent points. In addition, we introduce another branch of GAT to find a mapping from physical features (including the absolute target position and the anthropometric characteristics) to HRTFs. By combining such two GATs in a parallel architecture, the D-GAT is built. Furthermore, a dynamic edge weighting mechanism is adopted in the D-GAT, which allows the model to learn geometry relationships among vertices more flexibly. Experimental results demonstrate the efficacy of the proposed D-GAT in accurately predicting HRTFs, yielding state-of-the-art performance.
Deep learning based spatial aliasing reduction in beamforming for audio capture
Authors: Mateusz Guzik; Giulio Cengarle; Daniel Arteaga
Venue: Interspeech 2025 (main)

Abstract: Spatial aliasing affects spaced microphone arrays, causing directional ambiguity above certain frequencies, degrading spatial and spectral accuracy of beamformers. Given the limitations of conventional signal processing and the scarcity of deep learning approaches to spatial aliasing mitigation, we propose a novel approach using a U-Net architecture to predict a signal-dependent de-aliasing filter, which reduces aliasing in conventional beamforming for spatial capture. Two types of multichannel filters are considered, one which treats the channels independently and a second one that models cross-channel dependencies. The proposed approach is evaluated in two common spatial capture scenarios: stereo and first-order Ambisonics. The results indicate a very significant improvement, both objective and perceptual, with respect to conventional beamforming. This work shows the potential of deep learning to reduce aliasing in beamforming, leading to improvements in multi-microphone setups.