GenAI4ELab/papercli-papers huggingface dataset:
Fine-Tuning Text-to-Speech Diffusion Models Using Reinforcement Learning with Human Feedback
Authors: Jingyi Chen; Ju Seung Byun; Micha Elsner; Pichao Wang; Andrew Perrault
Venue: Interspeech 2025 (main)

Abstract: Diffusion models produce high-fidelity speech but are inefficient for real-time use due to long denoising steps and challenges in modeling intonation and rhythm. To improve this, we propose Diffusion Loss-Guided Policy Optimization (DLPO), an RLHF framework for TTS diffusion models. DLPO integrates the original training loss into the reward function, preserving generative capabilities while reducing inefficiencies. Using naturalness scores as feedback, DLPO aligns reward optimization with the diffusion model's structure, improving speech quality. We evaluate DLPO on WaveGrad 2, a non-autoregressive diffusion-based TTS model. Results show significant improvements in objective metrics (UTMOS 3.65, NISQA 4.02) and subjective evaluations, with DLPO audio preferred 67% of the time. These findings demonstrate DLPO's potential for efficient, high-quality diffusion TTS in real-time, resource-limited settings.
Accelerating Diffusion-based Text-to-Speech Model Trainingwith Dual Modality Alignment
Authors: Jeongsoo Choi; Zhikang Niu; Ji-Hoon Kim; Chunhui Wang; Joon Son Chung; Xie Chen
Venue: Interspeech 2025 (main)

Abstract: The goal of this paper is to optimize the training process of diffusion-based text-to-speech models. While recent studies have achieved remarkable advancements, their training demands substantial time and computational costs, largely due to the implicit guidance of diffusion models in learning complex intermediate representations. To address this, we propose A-DMA, an effective strategy for Accelerating training with Dual Modality Alignment. Our method introduces a novel alignment pipeline leveraging both text and speech modalities: text-guided alignment, which incorporates contextual representations, and speech-guided alignment, which refines semantic representations. By aligning hidden states with discriminative features, our training scheme reduces the reliance on diffusion models for learning complex representations. Extensive experiments demonstrate that A-DMA doubles the convergence speed while achieving superior performance over baselines.
SpeechSEC: A Unified Multi-Task Framework for Speech Synthesis, Editing, and Continuation
Authors: Liming Liang; Dongchao Yang; Xianwei Zhuang; Yuxin Xie; Luo Chen; Yuehan Jin; Yuexian Zou
Venue: Interspeech 2025 (main)

Abstract: Recent advancements in non-autoregressive single-task speech synthesis have garnered significant attention. However,traditional single-task speech synthesis methods focus primarily on mapping semantic tokens to acoustic tokens, which overlooking the internal relationships within acoustic features. Addressing this gap, we propose SpeechSEC, a unified multi-task framework designed for Speech Synthesis, Editing, and Continuation tasks by dynamically adjusting input conditions. SpeechSEC not only surpasses previous state-of-the-art method in audio quality (4.20 vs 4.00), and voice preservation (0.72 vs 0.58) for synthesis task by acquiring shared knowledge, but also efficiently executes editing and continuation tasks with good performance via non-autoregressive techniques. Additionally, SpeechSEC exhibits a strong adaptability to current speech discretization methods, like Hubert, Descript-Audio-Codec and SpeechTokenizer, which showcases robustness of our approach. Audio samples are available.
VoiceNoNG: Robust High-Quality Speech Editing Model without Hallucinations
Authors: Sung-Feng Huang; Heng-Cheng Kuo; Zhehuai Chen; Xuesong Yang; Pin-Jui Ku; Ante Jukic; Huck Yang; Yu Tsao; Yu-Chiang Frank Wang; Hung-yi Lee; Szu-Wei Fu
Venue: Interspeech 2025 (main)

Abstract: Voicebox and VoiceCraft are the current most representative models for non-autoregressive and autoregressive speech editing, respectively. Although both of them can generate high-quality speech edits, we identify their limitations: Voicebox is not good at editing speech with background audio, while VoiceCraft suffers from the hallucination-like problem. To maintain speech quality for varying audio scenarios and address the hallucination issue, we introduce VoiceNoNG, which combines the strengths of both model frameworks. VoiceNoNG utilizes a latent flow-matching framework to model the pre-quantization features of a neural codec. The vector quantizer in the neural codec provides additional robustness against minor prediction errors from the editing model, which enables VoiceNoNG to achieve state-of-the-art performance in both objective and subjective evaluations under diverse audio conditions. Audio examples and ablations can be found on the demo page1 and supplementary material2.
A Watermark for Auto-Regressive Speech Generation Models
Authors: Yihan Wu; Ruibo Chen; Georgios Milis; Junfeng Guo; Heng Huang
Venue: Interspeech 2025 (main)

Abstract: As auto-regressive speech generation models have recently demonstrated exceptional ability in producing realistic and contextually appropriate spoken language, concerns about their potential misuse also grow. To mitigate these risks, our research introduces a pioneering statistical watermarking framework tailored for auto-regressive speech generation models. This framework integrates the statistical algorithms from existing watermark techniques to ensure that the audio outputs remain traceable and accountable without compromising audio quality. Besides, we identify the re-encoded mismatch, a significant hurdle in maintaining detection accuracy when audio outputs are re-encoded for verification. Through comprehensive experiments, we valid the detectability of our watermark and provide a detailed examination of how the re-encoded mismatch impacts watermark detection efficiency.
Speech Kinematic Analysis from Acoustics: Scientific, Clinical and Practical Applications
Authors: Carol Espy-Wilson
Venue: Interspeech 2025 (main)

Abstract: AbstractMuch of my research has involved studying how small changes in the spatiotemporal coordination of speech articulators affect variability in the acoustic characteristics of the speech signal. This interest in speech variability ultimately led me to develop a speech inversion (SI) system that recovers articulatory movements of the lips, tongue tip, and tongue body from the speech signal. Recently, we were able to extend the SI system to provide information about the velopharyngeal port opening (nasality) and will soon investigate a methodology to uncover information about the tongue root and the size of the glottal opening. Our SI system has proven to be speaker independent and generalizes well across acoustic databases. In this talk, I will explain how we developed the SI system, and ways in which we have used it to date: for clinical purposes in mental health and speech disorder assessment, in scientific analysis of cross-linguistic speech patterns, and for improving automatic speech recognition.BiographyCarol Espy-Wilson is a full professor in the Electrical and Computer Engineering Department and the Institute for Systems Research at the University of Maryland College Park. She received her BS in electrical engineering from Stanford University and her MS, EE and PhD degrees in electrical engineering from the Massachusetts Institute of Technology. Dr. Espy-Wilson is a Fellow of the Acoustical Society of America (ASA), the International Speech Communication Association (ISCA) and the IEEE. She was recently elected VP-elect of ASA, and to the ISCA Advisory Board. She is currently serving on the Editorial Board of Computer, Speech and Language. She has been Chair of the Speech Communication Technical Committee of ASA, elected member of the Speech and Language Technical Committee of IEEE and Associate Editor of the Journal of the Acoustical Society of America. Finally, at the National Institutes of Health, she has served on the Advisory Councils for the National Institute on Deafness and Communication Disorders and the National Institutes of Biomedical Imaging and Bioengineering, on the Medical Rehabilitation Advisory Board of the National Institute of Child Health and Human Development, and she has been a member of the Language and Communication Study Section.
Carol directs the Speech Communication Lab where they combine digital signal processing, speech science, linguistics and machine learning to conduct research in speech communication. Current research projects include speech inversion, mental health assessment based on speech, video and text, speech recognition for elementary school classrooms, entrainment based on articulatory and facial gestures in unstructured conversations between neurotypical and neurodiverse participants, and speech enhancement. Her laboratory has received federal funding (NSF, NIH and DoD) and industry grants and she has 13 patents.
Influence of wall coverings of 3D-printed vocal tract models on measured transfer functions
Authors: Peter Birkholz; Dominik SchÃ¤fer; Patrick HÃ¤sner; Jihyeon Yun; Iris Kruppke; RÃ©mi Blandin
Venue: Interspeech 2025 (main)

Abstract: Physical replicas of the vocal tract are often produced by 3D-printing. A proven method to obtain the acoustic transfer function of these models is based on the principle of reciprocity, where an external sound source is used and the acoustic response is measured inside the model at the glottal end. However, earlier investigations showed that for some model geometries the measured transfer functions deviated from the expected resonance pattern due to spurious poles and zeros. These artifacts are probably caused by sound transmission through the model walls, or by whole-body vibrations of the replicas. In this empirical study, two methods to reduce these artifacts were compared, namely wrapping the models with sound-absorbing fabric and embedding the models in sand. Axisymmetric 3D-printed tubes for ten different vowels were analyzed. The results showed that both methods reduce artifacts and at the same time improve the repeatability of the measurements.
Supralaryngeal Kinematics of Implosives in Central Vietnamese: An EMA Study
Authors: Paul McGuire; Kye Shibata; Thanh Viet Cao; Feng-fan Hsieh; Yueh-chin Chang
Venue: Interspeech 2025 (main)

Abstract: This study, using electromagnetic articulography (EMA), examines the supralaryngeal kinematics of bilabial voiced implosives in Central Vietnamese, focusing on lip aperture trajectories and gestural timing during stop formation and release. Results indicate that implosives exhibited a higher peak velocity away from closure, whereas their voiceless plosive counterparts were found to have a longer gestural plateau. To investigate whether these differences are inherently tied to implosivity or simply a consequence of voicing, we analysed voiced and voiceless bilabial plosives in Taiwanese Southern Min. The Southern Min data confirmed that voiced plosives did not exhibit the same rapid lip aperture movement, supporting the conclusion that, in the Central Vietnamese data, higher peak velocity away from the stop closure is a defining supralaryngeal feature of implosives.
Lateral Channel Formation in Australian English /l/: Insights from Magnetic Resonance Imaging
Authors: TÃ¼nde Szalay; Michael Proctor; Amelia Gully; Tharinda Piyadasa; Craig Jin; David Waddington; Naeim Sanaei; Sheryl Foster; Kirrie Ballard
Venue: Interspeech 2025 (main)

Abstract: Goals of lateral approximant production are imperfectly understood, partly because of the limitations of most existing data, restricted to the midsagittal plane. To provide more complete information about the configuration of the vocal tract for laterals, /l/-production in three vowel contexts by three Australian English speakers was examined for the first time using a combination of real-time and volumetric Magnetic Resonance Imaging (MRI). Laterals produced by all speakers were characterised by bilateral parasagittal airflow, with some asymmetries in side channel geometries. Central occlusion of the oral airway varied in location, timing, and duration across speakers and vowel contexts, but was consistently associated with reduction in overall acoustic intensity relative to context vowels. These data provide further insights into the complex relationships between articulatory, coarticulatory and acoustic properties of lateral approximants, and their realisation in Australian English.
Articulatory variations in Apical Vowels in Southwestern Mandarin
Authors: Jing Huang; Feng-fan Hsieh; Yueh-chin Chang
Venue: Interspeech 2025 (main)

Abstract: This study examines the acoustic and articulatory characteristics of apical vowels in Southwestern Mandarin (SWM) using electromagnetic articulography (EMA). Data were collected from seven speakers in their twenties, focusing apical vowels, the high vowel [i], and their er-suffixed forms. Two distinct patterns emerged: Type A speakers demonstrated differential tongue configurations for apical vowels (si with dental/alveolar position, shi with alveolo-palatal position), while Type B speakers showed similar configurations for both vowels. Type A findings align with previous research on Standard Chinese, confirming that apical vowels share place features with preceding sibilants and showing higher F2 values for shi. However, Type B revealed an intriguing articulatoryacoustic mismatch: despite showing significant differences in horizontal Tongue Tip and Tongue Body positions, these vowels exhibited no significant formant differences. This unexpected pattern suggests a more complex relationship between articulatory configuration and acoustic output in apical vowel production than previously documented.
Rhotic Articulation in Australian English: Insights from MRI
Authors: Michael Proctor; TÃ¼nde Szalay; Tharinda Piyadasa; Craig Jin; Naeim Sanaei; Amelia Gully; David Waddington; Sheryl Foster; Kirrie Ballard
Venue: Interspeech 2025 (main)

Abstract: English rhotics are realized with rich allophony across speakers, contexts and varieties, but Australian English /Ã´/ has not previously been examined in detail. Rhotic approximants produced in three vowel contexts by four speakers of Australian English were captured using real-time and volumetric structural magnetic resonance imaging. /Ã´/ was articulated with bunched tongue postures by two speakers and more apical configurations by two speakers, but all rhotics were characterized by three coordinated gestures: tongue tip, tongue body and labial constrictions. These data shed new light on the complex goals of production of rhotic approximants beyond the midsagittal plane, and their realization and extent of variation in Australian English.
Articulatory Strategy in Vowel Production as a Basis for Speaker Discrimination
Authors: Justin J. H. Lo; Patrycja Strycharczuk; Sam Kirkham
Venue: Interspeech 2025 (main)

Abstract: The way speakers articulate is well known to be variable across individuals while at the same time subject to anatomical and biomechanical constraints. In this study, we ask whether articulatory strategy in vowel production can be sufficiently speaker-specific to form the basis for speaker discrimination. We conducted Generalised Procrustes Analyses of tongue shape data from 40 English speakers from the North West of England, and assessed the speaker-discriminatory potential of orthogonal tongue shape features within the framework of likelihood ratios. Tongue size emerged as the individual dimension with the strongest discriminatory power, while tongue shape variation in the more anterior part of the tongue generally outperformed tongue shape variation in the posterior part. When considered in combination, shape-only information may offer comparable levels of speaker specificity to size-and-shape information, but only when features do not exhibit speaker-level co-variation.
PAST: Phonetic-Acoustic Speech Tokenizer
Authors: Nadav Har-Tuv; Or Tal; Yossi Adi
Venue: Interspeech 2025 (main)

Abstract: We present PAST, a novel end-to-end framework that jointly models phonetic information alongside signal reconstruction, eliminating the need for external pretrained models. Unlike previous approaches that rely on pretrained self-supervised models, PAST employs supervised phonetic data, directly integrating domain knowledge into the tokenization process via auxiliary tasks. Additionally, we introduce a streamable, causal variant of PAST, enabling real-time speech applications. Results demonstrate that PAST surpasses existing evaluated baseline tokenizers across common evaluation metrics, including phonetic representation and speech reconstruction. Notably, PAST also achieves superior performance when serving as a speech representation for speech language models, further highlighting its effectiveness as a foundation for spoken language generation. To foster further research, we release the full implementation.
Factorized RVQ-GAN For Disentangled Speech Tokenization
Authors: Sameer Khurana; Dominik Klement; Antoine Laurent; Dominik BoboÅ¡; Juraj Novosad; Peter Gazdik; Ellen Zhang; Zili Huang; Amir Hussein; Ricard Marxer; Yoshiki Masuyama; Ryo Aihara; Chiori Hori; FranÃ§ois G. Germain; Gordon Wichern; Jonathan Le Roux
Venue: Interspeech 2025 (main)

Abstract: We propose Hierarchical Audio Codec (HAC), a unified neural speech codec that factorizes its bottleneck into three linguistic levelsâacoustic, phonetic, and lexicalâwithin a single model. HAC leverages two knowledge distillation objectives: one from a pre-trained speech encoder (HuBERT) for phoneme-level structure, and another from a text-based encoder (LaBSE) for lexical cues. Experiments on English and multilingual data show that HACâs factorized bottleneck yields disentangled token sets: one aligns with phonemes, while another captures word-level semantics. Quantitative evaluations confirm that HAC tokens preserve naturalness and provide interpretable linguistic information, outperforming single-level baselines in both disentanglement and reconstruction quality. These findings underscore HACâs potential as a unified discrete speech representation, bridging acoustic detail and lexical meaning for downstream speech generation and understanding tasks.
EnCodecMAE: leveraging neural codecs for universal audio representation learning
Authors: Leonardo Pepino; Pablo Riera; Luciana Ferrer
Venue: Interspeech 2025 (main)

Abstract: Universal audio representation learning aims to obtain foundational models that are useful for diverse tasks involving speech, music and environmental sounds. To achieve this, often methods inspired by self-supervised models from NLP (e.g., BERT), and vision, like masked autoencoders (MAE), are adapted to the audio domain. In this work, we explore the use of EnCodec, a neural audio codec, to generate discrete targets for a MAE-based universal audio model. We evaluate our approach, EnCodecMAE, across various tasks and find that, on average, it outperforms state-of-the-art audio representation models. Moreover, we analyze the impact of several factors on downstream performance, concluding that increasing model size leads to performance improvements, that the optimal input representation depends on the type of task, that self-training is benefical and that diversity in the training dataset is essential to achieve good performance across different audio tasks.
AxLSTMs: learning self-supervised audio representations with xLSTMs
Authors: Sarthak Yadav; Sergios Theodoridis; Zheng-Hua Tan
Venue: Interspeech 2025 (main)

Abstract: While the transformer has emerged as the eminent neural architecture, several independent lines of research have emerged to address its limitations. Recurrent neural approaches have observed a lot of renewed interest, including the extended long short-term memory (xLSTM) architecture, which reinvigorates the original LSTM. However, while xLSTMs have shown competitive performance compared to the transformer, their viability for learning self-supervised general-purpose audio representations has not been evaluated. This work proposes Audio xLSTM (AxLSTM), an approach for learning audio representations from masked spectrogram patches in a self-supervised setting. Pretrained on the AudioSet dataset, the proposed AxLSTM models outperform comparable self-supervised audio spectrogram transformer (SSAST) baselines by up to 25% in relative performance across a set of ten diverse downstream tasks while having up to 45% fewer parameters.
Real-time TSE demonstration via SoundBeam with KD
Authors: Keigo Wakayama; Tomoko Kawase; Takafumi Moriya; Marc Delcroix; Hiroshi Sato; Tsubasa Ochiai; Masahiro Yasuda; Shoko Araki
Venue: Interspeech 2025 (main)

Abstract: The objective of target sound extraction (TSE) is to extract sound sources of a specified class from mixed signals. Research into TSE has been actively conducted with the aim of applying it to immersive systems and auditory devices. We propose to demonstrate a real-time TSE system, which can isolate the signal from a desired sound class from sound mixtures recorded on the fly. This demonstration is based on the recently proposed causal SoundBeam model, which is trained using knowledge distillation (KD) from a non-causal TSE system. Experiments have demonstrated that SoundBeam with KD exhibits superior extraction accuracy compared to a state-of-the-art (SOTA) TSE, i.e., Waveformer. This paper explains the implementation of the proposed real-time TSE demonstration system. It is noteworthy that this demonstration will show for the first time at Interspeech, the ability to extract sound signals of a selected sound event (SE) class in real-time on a laptop.
Real-Time Diffusion Buffer for Speech Enhancement On A Laptop
Authors: Bunlong Lay; Rostilav Makarov; Timo Gerkmann
Venue: Interspeech 2025 (main)

Abstract: We demonstrate a real-time, diffusion-based speech enhancement system capable of removing background noise in a reverberant environment using a single microphone. Building on recent advances in diffusion models for speech enhancement, the proposed system achieves sub-second latency while preserving speech quality in challenging conference scenarios, e.g. loud babble noise. Attendees will personally experience the difference between turning on/off the speech enhancement system and regulating the quality/latency trade off. This demonstration showcases, for the first time, an online, diffusion-based enhancement framework suitable for live events.
Co-Speech Motion for Virtual Agents in Dialogue Using LLM-Driven Primitive Action Selection
Authors: Muhammad Yeza Baihaqi; Angel GarcÃ­a Contreras; Seiya Kawano; Koichiro Yoshino
Venue: Interspeech 2025 (main)

Abstract: Non-verbal behaviors, such as co-speech motion, are essential for making artificial agents more lifelike and engaging. However, existing approaches to generating co-speech motions still face significant challenges. Rule-based systems, while capable of producing natural and engaging motions, struggle with generalizability. In contrast, the common data-driven methods can generate a wide variety of gestures but come with high costs and require substantial adaptation to be applied to agents. Leveraging the power of Large Language Models (LLMs) for contextual planning and understanding, we propose an LLM-based motion control model that uses a primitive action selection strategy. This approach is expected to provide a more flexible and scalable solution for generating contextually appropriate co-speech motions across various embodied systems, including virtual agents and robots.