GenAI4ELab/papercli-papers huggingface dataset:
MultiActor-Audiobook: Zero-Shot Audiobook Generation with Faces and Voices of Multiple Speakers
Authors: Kyeongman Park; Seongho Joo; Kyomin Jung
Venue: Interspeech 2025 (main)

Abstract: We introduce MultiActor-Audiobook, a zero-shot approach for generating audiobooks that automatically produces consistent, expressive, and speaker-appropriate prosody, including intonation and emotion. Previous audiobook systems have several limitations: they require users to manually configure the speaker's prosody, read each sentence with a monotonic tone compared to voice actors, or rely on costly training. However, our MultiActor-Audiobook addresses these issues by introducing two novel processes: (1) MSP (Multimodal Speaker Persona Generation) and (2) LSI (LLM-based Script Instruction Generation). With these two processes, MultiActor-Audiobook can generate more emotionally expressive audiobooks with a consistent speaker prosody without additional training. We compare our system with commercial products, through human and MLLM evaluations, achieving competitive results. Furthermore, we demonstrate the effectiveness of MSP and LSI through ablation studies.
Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction
Authors: Wenxuan Wu; Shuai Wang; Xixin Wu; Helen Meng; Haizhou Li
Venue: Interspeech 2025 (main)

Abstract: Audio-visual target speaker extraction (AV-TSE) models primarily rely on target visual cues to isolate the target speaker's voice from others. We know that humans leverage linguistic knowledge, such as syntax and semantics, to support speech perception. Inspired by this, we explore the potential of pre-trained speech-language models (PSLMs) and pre-trained language models (PLMs) as auxiliary knowledge sources for AV-TSE. In this study, we propose incorporating the linguistic constraints from PSLMs or PLMs for the AV-TSE model as additional supervision signals. Without introducing any extra computational cost during inference, the proposed approach consistently improves speech quality and intelligibility. Furthermore, we evaluate our method in multi-language settings and visual cue-impaired scenarios and show robust performance gains.
Multimodal Silent Recognition of Phonemes Using Radar and Optopalatographic Silent Speech Interfaces
Authors: JoÃ£o Menezes; Aubin Mouras; Arne-Lukas Fietkau; Dani Kazzy; Peter Birkholz
Venue: Interspeech 2025 (main)

Abstract: Silent Speech Interfaces (SSIs) have a broad range of applications and rely on sensing techniques to measure speech-related bio-signals. To take advantage of the strengths and minimize weaknesses of different sensing techniques, multimodal recordings and pipelines have been investigated lately on SSIs. This study presents a multimodal articulatory recording combining Radar and Optopalatographic (OPG) SSIs and evaluates the performance of each data type and their combination in a phoneme recognition task. A corpus of 26 phonemes was recorded by 3 native German speakers and Support Vector Machines were used to classify the recorded multimodal data into the phonemes. Considering all speakers together, accuracies of 85.40%, 66.99% and 86.55% were obtained with Radar, OPG and the concatenation of Radar and OPG, respectively. The accuracy increase of 1.15% resulting of adding OPG to Radar data is small but statistically significant and points to the potential of such multimodal recordings.
GoP2Vec: A few shot learning for pronunciation assessment with goodness of pronunciation (GoP) based representations from an i-vector framework and augmentation
Authors: Meenakshi Sirigiaju; Chiranjeevi Yarra
Venue: Interspeech 2025 (main)

Abstract: Automatic pronunciation assessment is a critical component in computer assisted language learning. Typically, modeling pronunciation assessment tasks need labels, which are difficult to obtain as it requires expert annotators. Thus, it is essential to build an accurate model with less annotated data. In this work, an approach is proposed that considers a few speech samples using the i-vector framework. Each sample, first, is lengthened by T factor by concatenating the augmented samples of the same speech. The augmentation is obtained using time-scale modification (TSM), pitch-scale modification (PSM) and both. Next, phoneme-level goodness-of-pronunciation scores of concatenated speech are converted to a vector (GoP2Vec) with the i-vector framework. Experiments on two datasets revealed that the proposed GoP2Vec outperforms the state-of-the-art (SOTA) unsupervised methods and is on par with the SOTA supervised methods when it is used to train a simple neural model with a few samples.
Enhancing GOP in CTC-Based Mispronunciation Detection with Phonological Knowledge
Authors: Aditya Kamlesh Parikh; Cristian Tejedor-Garcia; Catia Cucchiarini; Helmer Strik
Venue: Interspeech 2025 (main)

Abstract: Computer-Assisted Pronunciation Training (CAPT) systems employ automatic measures of pronunciation quality, such as the goodness of pronunciation (GOP) metric. GOP relies on forced alignments, which are prone to labeling and segmentation errors due to acoustic variability. While alignment-free methods address these challenges, they are computationally expensive and scale poorly with phoneme sequence length and inventory size. To enhance efficiency, we introduce a substitution-aware alignment-free GOP that restricts phoneme substitutions based on phoneme clusters and common learner errors. We evaluated our GOP on two L2 English speech datasets, one with child speech, My Pronunciation Coach (MPC), and SpeechOcean762, which includes child and adult speech. We compared RPS (restricted phoneme substitutions) and UPS (unrestricted phoneme substitutions) setups within alignment-free methods, which outperformed the baseline. We discuss our results and outline avenues for future research.
Multilingual Speech Assessment Using Cross-Attention and Multitask Learning
Authors: Sehyun Oh; Minhwa Chung; Sunhee Kim
Venue: Interspeech 2025 (main)

Abstract: Automatic speech assessment plays a vital role in language learning by providing essential feedback on pronunciation, fluency, and overall speaking ability. However, developing effective multilingual speech assessment systems poses significant challenges with the complexity of modeling multiple languages and limited availability of labeled data, especially for languages other than English. In this study, we propose a multilingual speech assessment system for three languages -- English, German, and French, which are produced by Korean learners. Enhanced by cross-attention and multitask learning mechanisms, our model utilizes pre-trained models to capture both language-specific and cross-linguistic features, predicting overall speaking proficiency scores directly from raw speech audio. Experimental results demonstrate that our proposed method, especially with wav2vec 2.0, presents superior performance on both seen and unseen data compared to monolingual models.
Assessment of L2 Oral Proficiency using Speech Large Language Models
Authors: Rao Ma; Mengjie Qian; Siyuan Tang; Stefano BannÃ²; Kate M. Knill; Mark J.F. Gales
Venue: Interspeech 2025 (main)

Abstract: The growing population of L2 English speakers has increased the demand for developing automatic graders for spoken language assessment (SLA). Historically, statistical models, text encoders, and self-supervised speech models have been utilised for this task. However, cascaded systems suffer from the loss of information, while E2E graders also have limitations. With the recent advancements of multi-modal large language models (LLMs), we aim to explore their potential as L2 oral proficiency graders and overcome these issues. In this work, we compare various training strategies using regression and classification targets. Our results show that speech LLMs outperform all previous competitive baselines, achieving superior performance on two datasets. Furthermore, the trained grader demonstrates strong generalisation capabilities in the cross-part or cross-task evaluation, facilitated by the audio understanding knowledge acquired during LLM pre-training.
Scaling and Prompting for Improved End-to-End Spoken Grammatical Error Correction
Authors: Mengjie Qian; Rao Ma; Stefano BannÃ²; Kate M. Knill; Mark J.F. Gales
Venue: Interspeech 2025 (main)

Abstract: Spoken Grammatical Error Correction (SGEC) and Feedback (SGECF) are crucial for second language learners, teachers and test takers. Traditional SGEC systems rely on a cascaded pipeline consisting of an ASR, a module for disfluency detection (DD) and removal and one for GEC. With the rise of end-to-end (E2E) speech foundation models, we investigate their effectiveness in SGEC and feedback generation. This work introduces a pseudo-labelling process to address the challenge of limited labelled data, expanding the training data size from 77 hours to approximately 2500 hours, leading to improved performance. Additionally, we prompt an E2E Whisper-based SGEC model with fluent transcriptions, showing a slight improvement in SGEC performance, with more significant gains in feedback generation. Finally, we assess the impact of increasing model size, revealing that while pseudo-labelled data does not yield performance gain for a larger Whisper model, training with prompts proves beneficial.
Bidirectional Spoken-Written Text Conversion with Large Language Models
Authors: Muyeol Choi; HyunJung Choi; Yohan Lim; Jeonguk Bang; Minkyu Lee; Seonhui Kim; Seung Yun; Donghyun Kim; Minsoo Kim; SangHun Kim
Venue: Interspeech 2025 (main)

Abstract: Traditional ASR systems normalize transcriptions into spoken form for training, leading to spoken form outputs. In contrast, modern Transformer-based models directly map speech-to-text, preserving the written form. However, existing speech databases mainly contain spoken form, causing inconsistencies in recognition results. To address this, Inverse Text Normalization (ITN) is required, and dual transcription data is crucial for effective training. However, constructing such datasets is costly and time-consuming. This study proposes a data augmentation method leveraging LLMs to automatically generate dual transcription data with minimal effort. It employs iterative learning to expand the dataset through supervised and semi-supervised methods. Additionally, the bidirectional text conversion (BTC) model supports both ITN and TN within a unified framework. Experimental results demonstrate that the proposed method achieved an ERR of 13.4% and 4.7%, outperforming prior approaches.
WAKE: Watermarking Audio with Key Enrichment
Authors: Yaoxun Xu; Jianwei Yu; Hangting Chen; Zhiyong Wu; Xixin Wu; Dong Yu; Rongzhi Gu; Yi Luo
Venue: Interspeech 2025 (main)

Abstract: As deep learning advances in audio generation, challenges in audio security and copyright protection highlight the need for robust audio watermarking. Recent neural network-based methods have made progress but still face three main issues: preventing unauthorized access, decoding initial watermarks after multiple embeddings, and embedding varying lengths of watermarks. To address these issues, we propose WAKE, the first key-controllable audio watermark framework. WAKE embeds watermarks using specific keys and recovers them with corresponding keys, enhancing security by making incorrect key decoding impossible. It also resolves the overwriting issue by allowing watermark decoding after multiple embeddings and supports variable-length watermark insertion. WAKE outperforms existing models in both watermarked audio quality and watermark detection accuracy.
Defend for Self-Vocoding: A Novel Enhanced Decoder Network for Watermark Recovery
Authors: Yu-Sheng Lin; Ching-Yu Yang; Hsing-Hang Chou; Ya-Tse Wu; Bo-Hao Su; Chi-Chun Lee
Venue: Interspeech 2025 (main)

Abstract: Recent advances in voice cloning technology have raised security concerns due to its ability to generate highly realistic synthetic speech, making it challenging to detect malicious usage. Proactive watermarking approaches embed authentication information in target voices to prevent unauthorized synthesis. While existing methods show resilience against traditional preprocessing attacks, we identify a novel threat, self-vocoding, which reconstructs audio using neural vocoders, can cause severe watermark degradation but preserve high audio fidelity. To address this, we propose an enhanced decoding framework to handle self-vocoding distortions on watermarks. In addition to general vocoder distortions, we systematically categorize them into two vocoder types for further analysis. Experimental results demonstrate that our approach significantly improves watermark decoding accuracy, offering an effective defense against self-vocoding attacks.
Cross-Modal Watermarking for Authentic Audio Recovery and Tamper Localization in Synthesized Audiovisual Forgeries
Authors: Minyoung Kim; Sehwan Park; Sungmin Cha; Paul Hongsuck Seo
Venue: Interspeech 2025 (main)

Abstract: Recent advances in voice cloning and lip synchronization models have enabled Synthesized Audiovisual Forgeries (SAVFs), where both audio and visuals are manipulated to mimic a target speaker. This significantly increases the risk of misinformation by making fake content seem real. To address this issue, existing methods detect or localize manipulations but cannot recover the authentic audio that conveys the semantic content of the message. This limitation reduces their effectiveness in combating audiovisual misinformation. In this work, we introduce the task of Authentic Audio Recovery (AAR) and Tamper Localization in Audio (TLA) from SAVFs and propose a cross-modal watermarking framework to embed authentic audio into visuals before manipulation. This enables AAR, TLA, and a robust defense against misinformation. Extensive experiments demonstrate the strong performance of our method in AAR and TLA against various manipulations, including voice cloning and lip synchronization.
VoiceMark: Zero-Shot Voice Cloning-Resistant Watermarking Approach Leveraging Speaker-Specific Latents
Authors: Haiyun Li; Zhiyong Wu; Xiaofeng Xie; Jingran Xie; Yaoxun Xu; Hanyang Peng
Venue: Interspeech 2025 (main)

Abstract: Voice cloning (VC)-resistant watermarking is an emerging technique for tracing and preventing unauthorized cloning. Existing methods effectively trace traditional VC models by training them on watermarked audio but fail in zero-shot VC scenarios, where models synthesize audio from an audio prompt without training. To address this, we propose VoiceMark, the first zero-shot VC-resistant watermarking method that leverages speaker-specific latents as the watermark carrier, allowing the watermark to transfer through the zero-shot VC process into the synthesized audio. Additionally, we introduce VC-simulated augmentations and VAD-based loss to enhance robustness against distortions. Experiments on multiple zero-shot VC models demonstrate that VoiceMark achieves over 95% accuracy in watermark detection after zero-shot VC synthesis, significantly outperforming existing methods, which only reach around 50%.
A Comprehensive Real-World Assessment of Audio Watermarking Algorithms: Will They Survive Neural Codecs?
Authors: Yigitcan Ãzer; Woosung Choi; Joan SerrÃ; Mayank Kumar Singh; Wei-Hsiang Liao; Yuki Mitsufuji
Venue: Interspeech 2025 (main)

Abstract: We present the Robust Audio Watermarking Benchmark (RAW-Bench) to foster the evaluation of deep learning-based audio watermarking algorithms, establishing a standardized benchmark and allowing systematic comparisons. To simulate real-world usage, we introduce a comprehensive audio attack pipeline featuring various distortions such as compression, background noise, and reverberation and propose a diverse test dataset, including speech, environmental sounds, and music recordings. By assessing the performance of four existing watermarking algorithms on our framework, two main insights stand out: (i) neural compression techniques pose the most significant challenge, even when algorithms are trained with such compressions; and (ii) training with audio attacks generally improves robustness, although it is insufficient in some cases. Furthermore, we find that specific distortions, such as polarity inversion, time stretching, or reverb, seriously affect certain algorithms. Our contributions strengthen the robustness and perceptual assessment of audio watermarking algorithms across a wide range of applications while ensuring a fair and consistent evaluation approach.
How to Recover Long Audio Sequences Through Gradient Inversion Attack With Dynamic Segment-based Reconstruction
Authors: Xijie Zeng; Frank Rudzicz
Venue: Interspeech 2025 (main)

Abstract: Recent advancements in gradient inversion attacks have demonstrated the vulnerability of shared gradients in distributed learning systems, particularly in image and text domains. However, applying these techniques to audio data, especially longer speech sequences, remains largely unexplored. Our study introduces a novel approach that builds upon the principles of gradient inversion attacks to retrieve high-quality audio recordings from shared gradients. We propose an optimized spectrogram segmentation technique that enables extracting longer audio sequences with diverse acoustic features, without requiring complex post-processing techniques. Through this study, we overcome the limitations of previous methods that were restricted to short audio clips with simple acoustic features and limited semantic information.
First Steps Towards Voice Anonymization for Code-Switching Speech
Authors: Sarina Meyer; Ekaterina Kolos; Ngoc Thang Vu
Venue: Interspeech 2025 (main)

Abstract: The goal of voice anonymization is to modify an audio such that the true identity of its speaker is hidden. Research on this task is typically limited to the same English read speech datasets, thus the efficacy of current methods for other types of speech data remains unknown. In this paper, we present the first investigation of voice anonymization for the multilingual phenomenon of code-switching speech. We prepare two corpora for this task and propose adaptations to a multilingual anonymization model to make it applicable for code-switching speech. By testing the anonymization performance of this and two language-independent methods on the datasets, we find that only the multilingual system performs well in terms of privacy and utility preservation. Furthermore, we observe challenges in performing utility evaluations on this data because of its spontaneous character and the limited code-switching support by the multilingual speech recognition model.
Exploiting Context-dependent Duration Features for Voice Anonymization Attack Systems
Authors: Natalia Tomashenko; Emmanuel Vincent; Marc Tommasi
Venue: Interspeech 2025 (main)

Abstract: The temporal dynamics of speech, encompassing variations in rhythm, intonation, and speaking rate, contain important and unique information about speaker identity. This paper proposes a new method for representing speaker characteristics by extracting context-dependent duration embeddings from speech temporal dynamics. We develop novel attack models using these representations and analyze the potential vulnerabilities in speaker verification and voice anonymization systems. The experimental results show that the developed attack models provide a significant improvement in speaker verification performance for both original and anonymized data in comparison with more simple representations of speech temporal dynamics reported in the literature.
Mitigating Language Mismatch in SSL-Based Speaker Anonymization
Authors: Zhe Zhang; Wen-Chin Huang; Xin Wang; Xiaoxiao Miao; Junichi Yamagishi
Venue: Interspeech 2025 (main)

Abstract: Speaker anonymization aims to protect speaker identity while preserving content information and the intelligibility of speech. However, most speaker anonymization systems (SASs) are developed and evaluated using only English, resulting in degraded utility for other languages. This paper investigates language mismatch in SASs for Japanese and Mandarin speech. First, we fine-tune a self-supervised learning (SSL)-based content encoder with Japanese speech to verify effective language adaptation. Then, we propose fine-tuning a multilingual SSL model with Japanese speech and evaluating the SAS in Japanese and Mandarin. Downstream experiments show that fine-tuning an English-only SSL model with the target language enhances intelligibility while maintaining privacy and that multilingual SSL further extends SASs' utility across different languages. These findings highlight the importance of language adaptation and multilingual pre-training of SSLs for robust multilingual speaker anonymization.
MSFNet: A Nested Model for Multi-Sampling-Frequency Speech Enhancement
Authors: Venkatesh Parvathala; K. Sri Rama Murty
Venue: Interspeech 2025 (main)

Abstract: Recently, speech enhancement research has shifted towards universal models handling various input conditions under multiple degradations. While diverse training data improves robustness, handling data acquisition variations like sampling frequency (SF) requires architectural changes. This study focuses on SF dependency in networks involving fully-connected (FC) layers, as current fixed-resolution STFT-based methods are not directly compatible with FC layers. We propose a multi-SF network (MSFNet) incorporating an FC layer with the nested structure and learnable weight scaling (LWS) mechanism. LWS mitigates the distributional changes at the FC layer's output, and the nested structure allows compact submodel extraction. Our experiments demonstrate that the proposed method is computationally efficient and performs comparably to the independently trained models. Notably, an 8 kHz model has 33 times fewer parameters and 6 times fewer multiply-accumulate operations than a 48 kHz model.
TF-SkiMNet: Speech Enhancement Based on Inplace Modeling and Skipping Memory in Time-Frequency Domain
Authors: Zixuan Li; Shulin He; Jinglin Bai; Xueliang Zhang
Venue: Interspeech 2025 (main)

Abstract: Neural networks that leverage both full-band and sub-band information have demonstrated exceptional performance across various speech processing tasks. In this paper, we examine the essential factors that allow these architectures to achieve state-of-the-art (SOTA) performance, identifying their 'inplace modeling' capability as a critical component of this success. Adhering to this principle, we introduce TF-SkiMNet, which employs Skipping Memory (SkiM) to efficiently perform global temporal modeling using full-band information with low computational overhead. For single-channel speech enhancement, TF-SkiMNet achieves comparable performance to the SOTA model TF-CrossNet while reducing MACs by 87%. Furthermore, TF-SkiMNet is evaluated on both single-channel and multi-channel speech enhancement tasks, achieving SOTA performance under similar computational budgets.
xLSTM-SENet: xLSTM for Single-Channel Speech Enhancement
Authors: Nikolai Lund KÃ¼hne; Jan Ãstergaard; Jesper Jensen; Zheng-Hua Tan
Venue: Interspeech 2025 (main)

Abstract: While attention-based architectures, such as Conformers, excel in speech enhancement, they face challenges such as scalability with respect to input sequence length. In contrast, the recently proposed Extended Long Short-Term Memory (xLSTM) architecture offers linear scalability. However, xLSTM-based models remain unexplored for speech enhancement. This paper introduces xLSTM-SENet, the first xLSTM-based single-channel speech enhancement system. A direct comparative analysis reveals that xLSTM---and notably, even LSTM---can match or outperform state-of-the-art Mamba- and Conformer-based systems across various model sizes in speech enhancement on the VoiceBank+Demand dataset. Through ablation studies, we identify key architectural design choices such as exponential gating and bidirectionality contributing to its effectiveness. Our best xLSTM-based model, xLSTM-SENet2, outperforms state-of-the-art Mamba- and Conformer-based systems of similar complexity on the Voicebank+DEMAND dataset.