GenAI4ELab/papercli-papers huggingface dataset:
Pretraining Multi-Speaker Identification for Neural Speaker Diarization
Authors: Shota Horiguchi; Atsushi Ando; Naohiro Tawara; Marc Delcroix
Venue: Interspeech 2025 (main)

Abstract: End-to-end speaker diarization enables accurate overlap-aware diarization by jointly estimating multiple speakers' speech activities in parallel. This approach is data-hungry, requiring a large amount of labeled conversational data, which cannot be fully obtained from real datasets alone. To address this issue, large-scale simulated data is often used for pretraining, but it requires enormous storage and I/O capacity, and simulating data that closely resembles real conversations remains challenging. In this paper, we propose pretraining a model to identify multiple speakers from an input fully overlapped mixture as an alternative to pretraining a diarization model. This method eliminates the need to prepare a large-scale simulated dataset while leveraging large-scale speaker recognition datasets for training. Through comprehensive experiments, we demonstrate that the proposed method enables a highly accurate yet lightweight local diarization model without simulated conversational data.
Parameter-Efficient Fine-Tuning for Low-Resource Text-to-Speech via Cross-Lingual Continual Learning
Authors: Ki-Joong Kwon; Jun-Ho So; Sang-Hoon Lee
Venue: Interspeech 2025 (main)

Abstract: As generative models gain attention, it is crucial to adapt these models efficiently even with limited high-quality data and computational resources. In this work, we investigate a parameter-efficient fine-tuning (PEFT) for low-resource text-to-speech to transfer pre-trained knowledge to a new language leveraging only a single-speaker dataset and a single NVIDIA TITAN RTX GPU. We propose three types of adapters: Conditioning Adapter, Prompt Adapter, and DiT LoRA Adapter, where Conditioning Adapter enhances text embeddings, Prompt Adapter refines input representations, and DiT LoRA Adapter enables speech generation efficiency. We further explore the respective optimal configuration of adapters for single-speaker and multi-speaker scenarios. Consequently, under resource constraints, we successfully achieve effective adaptation to a new language using only 1.72% of the total parameters. Audio samples, source code and checkpoints will be available.
Accent Normalization Using Self-Supervised Discrete Tokens with Non-Parallel Data
Authors: Qibing Bai; Sho Inoue; Shuai Wang; Zhongjie Jiang; Yannan Wang; Haizhou Li
Venue: Interspeech 2025 (main)

Abstract: Accent normalization converts foreign-accented speech into native-like speech while preserving speaker identity. We propose a novel pipeline using self-supervised discrete tokens and non-parallel training data. The system extracts tokens from source speech, converts them through a dedicated model, and synthesizes the output using flow matching. Our method demonstrates superior performance over a frame-to-frame baseline in naturalness, accentedness reduction, and timbre preservation across multiple English accents. Through token-level phonetic analysis, we validate the effectiveness of our token-based approach. We also develop two duration preservation methods, suitable for applications such as dubbing.
LIST: Language-Independent Speech Token for Multilingual Speech Synthesis with Language Models
Authors: Chang Liu; Zhen-Hua Ling; Yu Gu
Venue: Interspeech 2025 (main)

Abstract: Recently, language model (LM)-based speech synthesis models have shown remarkable naturalness and powerful zero-shot capabilities. In this paradigm, discrete speech tokens play a critical role. Prior work has proposed using automatic speech recognition (ASR) tasks to enhance the semantic information and alignment with text in tokens. However, the commonly used byte-pair encoding (BPE) tokenizer in ASR task leads to significant differences in the text token sets of different languages, making it difficult to exploit language-shared information. This paper proposes to use the International Phonetic Alphabet (IPA) as the training target for ASR to learn language-independent speech tokens. In addition, we propose to use a timbre converter for speaker disentanglement in the speech synthesis model. Our proposed approach effectively improves the speaker similarity and expressiveness in both multilingual and cross-lingual zero-shot speech synthesis.
Developing High-Quality TTS for Punjabi and Urdu: Benchmarking against MMS Models
Authors: Fatima Naseem; Maham Sajid; Farah Adeeba; Sahar Rauf; Asad Mustafa; Sarmad Hussain; Faisal Kamiran
Venue: Interspeech 2025 (main)

Abstract: Existing Punjabi text-to-speech (TTS) solutions focus on Gurumukhi script, requiring transliteration from Shahmukhi. This leads to letter substitutions and omissions, resulting in pronunciation errors. In this study, speech corpus, phonetic lexicon, and text analysis module for Punjabi Shahmukhi were developed. Two model architectures: Tacotron 1 and Tacotron 2 with WaveGlow were used to build TTS models. In addition to Punjabi, Urdu TTS models were also developed. These models were benchmarked against Urdu and Punjabi Gurumukhi TTS models provided by Metaâs Massively Multilingual Speech (MMS) which is a top profile multilingual speech project. Objective and subjective evaluations indicate that tacotron based Urdu and Punjabi models outperform MMS in intelligibility, naturalness, and phonetic accuracy, enhancing TTS quality for these languages.
Synthesizing Speech with Selected Perceptual Voice Qualities â A Case Study with Creaky Voice
Authors: Frederik Rautenberg; Fritz Seebauer; Jana Wiechmann; Michael Kuhlmann; Petra Wagner; Reinhold Haeb-Umbach
Venue: Interspeech 2025 (main)

Abstract: The control of perceptual voice qualities in a text-to-speech (TTS) system is of interest for applications where unmanipulated and manipulated speech probes can serve to illustrate phonetic concepts that are otherwise difficult to grasp. Here, we show that a TTS system, that is augmented with a global speaker attribute manipulation block based on normalizing flows, is capable of correctly manipulating the non-persistent, localized quality of creaky voice, thus avoiding the necessity of a, typically unreliable, frame-wise creak predictor. Subjective listening tests confirm successful creak manipulation at a slightly reduced MOS score compared to the original recording.
Intrasentential English in Swedish TTS: perceived English-accentedness
Authors: Christina TÃ¥nnander; David House; Jonas Beskow; Jens Edlund
Venue: Interspeech 2025 (main)

Abstract: English names and expressions are frequently inserted into Swedish text. Humans intuitively adjust the degree of English pronunciation of such insertions. This work aims at a Swedish text-to-speech synthesis (TTS) capable of similar controlled adaptation. We focus on two key aspects: (1) the development of a TTS system with controllable degrees of perceived English-accentedness (PEA); and (2) the exploration of human preferences related to PEA. We trained a Swedish TTS voice on Swedish and English sentences with a conditioning parameter for language (English-accentedness, EA) on a scale from 0 to 1, and estimated a psychometric mapping of the perceived effect of EA to a perceptual scale (PEA) through perception tests. PEA was then used in Best-Worst listening tests presenting English insertions with varying PEA. The results confirm the effectiveness of the training and the PEA scale, and that listener preferences change with different insertions.
Parameter-Efficient Fine-tuning with Instance-Aware Prompt and Parallel Adapters for Speaker Verification
Authors: Shengyu Peng; Wu Guo; Jie Zhang; Yu Guan; Lipeng Dai; Zuoliang Li
Venue: Interspeech 2025 (main)

Abstract: In this paper, we propose a parameter-efficient fine-tuning method to tailor a pre-trained model for speaker verification. The proposed method simultaneously considers adapter tuning and prompt tuning in one framework. Instead of conventional static prompts, we first insert a prompt generator between two neighboring transformer layers of the pre-trained model, which can incorporate utterance-specific clues to dynamically generate instance-aware prompts. Meanwhile, we append parallel adapter branches to the multi-head attention and feed-forward modules in the transformer layers in order to capture speaker-related information. Experimental results on the VoxCeleb datasets demonstrate the superiority of our method in case of updating fewer than 10% of the parameters.
Unified Text and Speaker Verification using SSL model for Text-Dependent Speaker Verification
Authors: Nathan Griot; Driss Matrouf; Raphael Blouet; Jean-FranÃ§ois Bonastre; Ana Mantecon
Venue: Interspeech 2025 (main)

Abstract: The work presented in this article falls within text-dependent speaker recognition. In our framework, each speaker owns and pronounces a secret phrase. It corresponds to two tasks: verification of the spoken text (Text Validation) and  verification of the speaker's identity (SV). These tasks are usually carried-out in tandem by two different systems. Maintaining two systems involves a certain amount of complexity and may present shortcomings in terms of reliability. In this paper, we propose to use a Self-Supervised Learning Model (SSL) to develop a unified system capable of performing both tasks simultaneously. The proposed approach combines two models on a common SSL and takes advantage of a teacher-student paradigm to integrate textual constraints into the SV part, without requiring lexical labels during its learning phase. Evaluation on different datasets demonstrates the effectiveness of the approach.
Towards Robust Overlapping Speech Detection: A Speaker-Aware Progressive Approach Using WavLM
Authors: Zhaokai Sun; Li Zhang; Qing Wang; Pan Zhou; Lei Xie
Venue: Interspeech 2025 (main)

Abstract: Overlapping Speech Detection (OSD) aims to identify regions where multiple speakers overlap in a conversation, a critical challenge in multi-party speech processing. This work proposes a speaker-aware progressive OSD model that leverages a progressive training strategy to enhance the correlation between subtasks such as voice activity detection (VAD) and overlap detection. To improve acoustic representation, we explore the effectiveness of state-of-the-art self-supervised learning (SSL) models, including WavLM and wav2vec 2.0, while incorporating a speaker attention module to enrich features with frame-level speaker information. Experimental results show that the proposed method achieves state-of-the-art performance, with an F1 score of 82.76% on the AMI test set, demonstrating its robustness and effectiveness in OSD.
Towards Secure User Authentication for Headphones via In-Ear or In-Earcup Microphones
Authors: N Shashaank; Xiao Quan; Andrew Kaluzny; Leonard Varghese; Marko Stamenovic; Chuan-Che Huang
Venue: Interspeech 2025 (main)

Abstract: As headphones and earbuds with integrated AI assistants become more prevalent, it is important to prevent other people from gaining unauthorized access to these devices and accessing personal, sensitive user information. One solution to this problem is to implement a speaker identification (SID) model that can register authorized users onto the headphones and verify their identity in real-time, but it is challenging for these models to work in loud and/or noisy conditions, and they can be easily hacked using voice cloning, spoofing, and other adversarial attack techniques. In this paper, we propose using speech data collected from in-ear and in-earcup microphones commonly found on noise-cancelling headphones and earbuds to fine-tune SID models to address these issues. We collected inside mic data from 195 speakers across 4 headphones and earbuds and show that fine-tuning the model on multiple devices can improve performance when evaluating on a single device.
Mimic Blocker: Self-Supervised Adversarial Training for Voice Conversion Defense with Pretrained Feature Extractors
Authors: Gwangyeol Yu; Junhyeok Lee; Seoryeong Kim; Jimin Lee; Jehyuk Lee
Venue: Interspeech 2025 (main)

Abstract: Voice conversion (VC) enables natural speech synthesis with minimal data; however, it poses security risks, e.g., identity theft and privacy breaches. To address this, we propose Mimic Blocker, an active defense mechanism that prevents VC models from extracting speaker characteristics while preserving audio quality. Our method employs adversarial training, an audio quality preservation strategy, and an attack strategy. It relies on only publicly available pretrained feature extractors, which ensures model-agnostic protection. Furthermore, it enables self-supervised learning using only the original speaker's speech. Experimental results demonstrate that our method achieves robust defense performance in both white-box and black-box scenarios. Notably, the proposed approach maintains audio quality by generating noise imperceptible to human listeners, thereby enabling protection while retaining natural voice characteristics in practical applications.
A Siamese Network-Based Framework for Voice Mimicry Proficiency Assessment Using X-Vector Embeddings
Authors: Bhasi K.C.; Rajeev Rajan
Venue: Interspeech 2025 (main)

Abstract: This paper evaluates the quality of mimicked speech by computing X-vector based speaker embeddings derived from spectral and prosodic features. These embeddings are processed through a long short term memory (LSTM)-based Siamese Network.The best mimicry artist is first identified through a perception test, and we then examine whether the Siamese Network predicts the same artist. A correct prediction is counted as a hit when the model assigns the highest similarity score (rank-1) to the artist identified by the mean opinion score (MOS). Performance evaluation is conducted on the MIMICz data set using Top-K hit rate. The results highlight the effectiveness of score-level fusion of spectral and prosodic speaker embeddings in assessing voice mimickry competency, with the fusion approach achieving a Top-1 hit rate of 72% and a Top-2 hit rate of 88%.
Towards Source Attribution of Singing Voice Deepfake with Multimodal Foundation Models
Authors: Orchid Chetia Phukan; Girish; Mohd Mujtaba Akhtar; Swarup Ranjan Behera; Priyabrata Mallick; Pailla Balakrishna Reddy; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2025 (main)

Abstract: In this work, we introduce the task of singing voice deepfake source attribution (SVDSA). We hypothesize that multimodal foundation models (MMFMs) such as ImageBind, LanguageBind will be most effective for SVDSA as they are better equipped for capturing subtle source-specific characteristicsâsuch as unique timbre, pitch manipulation, or synthesis artifacts of each singing voice deepfake source due to their crossmodality pre-training. Our experiments with MMFMs, speech foundation models and music foundation models verify the hypothesis that MMFMs are the most effective for SVDSA. Furthermore, inspired from related research, we also explore fusion of foundation models (FMs) for improved SVDSA. To this end, we propose a novel framework, COFFE which employs Chernoff Distance as novel loss function for effective fusion of FMs. Through COFFE with the symphony of MMFMs, we attain the topmost performance in comparison to all the individual FMs and baseline fusion methods.
Multimodal Zero-Shot Framework for Deepfake Hate Speech Detection in Low-Resource Languages
Authors: Rishabh Ranjan; Likhith Ayinala; Mayank Vatsa; Richa Singh
Venue: Interspeech 2025 (main)

Abstract: This paper introduces a novel multimodal framework for hate speech detection in deepfake audio, excelling even in zero-shot scenarios. Unlike previous approaches, our method uses contrastive learning to jointly align audio and text representations across languages. We present the first benchmark dataset with 127,290 paired text and synthesized speech samples in six languages: English and five low-resource Indian languages (Hindi, Bengali, Marathi, Tamil, Telugu). Our model learns a shared semantic embedding space, enabling robust cross-lingual and cross-modal classification. Experiments on two multilingual test sets show our approach outperforms baselines, achieving accuracies of 0.819 and 0.701, and generalizes well to unseen languages. This demonstrates the advantage of combining modalities for hate speech detection in synthetic media, especially in low-resource settings where unimodal models falter. The Dataset is available.
Joint Target-Speaker ASR and Activity Detection
Authors: Chikara Maeda; Muhammad Shakeel; Yui Sudo
Venue: Interspeech 2025 (main)

Abstract: Target-speaker automatic speech recognition (TS-ASR) has shown promise in transcribing speech in multi-speaker scenarios by focusing on a specific speaker. However, existing approaches employ a cascaded design, where voice activity detection (VAD) and TS-ASR are optimized separately. This separation leads to unstable training and error accumulation, limiting overall performance. We address these issues by proposing TS-ASR-AD, a joint end-to-end model that integrates VAD with TS-ASR, enabling stable training and reducing error accumulation. Moreover, improved training stability leads to better connectionist temporal classification (CTC) alignment in token probabilities, further enhancing transcription accuracy. Our approach outperforms previous studies and achieves a word error rate (WER) of 6.61% and 14.81%, and a diarization error rate (DER) of 1.23% and 2.66% on the Libri2Mix and Libri3Mix datasets, respectively.
DLF-EEND: Dynamic Layer Fusion for End-to-End Speaker Diarization
Authors: Wooil Kim; Bongsu Jung
Venue: Interspeech 2025 (main)

Abstract: In this study, we propose Dynamic Layer Fusion for EEND (DLF-EEND), a novel approach for integrating Transformer layer information in end-to-end speaker diarization. The model introduces an auxiliary branch during training, using dynamic routing to adaptively fuse multi-resolution representations at each time step. Applying Permutation-Invariant Training (PIT) loss to the fused features enhances hierarchical learning, from low-level acoustic cues to high-level speaker separation. This preserves distinct layer-specific information and improves diarization accuracy, particularly in overlapping speech and speaker transitions. During inference, only the main branch is used, reducing computation while maintaining inter-layer fusion benefits. Experiments show DLF-EEND reduces Diarization Error Rate (DER) by 59.18% on simulated datasets and improves CALLHOME performance by 14.26%, outperforming SA-EEND.
Analysis of Avian Biphonic Vocalization Using Computational Modelling
Authors: Noumida A; Rajeev Rajan
Venue: Interspeech 2025 (main)

Abstract: Birds produce complex vocalizations through the coordinated dynamics of neuromuscular control, where the syrinx acts as the sound source, and suprasyringeal vocal tract structures function as acoustic filters. In this study, we developed computational models of the avian syrinx and upper vocal tract, reconstructed from high-resolution micro-CT imaging. Using finite element analysis in COMSOL Multiphysics, we introduce dual sources at the bottom of the trachea to analyze the filtering mechanisms underlying biphonic vocalization. The simulation results are experimentally validated using real bird vocalizations. To further elucidate the biomechanical factors governing resonance modulation, we systematically vary tracheal length, glottal radius, and beak angle. Additionally, we perform an analysis of syllable bandwidth to assess the influence of vocal tract dynamics on the frequency range observed in biphonic vocalizations.
Dog2vec: Self-Supervised Pre-Training for Canine Vocal Representation
Authors: Xingyuan Li; Kenny Zhu; Mengyue Wu
Venue: Interspeech 2025 (main)

Abstract: Previous generalized biological voice models were trained on large amounts of data from multiple species. However, on average, there is very little training data on species-specific voices, while large differences between the vocalizations of species may even be a barrier to encoding vocal features. This leads to potentially large errors in using generic models for species-specific vocalization studies. We collected over 6000 hours of dog barking videos and presented the first animal-specific bioacoustic embedding model, Dog2vec.1 The results indicate that Dog2vec outperforms species-independent pre-trained models and achieves state-of-the-art results on a series of dog-related tasks, including dog bark type recognition and dog sound event detection, and obtain a relative 8.2% performance increase.
Improving Bird Classification with Primary Color Additives
Authors: Ezhini Rasendiran R; Chandresh Kumar Maurya
Venue: Interspeech 2025 (main)

Abstract: We address the problem of classifying bird species using their song recordings, a challenging task due to environmental noise, overlapping vocalizations, and missing labels. Existing models struggle with low-SNR or multi-species recordings. We hypothesize that birds can be classified by visualizing their pitch pattern, speed, and repetitionâcollectively called motifs. Deep learning models applied to spectrogram images help, but similar motifs across species cause confusion. To mitigate this, we embed frequency information into spectrograms using primary color additives. This enhances species distinction, improving classification accuracy. Our experiments show that the proposed approach achieves statistically significant gains over models without colorization and surpasses the BirdCLEF 2024 winner, improving F1 by 7.3%, ROC-AUC by 6.2%, and CMAP by 6.6%. These results show the effectiveness of incorporating frequency information via colorization.
Exploring the Power of Empirical Mode Decomposition for Sensing the Sound of Silence: A Pilot Study on Mice Autism Detection via Ultrasonic Vocalisation
Authors: Chenhao Wu; Xiangjun Cai; Haojie Zhang; Tianrui Jia; Yilu Deng; Kun Qian; BjÃ¶rn W. Schuller; Yoshiharu Yamamoto; Jiang Liu
Venue: Interspeech 2025 (main)

Abstract: Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder, and mice models have become essential for studying its genetic and behavioural aspects. Ultrasonic Vocalisations (USVs) emitted by mice provide a promising biomarker for ASD detection, but existing methods relying on spectrogram-based features struggle to capture the complex, non-stationary, and multi-scale nature of USVs. To address this, we propose a novel multi-branch fusion model that integrates spectrogram-based features with multi-scale features extracted using Empirical Mode Decomposition (EMD), which decomposes USVs into Intrinsic Mode Functions (IMFs) to represent their inherent complexity better. Through systematic occlusion experiments, we identify high-frequency components, particularly IMF1, as critical for accurate ASD detection, highlighting the diagnostic relevance of high-frequency USV patterns. Our model achieves an Unweighted Average Recall (UAR) of 0.75 in subject-level classification, significantly outperforming existing methods. These findings provide valuable insights into the importance of multi-scale feature extraction and offer a robust framework for improving ASD diagnostics and research.