GenAI4ELab/papercli-papers huggingface dataset:
Unmasking real-world audio deepfakes: A data-centric approach
Authors: David Combei; Adriana Stan; Dan Oneata; Nicolas MÃ¼ller; Horia Cucu
Venue: Interspeech 2025 (main)

Abstract: The growing prevalence of real-world deepfakes presents a critical challenge for existing detection systems, which are often evaluated on datasets collected just for scientific purposes. To address this gap, we introduce a novel dataset of real-world audio deepfakes. Our analysis reveals that these real-world examples pose significant challenges, even for the most performant detection models. Rather than increasing model complexity or exhaustively search for a better alternative, in this work we focus on a data-centric paradigm, employing strategies like dataset curation, pruning, and augmentation to improve model robustness and generalization. Through these methods, we achieve a 55% relative reduction in EER on the In-the-Wild dataset, reaching an absolute EER of 1.7%, and a 63% reduction on our newly proposed real-world deepfakes dataset, AI4T. These results highlight the transformative potential of data-centric approaches in enhancing deepfake detection for real-world applications.
A Data-Driven Diffusion-based Approach for Audio Deepfake Explanations
Authors: Petr Grinberg; Ankur Kumar; Surya Koppisetti; Gaurav Bharaj
Venue: Interspeech 2025 (main)

Abstract: Evaluating explainability techniques, such as SHAP and LRP, in the context of audio deepfake detection is challenging due to lack of clear ground truth annotations. In the cases when we are able to obtain the ground truth, we find that these methods struggle to provide accurate explanations. In this work, we propose a novel data-driven approach to identify artifact regions in deepfake audio. We consider paired real and vocoded audio, and use the difference in time-frequency representation as the ground-truth explanation. The difference signal then serves as a supervision to train a diffusion model to expose the deepfake artifacts in a given vocoded audio. Experimental results on the VocV4 and LibriSeVoc datasets demonstrate that our method outperforms traditional explainability techniques, both qualitatively and quantitatively.
PartialEdit: Identifying Partial Deepfakes in the Era of Neural Speech Editing
Authors: You Zhang; Baotong Tian; Lin Zhang; Zhiyao Duan
Venue: Interspeech 2025 (main)

Abstract: Neural speech editing enables seamless partial edits to speech utterances, allowing modifications to selected content while preserving the rest of the audio unchanged. This useful technique, however, also poses new risks of deepfakes. To encourage research on detecting such partially edited deepfake speech, we introduce PartialEdit, a deepfake speech dataset curated using advanced neural editing techniques. We explore both detection and localization tasks on PartialEdit. Our experiments reveal that models trained on the existing PartialSpoof dataset fail to detect partially edited speech generated by neural speech editing models. As recent speech editing models almost all involve neural audio codecs, we also provide insights into the artifacts the model learned on detecting these deepfakes. Further information about the PartialEdit dataset and audio samples can be found on the project page.
Rehearsal with Auxiliary-Informed Sampling for Audio Deepfake Detection
Authors: Falih Gozi Febrinanto; Kristen Moore; Chandra Thapa; Jiangang Ma; Vidya Saikrishna; Feng Xia
Venue: Interspeech 2025 (main)

Abstract: The performance of existing audio deepfake detection frameworks degrades when confronted with new deepfake attacks. Rehearsal-based continual learning (CL), which updates models using a limited set of old data samples, helps preserve prior knowledge while incorporating new information. However, existing rehearsal techniques don't effectively capture the diversity of audio characteristics, introducing bias and increasing the risk of forgetting. To address this challenge, we propose Rehearsal with Auxiliary-Informed Sampling (RAIS), a rehearsal-based CL approach for audio deepfake detection. RAIS employs a label generation network to produce auxiliary labels, guiding diverse sample selection for the memory buffer. Extensive experiments show RAIS outperforms state-of-the-art methods, achieving an average Equal Error Rate (EER) of 1.953% across five experiences. The code is available.
The Prosodic Characteristics of Standard Chinese Rhetorical Questions in Naturalistic Settings
Authors: Shuwen Chen; Qingke Sun; Yue Huang; Yingyi Luo
Venue: Interspeech 2025 (main)

Abstract: The current study investigated the prosodic features of rhetorical questions in Standard Chinese within naturalistic settings, utilizing data collected from 103 native Mandarin speakers. String-identical information-seeking questions (ISQs) and rhetorical questions (RQs) were recorded through an online platform, ensuring a diverse and representative sample. The results revealed that, when freely reading aloud to the phone, speakers tended to convey rhetorical meaning by either shifting prominence toward or enhancing the prominence of the verb or modal verb. Acoustically, the prominent verbs were characterized by higher pitch and longer duration. In other positions, phonetic markers varied depending on the sentence structure. Our large-sample findings highlight the interface of prosody and syntax in conveying communicative intention. This research also addresses a critical gap by moving beyond laboratory-based data to examine speech phenomena in more ecologically valid contexts.
ProBiEM: Acoustic and Lexical Correlates of Prosodic Prominence in English-Malayalam Bilingual Speech
Authors: Anindita Mondal; Rahul Biju; Anil Kumar Vuppala; Reni K Cherian; Chiranjeevi Yarra
Venue: Interspeech 2025 (main)

Abstract: Despite advancements in Speech-to-Speech Translation, maintaining expressiveness between the source and target speech, particularly between English and Indian languages, remains challenging. This study investigates prosodic similarities and variations between English and Malayalam, a language spoken in southern India. A set of 22 prompts from the IViE corpus, covering five categoriesâsimple sentences, WH-questions, questions without morphosyntactic markers, inversion questions, and coordinationsâwas selected. These prompts, originally spoken by UK speakers, were translated into Malayalam and both language prompts were recorded by bilingual Malayalam speakers while preserving expressiveness. Word-level prominence was manually annotated, and comparisons were made across Indian English, Malayalam, and UK English. The analysis reveals that prominence is retained at key points in  Indian English, whereas in Malayalam, it is low due to question specific diacritics and agglutinative nature.
Are You Being Sarcastic? Prosodic Cues to Irony Perception in German
Authors: Sophia FÃ¼nfgeld; Angelika Braun; Katharina Zahner-Ritter
Venue: Interspeech 2025 (main)

Abstract: This perception study investigated the role of intonation, specifically pitch accent position and pitch accent type, in the interpretation of utterances as sarcastic. Participants from two regions in Germany, Freiburg and Trier, listened to short utterances such as Das sieht ja umwerfend aus (âThat looks stunningâ) in seven prosodic conditions. The recordings were taken from a production study, and participants classified them as sarcastic or sincere. Results show that in both regions, irony perception is driven by (a) the existence of a prenuclear accent and (b) the type of the nuclear accent, particularly L*+H. Reaction times for ironic responses were shorter for L*+H, as well as when a prenuclear accent was present or when the subject carried the nuclear accent. These findings underscore the importance of intonation in irony perception and have implications for the processing of non-canonical meaning in general and the interpretation of sarcasm across varieties in particular.
Can AI Understand Mandarin Speech Prosody? A Framework and Benchmark Showcase
Authors: Zilong Wang; Xiaoxue Zhang; Xinyang Jiang; Kaitao Song; Jue Yu
Venue: Interspeech 2025 (main)

Abstract: How to model and estimate speech prosody is considered as a challenging task in understanding and generating natural speech. We introduce the Mandarin Speech Prosody Benchmark (MSPB), a linguistically grounded dataset for evaluating Speech Large Language Models (Speech LLMs) in Mandarin. MSPB comprises eight tasks covering crucial prosodic features and their interactions with syntax, semantics, and pragmatics. All MSPB items, designed per Mandarin linguistic principles and validated by experts, were phonetically recorded and verified. We evaluated six Speech LLMs (GPT-4o, Gemini-1.5-Pro, Gemini-2-Flash, Qwen2-Audio-7B-Instruct, GLM-4-Voice, MiniCPM-o 2.6). Although some models perform well with context-rich cues (e.g., irony), they generally struggle with subtle prosodic variations (e.g., focus marking) and underperform humans. MSPB provides a valuable tool to assess and enhance prosodic comprehension, underscoring the need for improved prosodic integration in future research.
Generating Consistent Prosodic Patterns from Open-Source TTS Systems
Authors: Ha Eun Shim; Olivia Yung; Paige TuttÃ¶sÃ­; Boey Kwan; Angelica Lim; Yue Wang; H. Henny Yeung
Venue: Interspeech 2025 (main)

Abstract: Text-to-Speech (TTS) systems now closely approximate human speech prosody. Yet, current deep learning-based TTS systems may struggle to accurately represent some prosodic patterns, like phrase boundaries used to signal syntactic distinctions. Such prosodic parsing can reflect differences in meaning, hence, inconsistencies in synthesis can lead to miscommunications. In this study, we conduct a qualitative assessment of five open-source TTS systems and reveal that they fail to produce acoustic signals that accurately convey distinct prosodic boundaries when given punctuation contrasts (Study 1). To mitigate this gap, we propose a pipeline for improving output using a customized dataset (Study 2), which successfully generates predictable acoustic cues, but only for certain cases. Results suggest that TTS systems require additional training to effectively capture the prosodic subtleties. We conclude by discussing how TTS systems can better generate fine prosodic distinctions.
Multimodal Prosody Modeling: A Use Case for Multilingual Sentence Mode Prediction
Authors: Bogdan Vlasenko; Mathew Magimai Doss
Venue: Interspeech 2025 (main)

Abstract: Prosody modeling has garnered significant attention from the speech processing community. Recent developments in multilingual latent spaces for representing linguistic and acoustic information have become a new trend in various research directions. Therefore, we decided to evaluate the ability of multilingual acoustic neural embeddings and knowledge-based features to preserve sentence-mode-related information at the suprasegmental level. For linguistic information modeling, we selected neural embeddings based on word- and phoneme-level latent space representations. The experimental study was conducted using Italian, French, and German audiobook recordings, as well as emotional speech samples from EMO-DB. Both intra- and inter-language experimental protocols were used to assess classification performance for uni- and multimodal (early fusion approach) features. For comparison, we used a sentence mode prediction system built on top of automatically generated WHISPER-based transcripts.
HASRD: Hierarchical Acoustic and Semantic Representation Disentanglement
Authors: Amir Hussein; Sameer Khurana; Gordon Wichern; FranÃ§ois G. Germain; Jonathan Le Roux
Venue: Interspeech 2025 (main)

Abstract: Effective speech representations for spoken language models must balance semantic relevance with acoustic fidelity for high-quality reconstruction. However, existing approaches struggle to achieve both simultaneously. To address this, we introduce Hierarchical Acoustic and Semantic Representation Disentanglement (HASRD, pronounced ``hazard''), a framework that factorizes self-supervised learning representations into discrete semantic and acoustic tokens. HASRD assigns the semantic representation to the first codebook, while encoding acoustic residuals in subsequent codebooks. This preserves ASR performance while achieving high-quality reconstruction. Additionally, we enhance HASRDâs encoder efficiency, improving ASR performance without compromising reconstruction quality. Compared to SpeechTokenizer, HASRD achieves a 44% relative WER improvement, superior reconstruction quality, and 2x lower bitrate, demonstrating its effectiveness in disentangling acoustic and semantic information.
Performance of Montreal Forced Aligner on Cantonese Spontaneous Speech
Authors: Ka Ki SO; Chenzi Xu; Grace Wenling Cao; Peggy Mok
Venue: Interspeech 2025 (main)

Abstract: The study presents a comprehensive evaluation of the Montreal Forced Aligner (MFA) in aligning phone boundaries of Hong Kong Cantonese (HKC) spontaneous speech. We developed two tailored Cantonese MFA models, designed to address distinct Cantonese phonetic features, such as checked syllables. These models were applied to align the same set of recordings from spontaneous interviews, and their performance was compared against human annotations. Our results reveal that the updated Cantonese MFA models achieved decent alignment accuracy on spontaneous speech, with a satisfactory level of agreement with manually adjusted boundaries in vowels. However, Cantonese-specific features and connected speech process remain major challenges for the current models. This observation allows us to propose specific amendments to the models to improve alignment performance, as well as recommendations on manual boundary adjustments.
Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information
Authors: Nicholas Sanders; Yuanchao Li; Korin Richmond; Simon King
Venue: Interspeech 2025 (main)

Abstract: Quantization in SSL speech models (e.g., HuBERT) improves compression and performance in tasks like language modeling, resynthesis, and text-to-speech but often discards prosodic and paralinguistic information (e.g., emotion, prominence). While increasing codebook size mitigates some loss, it inefficiently raises bitrates. We propose Segmentation-Variant Codebooks (SVCs), which quantize speech at distinct linguistic units (frame, phone, word, utterance), factorizing it into multiple streams of segment-specific discrete features. Our results show that SVCs are significantly more effective at preserving prosodic and paralinguistic information across probing tasks. Additionally, we find that pooling before rather than after discretization better retains segment-level information. Resynthesis experiments further confirm improved style realization and slightly improved quality while preserving intelligibility.
AdaKWS: Towards Robust Keyword Spotting with Test-Time Adaptation
Authors: Yang Xiao; Tianyi Peng; Yanghao Zhou; Rohan Kumar Das
Venue: Interspeech 2025 (main)

Abstract: Spoken keyword spotting (KWS) aims to identify keywords in audio for wide applications, especially on edge devices. Current small-footprint KWS systems focus on efficient model designs. However, their inference performance can decline in unseen environments or noisy backgrounds. Test-time adaptation (TTA) helps models adapt to test samples without needing the original training data. In this study, we present AdaKWS, the first TTA method for robust KWS to the best of our knowledge. Specifically, 1) We initially optimize the model's confidence by selecting reliable samples based on prediction entropy minimization and adjusting the normalization statistics in each batch. 2) we introduce pseudo-keyword consistency (PKC) to identify critical, reliable features without overfitting to noise. Our experiments show that AdaKWS outperforms other methods across various conditions, including Gaussian noise and real-scenario noises. The code will be released in due course.
Multivariate Probabilistic Assessment of Speech Quality
Authors: Fredrik Cumlin; Xinyu Liang; Victor Ungureanu; Chandan K.A. Reddy; Christian SchÃ¼ldt; Saikat Chatterjee
Venue: Interspeech 2025 (main)

Abstract: The mean opinion score (MOS) is a standard metric for assessing speech quality, but its singular focus fails to identify specific distortions when low scores are observed. The NISQA dataset addresses this limitation by providing ratings across four additional dimensions: noisiness, coloration, discontinuity, and loudness, alongside MOS. In this paper, we extend the explored univariate MOS estimation to a multivariate framework by modeling these dimensions jointly using a multivariate Gaussian distribution. Our approach utilizes Cholesky decomposition to predict covariances without imposing restrictive assumptions, and extends probabilistic affine transformations to a multivariate context. Experimental results show that our model performs on par with state-of-the-art methods in point estimation, while uniquely providing uncertainty and correlation estimates across speech quality dimensions. This enables better diagnosis of poor speech quality and inform targeted improvements.
A Study on Speech Assessment with Visual Cues
Authors: Shafique Ahmed; Ryandhimas E. Zezario; Nasir Saleem; Amir Hussain; Hsin-Min Wang; Yu Tsao
Venue: Interspeech 2025 (main)

Abstract: Non-intrusive assessment of speech quality and intelligibility is essential when clean reference signals are unavailable. In this work, we propose a multimodal framework that integrates audio features and visual cues to predict PESQ and STOI scores. It employs a dual-branch architecture, where spectral features are extracted using STFT, and visual embeddings are obtained via a visual encoder. These features are then fused and processed by a CNN-BLSTM with attention, followed by multi-task learning to simultaneously predict PESQ and STOI. Evaluations on the LRS3-TED dataset, augmented with noise from the DEMAND corpus, show that our model outperforms the audio-only baseline. Under seen noise conditions, it improves LCC by 9.61% (0.8397â0.9205) for PESQ and 11.47% (0.7403â0.8253) for STOI. These results highlight the effectiveness of incorporating visual cues in enhancing the accuracy of non-intrusive speech assessment.
Efficient Streaming Speech Quality Prediction with Spiking Neural Networks
Authors: Mattias Nilsson; Riccardo Miccini; Julian Rossbroich; ClÃ©ment Laroche; Tobias Piechowiak; Friedemann Zenke
Venue: Interspeech 2025 (main)

Abstract: As speech processing systems become more ubiquitous, the need for real-time, efficient speech quality prediction (SQP) is growing. Conventional artificial neural networks (ANNs) offer strong prediction performance but can be computationally demanding, which limits their deployment on mobile and edge devices. Spiking neural networks (SNNs) present a promising alternative for ultra-low-power, streaming inference due to their sparse activity and event-driven processing. However, their potential for SQP remains largely unexplored. This article introduces deep convolutional SNNs for SQP and evaluates their performance against state-of-the-art ANN models. Our results show that SNNs achieve comparable accuracy while significantly reducing computational cost. These findings highlight the potential of SNNs to enable real-time, energy-efficient SQP in resource-constrained settings.
Unifying Listener Scoring Scales: Comparison Learning Framework for Speech Quality Assessment and Continuous Speech Emotion Recognition
Authors: Cheng Hung Hu; Yusuke Yasuda; Akifumi Yoshimoto; Tomoki Toda
Venue: Interspeech 2025 (main)

Abstract: Speech Quality Assessment (SQA) and Continuous Speech Emotion Recognition (CSER) are two key tasks in speech technology, both relying on listener ratings. However, these ratings are inherently biased due to individual listener factors. Previous approaches have introduced a mean listener scoring scale and modeled all listener scoring scales in the training set. However, the mean listener approach is prone to distortion from averaging ordinal data, leading to potential biases. Moreover, learning multiple listener scoring scales while inferring based only on the mean listener scale limits effectiveness. In contrast, our method focuses on modeling a unified listener scoring scale, using comparison scores to correctly capture the scoring relationships between utterances. Experimental results show that our method effectively improves prediction performance in both SQA and CSER tasks, proving its effectiveness and robustness.
EAA: Emotion-Aware Audio Large Language Models with Dual Cross-Attention and Context-Aware Instruction Tuning
Authors: Hongfei Du; Sidi Lu; Gang Zhou; Ye Gao
Venue: Interspeech 2025 (main)

Abstract: Understanding speech emotion through artificial intelligence (AI) is crucial for human-computer interaction and mental health monitoring. While audio large language models (ALLMs) excel in speech comprehension, they face challenges in accurately integrating emotional signals from acoustic and semantic features. Moreover, emotions often span dialogues, making sole reliance on current audio insufficient for comprehensive understanding. To address these challenges, we propose a novel emotion-aware audio large language model (EAA). Specifically, we design a dual cross-attention mechanism to fuse acoustic and semantic information for a more comprehensive emotional representation. Furthermore, we use context-aware instruction tuning by incorporating the current and immediately preceding utterances as contextual information, enhancing task understanding and emotion recognition. Our experimental results show that EAA outperforms existing ALLMs on the MELD dataset, improving accuracy by 11.4%.
Chain-of-Thought Distillation with Fine-Grained Acoustic Cues for Speech Emotion Recognition
Authors: Jialong Mai; Xiaofen Xing; Yangbiao Li; Xiangmin Xu
Venue: Interspeech 2025 (main)

Abstract: Recent advances in large language models (LLMs) have demonstrated strong reasoning abilities through chain-of-thought (CoT) prompting, yet their application in speech emotion recognition (SER) remains underexplored. Moreover, current SER models lack explainability based on emotion-related acoustic features. We propose AECoTD, a method that transfers reasoning abilities from a large LLM to a domain-specific SER LLM by leveraging fine-grained emotional acoustic features and text transcripts. It uses LoRA to distill the reasoning chain and an emotion-focused loss to preserve correct emotional attention, thereby enhancing the modelâs explainability. Ablation experiments highlight the impact of fine-grained acoustic information, emotional CoT reasoning, and emotion-focused loss. Without using pre-trained representations, our method achieves state-of-the-art performance both in-domain and out-of-domain, demonstrating strong generalization ability.
Exploring the Limits of Conformer CTC-Encoder for Speech Emotion Recognition using Large Language Models
Authors: Edmilson Morais; Hagai Aronowitz; Aharon Satt; Ron Hoory; Avihu Dekel; Brian Kingsbury; George Saon
Venue: Interspeech 2025 (main)

Abstract: Conformer CTC-Encoders have consistently delivered state-of-the-art results in the field of Automatic Speech Recognition (ASR); however, their merits for tasks that demand more semantic and paralinguistic information, such as Automatic Speech Understanding (ASRU), Speech Emotion Recognition (SER) and Speech Translation (ST), still need further investigation. In this paper, we introduce a Speech Large Language Model (SLLM) system based on a Conformer CTC-Encoder and on the Granite Large Language Model that allowed us to perform several experiments on ASR, SER and ST tasks. These experiments have not only confirmed the strength of Conformer CTC-encoders for ASR, but also, they have shown that the outputs of intermediate Conformer Blocks, of the Conformer CTC-Encoder, carry important information for SER tasks and that the Conformer CTC-Encoder can be efficiently fine-tuned for SER tasks.