GenAI4ELab/papercli-papers huggingface dataset:
Towards Fusion of Neural Audio Codec-based Representations with Spectral for Heart Murmur Classification via Bandit-based Cross-Attention Mechanism
Authors: Orchid Chetia Phukan; Girish; Mohd Mujtaba Akhtar; Swarup Ranjan Behera; Priyabrata Mallick; Santanu Roy; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2025 (main)

Abstract: In this study, we focus on heart murmur classification (HMC) and hypothesize that combining neural audio codec representations (NACRs) such as EnCodec with spectral features (SFs), such as MFCC, will yield superior performance. We believe such fusion will trigger their complementary behavior as NACRs excel at capturing fine-grained acoustic patterns such as rhythm changes, spectral features focus on frequency-domain properties such as harmonic structure, spectral energy distribution crucial for analyzing the complex of heart sounds. To this end, we propose, BAOMI, a novel framework banking on novel bandit-based cross-attention mechanism for effective fusion. Here, a agent provides more weightage to most important heads in multi-head cross-attention mechanism and helps in mitigating the noise. With BAOMI, we report the topmost performance in comparison to individual NACRs, SFs, and baseline fusion techniques and setting new state-of-the-art.
Perception of Emotional Speech by Individuals with High Borderline Personality Features
Authors: Yizhou Chen; Xiyu Wu
Venue: Interspeech 2025 (main)

Abstract: Borderline personality disorder (BPD) is characterized by emotional dysregulation. Prior research on facial emotion recognition has revealed that BPD individuals have emotion perception abnormalities. This study examined whether these findings extended to emotional speech recognition. Mandarin emotional speech (neutral, angry, happy, and sad) was synthesized at varying emotional intensities by adjusting the fundamental frequency. A perceptual experiment was conducted with Chinese university students with high and low borderline personality features (BPF). High-BPF participants showed lower accuracy in identifying neutral speech, more frequently misidentifying it as other emotions, and were less accurate in identifying high-intensity happy speech, tending to misclassify it as neutral. They also exhibited marginally higher confidence in recognizing angry speech. The results contribute to the phenomenological understanding of emotional dysregulation in BPD.
Visual features of the oral region in Polish sibilants produced by children with various sibilance patterns
Authors: Agata Sage; Zuzanna MiodoÅska; MichaÅ KrÄcichwost; Ewa KwaÅniok; PaweÅ Badura
Venue: Interspeech 2025 (main)

Abstract: This paper analyzes the 2D and 3D visual features of Polish sibilants /s/ and /Ê/ articulated by 189 children aged 4;11-8;0. Our goal was to (1) analyze differences in shape image features of delineated articulators (lips, tongue, and mouth - lips and space between them) in selected sibilants and (2) investigate a relationship between visual parameters and sibilance pattern (the character of a gap between teeth that allows the articulation of sibilant sounds). We employed 2D and 3D shape image features and linear mixed-effect models. The results proved significant differences between sibilants in 2D and 3D visual features of lips and mouth. Different sibilance patterns considered in the study also significantly impacted visual parameters distribution. Our findings indicate the potential of video data in computer-aided speech diagnosis.
Meta-Learning Approaches for Speaker-Dependent Voice Fatigue Models
Authors: Roseline Polle; Agnes Norbury; Alexandra Livia Georgescu; Nicholas Cummins; Stefano Goria
Venue: Interspeech 2025 (main)

Abstract: Speaker-dependent modelling can substantially improve performance in speech-based health monitoring applications. While mixed-effect models are commonly used for such speaker adaptation, they require computationally expensive retraining for each new observation, making them impractical in a production environment. We reformulate this task as a meta-learning problem and explore three approaches of increasing complexity: ensemble-based distance models, prototypical networks, and transformer-based sequence models. Using pre-trained speech embeddings, we evaluate these methods on a large longitudinal dataset of shift workers (N=1,185, 10,286 recordings), predicting time since sleep from speech as a function of fatigue, a symptom commonly associated with ill-health. Our results demonstrate that all meta-learning approaches tested outperformed both cross-sectional and conventional mixed-effects models, with a transformer-based method achieving the strongest performance.
Decoding Alzheimerâs: Interpretable Visual and Logical Attention in Picture Description Tasks
Authors: Ning Wang; Bingyang Wen; Minghui Wu; Yang Sun; Zongru Shao; Haojie Zhou; K.P. Subbalakshmi
Venue: Interspeech 2025 (main)

Abstract: Recent studies have started to incorporate imagery information from picture-description tasks in clinical interviews to automate Alzheimerâs disease detection in the elderly. However, the high-level logical flow of visual-attention cognition mechanisms has not yet been investigated for enhanced interpretability. In this study, we systematically analyze the elements of picture-description tasks and propose a set of top-to-bottom human-interpretable features to describe the cognitive behaviors of patients, focusing on visual attention patterns, description quality, and repetition characteristics. These features achieve 85% accuracy in AD detection without specialized equipment, offering valuable insights for clinical practices and non-expert caregivers. Our results demonstrate that these high-level descriptive features, particularly those related to visual attention and the logical flow of speech, serve as effective biomarkers for AD detection.
Defending Speech-enabled LLMs Against Adversarial Jailbreak Threats
Authors: Antonios Alexos; Raghuveer Peri; Sai Muralidhar Jayanthi; Metehan Cekic; Srikanth Vishnubhotla; Kyu J. Han; Srikanth Ronanki
Venue: Interspeech 2025 (main)

Abstract: The additional modality (such as speech) in multimodal large language models (LLM) increases the vulnerability of these models to adversarial jailbreak attacks. Adversarial training (AT) techniques have shown great promise as defenses in traditional adversarial robustness literature. But they are less explored as countermeasures in speech-enabled LLMs due to the limited availability of training data and computational complexity. In this work, we develop AT techniques tailored to speech LLMs using a combination of synthesized harmful and benign queries. We experiment with different training data configurations, and evaluate the methods on strong white-box adversarial attacks. We demonstrate through extensive ablations that using just 4hrs of harmful speech queries for AT (with 150 hours of benign speech) can provide significant gains compared to vanilla safety fine-tuning, improving safety by 45%-300% relative depending on the model.
Mitigating Subgroup Disparities in Multi-Label Speech Emotion Recognition: A Pseudo-Labeling and Unsupervised Learning Approach
Authors: Yi-Cheng Lin; Huang-Cheng Chou; Hung-yi Lee
Venue: Interspeech 2025 (main)

Abstract: While subgroup disparities and performance bias are increasingly studied in computational research, fairness in categorical Speech Emotion Recognition (SER) remains underexplored. Existing methods often rely on explicit demographic labels, which are difficult to obtain due to privacy concerns. To address this limitation, we introduce an Implicit Demography Inference (IDI) module that leverages pseudo-labeling from a pre-trained model and unsupervised learning using k-means clustering to mitigate bias in SER. Our experiments show that pseudo-labeling IDI reduces subgroup disparities, improving fairness metrics by over 28% with less than a 2% decrease in SER accuracy. Also, the unsupervised IDI yields more than a 4.6% improvement in fairness metrics with a drop of less than 3.6% in SER performance. Further analyses reveal that the unsupervised IDI consistently mitigates race and age disparities, demonstrating its potential when explicit demographic information is unavailable.
Who Gets the Mic? Investigating Gender Bias in the Speaker Assignment of a Speech-LLM
Authors: Dariia Puhach; Amir H. Payberah; Ãva SzÃ©kely
Venue: Interspeech 2025 (main)

Abstract: Similar to text-based Large Language Models (LLMs), Speech-LLMs exhibit emergent abilities and context awareness. However, whether these similarities extend to gender bias remains an open question. This study proposes a methodology leveraging speaker assignment as an analytic tool for bias investigation. Unlike text-based models, which encode gendered associations implicitly, Speech-LLMs must produce a gendered voice, making speaker selection an explicit bias cue. We evaluate Bark, a Text-to-Speech (TTS) model, analyzing its default speaker assignments for textual prompts. If Barkâs speaker selection systematically aligns with gendered associations, it may reveal patterns in its training data or model design. To test this, we construct two datasets: (i) Professions, containing gender-stereotyped occupations, and (ii) Gender-Colored Words, featuring gendered connotations. While Bark does not exhibit systematic bias, it demonstrates gender awareness and has some gender inclinations.
Evaluating Speech Foundation Models for Automatic Speech Recognition in the Low-Resource KanyenâkÃ©ha Language
Authors: Mengzhe Geng; Patrick Littell; Aidan Pine; Robbie Jimerson; Gilles Boulianne; Vishwa Gupta; Rolando Coto-Solano; Anna Kazantseva; Marc Tessier; Delaney Lothian; AkwiratÃ©kha' Martin; Eric Joanis; Samuel Larkin; Roland Kuhn
Venue: Interspeech 2025 (main)

Abstract: Despite recent progress in automatic speech recognition (ASR) and speech foundation models (SFMs) for widely spoken languages, their application to low-resource Indigenous languages remains limited. To this end, this paper presents a systematic evaluation of SFMs for ASR development in Kanyen'kÃ©ha, a polysynthetic Iroquoian language structurally and typologically distinct from mainstream languages. To address challenges posed by limited data and extensive vocabulary variation, we further investigate the impact of incorporating in-domain synthesized data and external language models during cross-lingual transfer learning. Experiments on the low-resource Kanyen'kÃ©ha corpus, under various train/test splits, show that the best system obtains a WER of 13.73% and a CER of 2.21% on the test set with a 59.2% OOV rate. Excluding easily correctable errors further reduces the WER and CER to 10.36% and 1.76%, demonstrating its potential to support language documentation and revitalization.
Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning
Authors: Debarpan Bhattacharya; Apoorva Kulkarni; Sriram Ganapathy
Venue: Interspeech 2025 (main)

Abstract: The popular success of text-based large language models (LLM) has streamlined the attention of the multimodal community to combine other modalities like vision and audio along with text to achieve similar multimodal capabilities. In this quest, large audio language models (LALMs) have to be evaluated on reasoning related tasks which are different from traditional classification or generation tasks. Towards this goal, we propose a novel dataset called temporal reasoning evaluation of audio (TREA). We benchmark open-source LALMs and observe that they are consistently behind human capabilities on the tasks in the TREA dataset. While evaluating LALMs, we also propose an uncertainty metric, which computes the invariance of the model to semantically identical perturbations of the input. Our analysis shows that the accuracy and uncertainty metrics are not necessarily correlated and thus, points to a need for wholesome evaluation of LALMs for high-stakes applications.1
Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples
Authors: Chun-Yi Kuan; Hung-yi Lee
Venue: Interspeech 2025 (main)

Abstract: Recent advancements in audio-aware large language models (ALLMs) enable them to process and understand audio inputs. However, these models often hallucinate non-existent sound events, reducing their reliability in real-world applications. To address this, we propose LISTEN (Learning to Identify Sounds Through Extended Negative Samples), a contrastive-like training method that enhances ALLMsâ ability to distinguish between present and absent sounds using synthesized data from the backbone LLM. Unlike prior approaches, our method requires no modification to LLM parameters and efficiently integrates audio representations via a lightweight adapter. Experiments show that LISTEN effectively mitigates hallucinations while maintaining impressive performance on existing audio question and reasoning benchmarks. At the same time, it is more efficient in both data and computation.
Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models
Authors: Ke-Han Lu; Chun-Yi Kuan; Hung-yi Lee
Venue: Interspeech 2025 (main)

Abstract: We introduce Speech-IFEval, an evaluation framework designed to assess instruction-following capabilities and quantify catastrophic forgetting in speech-aware language models (SLMs). Recent SLMs integrate speech perception with large language models (LLMs), often degrading textual capabilities due to speech-centric training. Existing benchmarks conflate speech perception with instruction-following, hindering evaluation of these distinct skills. To address this gap, we provide a benchmark for diagnosing the instruction-following abilities of SLMs. Our findings show that most SLMs struggle with even basic instructions, performing far worse than text-based LLMs. Additionally, these models are highly sensitive to prompt variations, often yielding inconsistent and unreliable outputs. We highlight core challenges and provide insights to guide future research, emphasizing the need for evaluation beyond task-level metrics.
Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models
Authors: Potsawee Manakul; Guangzhi Sun; Warit Sirichotedumrong; Kasima Tharnpipitchai; Kunat Pipatanakul
Venue: Interspeech 2025 (main)

Abstract: Audio language models process audio inputs using textual prompts for tasks like speech recognition and audio captioning. Although built on multilingual pre-trained components, most are trained primarily on English, limiting their usability for other languages. This paper evaluates audio language models on Thai, a low-resource language, and finds that they lack emergent cross-lingual abilities despite their multilingual foundations. To address this, we explore data mixtures that optimize audio language models for both a target language and English while integrating audio comprehension and speech instruction-following into a unified model. Our experiments provide insights into improving instruction-following in low-resource languages by balancing language-specific and multilingual training data. The proposed model, Typhoon-Audio, significantly outperforms existing open-source models and achieves performance comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai.
Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTC
Authors: Qingzheng Wang; Jiancheng Sun; Yifan Peng; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: Multilingual speech processing with self-supervised or supervised pre-trained Speech Foundation Models (SFM) has achieved strong performance on tasks like Language Identification (LID) and Automatic Speech Recognition (ASR). However, these models struggle with limited resources during fine-tuning. This paper enhances multilingual LID and ASR on ML-SUPERB 2.0 by exploring multiple strategies for adapting SFMs, including frozen upstream training, partial fine-tuning, and low-rank adaptation. Furthermore, we employ data augmentation to mitigate performance gaps in few-shot settings and introduce LID Connectionist Temporal Classification (CTC) loss for regularization. Our approach achieves a 14% relative improvement in LID accuracy and a 30% relative reduction in ASR CER over the baseline on ML-SUPERB 2.0, securing second place in the Interspeech 2025 ML-SUPERB 2.0 Challenge.
The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties
Authors: William Chen; Chutong Meng; Jiatong Shi; Martijn Bartelds; Shih-Heng Wang; Hsiu-Hsuan Wang; Rafael Mosquera; Sara Hincapie; Dan Jurafsky; Antonis Anastasopoulos; Hung-yi Lee; Karen Livescu; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: Recent improvements in multilingual ASR have not been equally distributed across languages and language varieties. To advance state-of-the-art (SOTA) ASR models, we present the Interspeech 2025 ML-SUPERB 2.0 Challenge. We construct a new test suite that consists of data from 200+ languages, accents, and dialects to evaluate SOTA multilingual speech models. The challenge also introduces an online evaluation server based on DynaBench, allowing for flexibility in model design and architecture for participants. The challenge received 5 submissions from 3 teams, all of which outperformed our baselines. The best-performing submission achieved an absolute improvement in LID accuracy of 23% and a reduction in CER of 18% when compared to the best baseline on a general multilingual test set. On accented and dialectal data, the best submission obtained 30.2% lower CER and 15.7% higher LID accuracy, showing the importance of community challenges in making speech technologies more inclusive.
TalTech Systems for the Interspeech 2025 ML-SUPERB 2.0 Challenge
Authors: Tanel AlumÃ¤e; Artem Fedorchenko
Venue: Interspeech 2025 (main)

Abstract: This paper describes the language identification and multilingual speech recognition system developed at Tallinn University of Technology for the Interspeech 2025 ML-SUPERB 2.0 Challenge. A hybrid language identification system is used, consisting of a pretrained language embedding model and a lightweight speech recognition model with a shared encoder across languages and language-specific bigram language models. For speech recognition, three models are used, where only a single model is applied for each language, depending on the training data availability and performance on held-out data. The model set consists of a finetuned version of SeamlessM4T, MMS-1B-all with custom language adapters and MMS-zeroshot. The system obtained the top overall score in the challenge.
Regularized Federated Learning for Privacy-Preserving Dysarthric and Elderly Speech Recognition
Authors: Tao Zhong; Mengzhe Geng; Shujie Hu; Guinan Li; Xunying Liu
Venue: Interspeech 2025 (main)

Abstract: Accurate recognition of dysarthric and elderly speech remains challenging to date. While privacy concerns have driven a shift from centralized approaches to federated learning (FL) to ensure data confidentiality, this further exacerbates the challenges of data scarcity, imbalanced data distribution and speaker heterogeneity. To this end, this paper conducts a systematic investigation of regularized FL techniques for privacy-preserving dysarthric and elderly speech recognition, addressing different levels of the FL process by 1) parameter-based, 2) embedding-based and 3) novel loss-based regularization. Experiments on the benchmark UASpeech dysarthric and DementiaBank Pitt elderly speech corpora suggest that regularized FL systems consistently outperform the baseline FedAvg system by statistically significant WER reductions of up to 0.55% absolute (2.13% relative). Further increasing communication frequency to one exchange per batch approaches centralized training performance.
Facilitating Personalized TTS for Dysarthric Speakers Using Knowledge Anchoring and Curriculum Learning
Authors: Yejin Jeon; Solee Im; Youngjae Kim; Gary Geunbae Lee
Venue: Interspeech 2025 (main)

Abstract: Dysarthric speakers experience substantial communication challenges due to impaired motor control of the speech apparatus, which leads to reduced speech intelligibility. This creates significant obstacles in dataset curation since actual recording of long, articulate sentences for the objective of training personalized TTS models becomes infeasible. Thus, the limited availability of audio data, in addition to the articulation errors that are present within the audio, complicates personalized speech synthesis for target dysarthric speaker adaptation. To address this, we frame the issue as a domain transfer task and introduce a knowledge anchoring framework that leverages a teacher-student model, enhanced by curriculum learning through audio augmentation. Experimental results show that the proposed zero-shot multi-speaker TTS model effectively generates synthetic speech with markedly reduced articulation errors and high speaker fidelity, while maintaining prosodic naturalness.
DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model
Authors: Xueyuan Chen; Dongchao Yang; Wenxuan Wu; Minglin Wu; Jing Xu; Xixin Wu; Zhiyong Wu; Helen Meng
Venue: Interspeech 2025 (main)

Abstract: Dysarthric speech reconstruction (DSR) aims to convert dysarthric speech into comprehensible speech while maintaining the speaker's identity. Despite significant advancements, existing methods often struggle with low speech intelligibility and poor speaker similarity. In this study, we introduce a novel diffusion-based DSR system that leverages a latent diffusion model to enhance the quality of speech reconstruction. Our model comprises: (i) a speech content encoder for phoneme embedding restoration via pre-trained self-supervised learning (SSL) speech foundation models; (ii) a speaker identity encoder for speaker-aware identity preservation by in-context learning mechanism; (iii) a diffusion-based speech generator to reconstruct the speech based on the restored phoneme embedding and preserved speaker identity. Through evaluations on the widely-used UASpeech corpus, our proposed model shows notable enhancements in speech intelligibility and speaker similarity.
Improved Intelligibility of Dysarthric Speech using Conditional Flow Matching
Authors: Shoutrik Das; Nishant Singh; Arjun Gangwar; S Umesh
Venue: Interspeech 2025 (main)

Abstract: Dysarthria is a neurological disorder that significantly impairs speech intelligibility, often rendering affected individuals unable to communicate effectively. This necessitates the development of robust dysarthric-to-regular speech conversion techniques. In this work, we investigate the utility and limitations of self-supervised learning (SSL) features and their quantized representations as an alternative to mel-spectrograms for speech generation. Additionally, we explore methods to mitigate speaker variability by generating clean speech in a single-speaker voice using features extracted from WavLM. To this end, we propose a fully non-autoregressive approach that leverages Conditional Flow Matching (CFM) with Diffusion Transformers to learn a direct mapping from dysarthric to clean speech. Our findings highlight the effectiveness of discrete acoustic units in improving intelligibility while achieving faster convergence compared to traditional mel-spectrogram-based approaches.