GenAI4ELab/papercli-papers huggingface dataset:
Exploring Pre-trained models on Ultrasound Modeling for Mice Autism Detection with Uniform Filter Bank and Attentive Scoring
Authors: Yuchen Song; Yucong Zhang; Ming Li
Venue: Interspeech 2025 (main)

Abstract: Genetically engineered mice, whose behaviors resemble those of individuals with Autism Spectrum Disorder (ASD), serve as valuable models for studying ASD through ultrasound vocalization (USV) analysis. In this paper, we investigate the effectiveness of pre-trained models in learning features of the USVs by fine-tuning. To bridge the gap between the pre-trained model and the inductive bias of the ultrasonic signal, we design a uniformly-spaced filter bank to reduce the dimension in the frequency domain. The extracted filter-bank energies of the ultrasonic spectrogram form a pseudo-spectrogram for pre-trained models. In the back-end, we employ an attentive frame-wise scoring method for classification, resulting in a comprehensive judgment. Experimental results demonstrate the effectiveness of our approach, achieving a segment-level Unweighted Average Recall (UAR) of 0.729 and a subject-level UAR of 0.882 on the validation set provided by the MADUV 2025 Challenge.
MADUV: The 1st INTERSPEECH Mice Autism Detection via Ultrasound Vocalization Challenge
Authors: Zijiang Yang; Meishu Song; Xin Jing; Haojie Zhang; Kun Qian; Bin Hu; Kota Tamada; Toru Takumi; BjÃ¶rn W. Schuller; Yoshiharu Yamamoto
Venue: Interspeech 2025 (main)

Abstract: The Mice Autism Detection via Ultrasound Vocalization (MADUV) Challenge introduces the first INTERSPEECH challenge focused on detecting Autism Spectrum Disorder (ASD) in mice through their vocalizations. Participants are tasked with developing models to automatically classify mice as either wild-type or ASD models based on recordings with a high sampling rate. Our baseline system employs a simple CNN-based model using three different features. Results demonstrate the feasibility of automated ASD detection, with the considered audible-range features achieving the best performance (UAR of 0.600 for segment-level and 0.625 for subject-level classification). This challenge bridges speech technology and biomedical research, offering opportunities to advance our understanding of ASD models through machine learning approaches. The findings suggest promising directions for vocalization analysis and highlight the potential value of audible and ultrasound vocalizations in ASD detection.
Significance of Time-Frequency preprocessing for automatic Ultrasonic Vocalization classification in Autism Spectrum Disorder model detection
Authors: Szymon SzmajdziÅski; Juliusz WÃ³jtowicz-Kruk; Ivan Ryzhankow; Åukasz Åazarski; Jakub Å»ak; WÅadysÅaw Åredniawa
Venue: Interspeech 2025 (main)

Abstract: Autism spectrum disorder (ASD) is a complex neurodevelopmental condition of unclear cause and varying severity, often studied using mice as animal models. To accurately distinguish between wild type and ASD model phenotypes, we present a deep learning approach that uses ultrasonic vocalization as input. The proposed method combines a simple model architecture, of convolutional and fully connected layers, with a high-resolution representation of auditory and ultrasound time-frequency patterns. Our approach surpasses baseline performance, achieving an unweighted average recall score of 0.806 on 30-second ultrasonic vocalization fragments. This work was conducted for the 1st INTERSPEECH Mice Autism Detection via Ultrasound Vocalization (MAD-UV) Challenge, where it achieved the highest score among all submitted solutions.
Robust Vocal Intensity Prediction: Overcoming Dataset Bias with Pretrained Deep Models
Authors: Quentin Le Tellier; Marc Evrard; Albert Rilliard; Jean-Sylvain LiÃ©nard
Venue: Interspeech 2025 (main)

Abstract: Vocal intensity prediction has been investigated in previous studies, where machine-learning (ML) models (e.g., linear regression, SVM) trained on basic speech features (such as spectrograms and Mel-spectrograms) have demonstrated good prediction accuracy at the utterance level. In this work, we revisit these methods by evaluating them on two calibrated datasets, including a bilingual corpus containing both French and English speech. Our findings show that prior approaches struggle to generalize across datasets. To address this limitation, we leverage embeddings from the Wav2Vec 2.0 model as input features to classical ML regressors. While previous work used such embeddings for vocal intensity classification, our study demonstrates their effectiveness in addressing cross-dataset generalization issues for the regression task. The results confirm that pretrained embeddings significantly improve generalization and mitigate overfitting issues linked to traditional acoustic features.
SLASH: Self-Supervised Speech Pitch Estimation Leveraging DSP-derived Absolute Pitch
Authors: Ryo Terashima; Yuma Shirahata; Masaya Kawamura
Venue: Interspeech 2025 (main)

Abstract: We present SLASH, a pitch estimation method of speech signals based on self-supervised learning (SSL). To enhance the performance of conventional SSL-based approaches that primarily depend on the relative pitch difference derived from pitch shifting, our method incorporates absolute pitch values by 1) introducing a prior pitch distribution derived from digital signal processing (DSP), and 2) optimizing absolute pitch through gradient descent with a loss between the target and differentiable DSP-derived spectrograms. To stabilize the optimization, a novel spectrogram generation method is used that skips complicated waveform generation. In addition, the aperiodic components in speech are accurately predicted through differentiable DSP, enhancing the method's applicability to speech signal processing. Experimental results showed that the proposed method outperformed both baseline DSP and SSL-based pitch estimation methods, attributed to the effective integration of SSL and DSP.
PruneSLU: Efficient On-device Spoken Language Understanding through Vocabulary and Structural Pruning
Authors: Truong Do; Minh-Phuong Nguyen; Le -Minh Nguyen
Venue: Interspeech 2025 (main)

Abstract: Recent advancements in Spoken Language Understanding (SLU) have been driven by pre-trained speech processing models. However, deploying these models on resource-constrained devices remains challenging due to their large parameter sizes. This paper presents PruneSLU, a new method for compressing pre-trained SLU models while maintaining performance. Our approach combines vocabulary pruning and structural layer-wise pruning to reduce model size while preserving essential knowledge. After pruning, the model undergoes knowledge refinement using integration distillation and contrastive learning. Experiments on the STOP and SLURP datasets demonstrate that PruneSLU compresses a 39M model to 15M while retaining 98\% of its original performance, outperforming previous compression techniques.
Leveraging LLMs for Written to Spoken Style Data Transformation to Enhance Spoken Dialog State Tracking
Authors: Haris Gulzar; Monikka Roslianna Busto; Akiko Masaki; Takeharu Eda; Ryo Masumura
Venue: Interspeech 2025 (main)

Abstract: Dialog State Tracking (DST) is an important part of Task-Oriented Dialog (TOD) systems, as it needs to navigate the complex human conversational flow to accomplish a task. Most TOD systems are trained on written-style text data, and their performance plunges when deployed in spoken scenarios due to natural disfluencies and human-speech recognition errors. Labeled spoken-style TOD data is limited because of the high data collection cost and privacy concerns. As Large Language Models (LLMs) emerge as a tool for synthetic text data generation, we explored their capability to generate spoken-style text-based TOD data. Through meticulously crafting LLM prompts, our generated labeled spoken style TOD data improved the absolute Joint Goal Accuracy (JGA) by 3.39% and relative JGA by 11.6%, for dedicated DST models. In this work, we showcase our divide-and-conquer-based data generation strategies and DST training to improve the performance of task-specific dialog models.
Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs
Authors: Å imon SedlÃ¡Äek; Bolaji Yusuf; JÃ¡n Å vec; Pradyoth Hegde; Santosh Kesiraju; OldÅich Plchot; Jan ÄernockÃ½
Venue: Interspeech 2025 (main)

Abstract: In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.
What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systems
Authors: Kiyotada Mori; Seiya Kawano; Chaoran Liu; Carlos Toshinori Ishi; Angel GarcÃ­a Contreras; Koichiro Yoshino
Venue: Interspeech 2025 (main)

Abstract: Spoken dialogue systems (SDSs) utilize automatic speech recognition (ASR) at the front end of their pipeline. The role of ASR in SDSs is to recognize information in user speech related to response generation appropriately. Examining selective listening of humans, which refers to the ability to focus on and listen to important parts of a conversation during the speech, will enable us to identify the ASR capabilities required for SDSs and evaluate them. In this study, we experimentally confirmed selective listening when humans generate dialogue responses by comparing human transcriptions for generating dialogue responses and reference transcriptions. Based on our experimental results, we discuss the possibility of a new ASR evaluation method that leverages human selective listening, which can identify the gap between transcription ability between ASR systems and humans.
SpeechDialogueFactory: A Framework for Natural Speech Dialogue Generation
Authors: Minghan Wang; Ye Bai; Yuxia Wang; Thuy-Trang Vu; Ehsan Shareghi; Gholamreza Haffari
Venue: Interspeech 2025 (main)

Abstract: High-quality speech conversational datasets are essential for developing and evaluating Speech-LLMs. However, collecting real-world recordings presents significant challenges including high costs, privacy concerns, and inconsistent quality, while existing synthetic approaches often lack authenticity due to limited acoustic variety and insufficient paralinguistic information. We present SpeechDialogueFactory, a framework that addresses these limitations through a three-stage pipeline: generating comprehensive metadata, creating detailed scripts, and producing utterances enriched with paralinguistic features. Our framework retrieves speaker voices from a voice bank and leverages paralinguistic tags for expressive TTS. We also introduce an automated evaluation protocol that shows strong correlation with human assessments. Experimental results demonstrate that our synthesized dialogues achieve quality comparable to human recordings while offering greater flexibility and control.
Who, When, and What: Leveraging the ``Three Ws'' Concept for Emotion Recognition in Conversation
Authors: Xiaohan Shi; Xingfeng Li; Tomoki Toda
Venue: Interspeech 2025 (main)

Abstract: Emotion Recognition in Conversation (ERC) is essential for dialogue systems in human-computer interaction. Most existing studies primarily focus on modeling contextual information from historical interactions but often overlook the effective integration of speaker and content information. To address these challenges, we propose the "Three Ws" concept -Who, When, and What, representing speaker, context, and content information- to comprehensively capture emotional cues from historical interactions. Building on this concept, we further introduce a novel model for ERC. Additionally, we incorporate a speaker similarity loss to enhance speaker information. Experimental results show that our model outperforms baselines, with each component making significant contributionsâespecially context information. Additionally, the speaker similarity loss further improves ERC performance. Notably, the "Three Ws"concept demonstrates robustness across both single-modal and multimodal scenarios.
``Alexa, can you forget me?'' Machine Unlearning Benchmark in Spoken Language Understanding
Authors: Alkis Koudounas; Claudio Savelli; Flavio Giobergia; Elena Baralis
Venue: Interspeech 2025 (main)

Abstract: Machine unlearning, the process of efficiently removing specific information from machine learning models, is a growing area of interest for responsible AI. However, few studies have explored the effectiveness of unlearning methods on complex tasks, particularly speech-related ones. This paper introduces UnSLU-BENCH, the first benchmark for machine unlearning in spoken language understanding (SLU), focusing on four datasets spanning four languages. We address the unlearning of data from specific speakers as a way to evaluate the quality of potential "right to be forgotten" requests. We assess eight unlearning techniques and propose a novel metric to simultaneously better capture their efficacy, utility, and efficiency. UnSLU-BENCH sets a foundation for unlearning in SLU and reveals significant differences in the effectiveness and computational feasibility of various techniques.
Evaluating Large Language Models in Data Generation for Low-Resource Scenarios: A Case Study on Question Answering
Authors: Ebru Arisoy; Merve Unlu Menevse; Yusufcan Manav; Arzucan Ozgur
Venue: Interspeech 2025 (main)

Abstract: Large Language Models (LLMs) are powerful tools for generating synthetic data, offering a promising solution to data scarcity in low-resource scenarios. This study evaluates the effectiveness of LLMs in generating question-answer pairs to enhance the performance of question answering (QA) models trained with limited annotated data. While synthetic data generation has been widely explored for text-based QA, its impact on spoken QA remains underexplored. We specifically investigate the role of LLM-generated data in improving spoken QA models, showing performance gains across both text-based and spoken QA tasks. Experimental results on subsets of the SQuAD, Spoken SQuAD, and a Turkish spoken QA dataset demonstrate significant relative F1 score improvements of 7.8%, 7.0%, and 2.7%, respectively, over models trained solely on restricted human-annotated data. Furthermore, our findings highlight the robustness of LLM-generated data in spoken QA settings, even in the presence of noise.
I want a horror â comedy â movie: Slips-of-the-Tongue Impact Conversational Recommender System Performance
Authors: Maria Teleki; Lingfeng Shi; Chengkai Liu; James Caverlee
Venue: Interspeech 2025 (main)

Abstract: Disfluencies are a characteristic of speech. We focus on the impact of a specific class of disfluency - whole-word speech substitution errors (WSSE) - on LLM-based conversational recommender system performance. We develop Syn-WSSE, a psycholinguistically-grounded framework for synthetically creating genre-based WSSE at varying ratios to study their impact on conversational recommender system performance. We find that LLMs are impacted differently: llama and mixtral have improved performance in the presence of these errors, while gemini, gpt-4o, and gpt-4o-mini have deteriorated performance. We hypothesize that this difference in model resiliency is due to differences in the pre- and post-training methods and data, and that the increased performance is due to the introduced genre diversity. Our findings indicate the importance of a careful choice of LLM for these systems, and more broadly, that disfluencies must be carefully designed for as they can have unforeseen impacts.
Towards a Japanese Full-duplex Spoken Dialogue System
Authors: Atsumoto Ohashi; Shinya Iizuka; Jingjing Jiang; Ryuichiro Higashinaka
Venue: Interspeech 2025 (main)

Abstract: Full-duplex spoken dialogue systems, which can model simultaneous bidirectional features of human conversations such as speech overlaps and backchannels, have attracted significant attention recently. However, the study of full-duplex spoken dialogue systems for the Japanese language has been limited, and the research on their development in Japanese remains scarce. In this paper, we present the first publicly available full-duplex spoken dialogue model in Japanese, which is built upon Moshi, a full-duplex dialogue model in English. Our model is trained through a two-stage process: pre-training on a large-scale spoken dialogue data in Japanese, followed by fine-tuning on high-quality stereo spoken dialogue data. We further enhance the model's performance by incorporating synthetic dialogue data generated by a multi-stream text-to-speech system. Evaluation experiments demonstrate that the trained model outperforms Japanese baseline models in both naturalness and meaningfulness.
SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information
Authors: Chih-Kai Yang; Neo Ho; Yen-Ting Piao; Hung-yi Lee
Venue: Interspeech 2025 (main)

Abstract: Large audio-language models (LALMs) extend the large language models with multimodal understanding in speech, audio, etc. While their performances on speech and audio-processing tasks are extensively studied, their reasoning abilities remain underexplored. Particularly, their multi-hop reasoning, the ability to recall and integrate multiple facts, lacks systematic evaluation. Existing benchmarks focus on general speech and audio-processing tasks, conversational abilities, and fairness, but overlook this aspect. To bridge this gap, we introduce SAKURA, a benchmark assessing LALMsâ multi-hop reasoning based on speech and audio information. Results show that LALMs struggle to integrate speech/audio representations for multi-hop reasoning, even when they extract the relevant information correctly, highlighting a fundamental challenge in multimodal reasoning. Our findings expose a critical limitation in LALMs, offering insights and resources for future research.
Continual Speech Learning with Fused Speech Features
Authors: Guitao Wang; Jinming Zhao; Hao Yang; Guilin Qi; Tongtong Wu; Gholamreza Haffari
Venue: Interspeech 2025 (main)

Abstract: Rapid growth in speech data demands adaptive models, as traditional static methods fail to keep pace with dynamic and diverse speech information. We introduce continuous speech learning, a new set-up targeting at bridging the adaptation gap in current speech models. We use the encoder-decoder Whisper model to standardize speech tasks into a generative format. We integrate a learnable gated-fusion layer on the top of the encoder to dynamically select task-specific features for downstream tasks. Our approach improves accuracy significantly over traditional methods in six speech processing tasks, demonstrating gains in adapting to new speech tasks without full retraining.
Uni-VERSA: Versatile Speech Assessment with a Unified Network
Authors: Jiatong Shi; Hye-jin Shim; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: Subjective listening tests remain the golden standard for speech quality assessment, but are costly, variable, and difficult to scale. In contrast, existing objective metrics, such as PESQ, F0 correlation, and DNSMOS, typically capture only specific aspects of speech quality. To address these limitations, we introduce Uni-VERSA, a unified network that simultaneously predicts various objective metrics, encompassing naturalness, intelligibility, speaker characteristics, prosody, and noise, for a comprehensive evaluation of speech signals. We formalize its framework, evaluation protocol, and applications in speech enhancement, synthesis, and quality control. A benchmark based on the URGENT24 challenge, along with a baseline leveraging self-supervised representations, demonstrates that Uni-VERSA provides a viable alternative to single-aspect evaluation methods. Moreover, it aligns closely with human perception, making it a promising approach for future speech quality assessment.
Evaluating ASR Robustness to Spontaneous Speech Errors: A Study of WhisperX Using a Speech Error Database
Authors: John Alderete; Macarious Kin Fung Hui; Aanchan Mohan
Venue: Interspeech 2025 (main)

Abstract: The Simon Fraser University Speech Error Database (SFUSED) is a public data collection developed for linguistic and psycholinguistic research. Here we demonstrate how its design and annotations can be used to test and evaluate speech recognition models. The database comprises systematically annotated speech errors from spontaneous English speech, with each error tagged for intended and actual error productions. The annotation schema incorporates multiple classificatory dimensions that are of some value to model assessment, including linguistic hierarchical level, contextual sensitivity, degraded words, word corrections, and both word-level and syllable-level error positioning. To assess the value of these classificatory variables, we evaluated the transcription accuracy of WhisperX across 5,300 documented word and phonological errors. This analysis demonstrates the database's effectiveness as a diagnostic tool for ASR system performance.
Is Synthetic Data Truly Effective for Training Speech Language Models?
Authors: Tomoya Mizumoto; Atsushi Kojima; Yusuke Fujita; Lianbo Liu; Yui Sudo
Venue: Interspeech 2025 (main)

Abstract: The development of Large Language Models (LLMs) has expanded beyond text-based tasks to speech applications such as Automatic Speech Recognition (ASR) and Automated Speech Translation (AST). However, training speech language models based on LLMs requires large-scale datasets, which are challenging to construct. To address data scarcity, previous studies have explored synthetic data generation using ASR for transcribing unlabeled speech and Text-to-Speech (TTS) for generating speech from text. While synthetic data enables large-scale dataset construction without human intervention, concerns persist regarding quality degradation and its impact on model performance. This study investigates the effects of synthetic data on ASR and AST tasks. Experimental results indicate that synthetic data alone may degrade performance, whereas combining it with real data can enhance performance, demonstrating its potential when integrated with other data sources.
How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not
Authors: Francesco Verdini; Pierfrancesco Melucci; Stefano Perna; Francesco Cariaggi; Marco Gaido; Sara Papi; Szymon Mazurek; Marek Kasztelnik; Luisa Bentivogli; Sebastien BratiÃ¨res; Paolo Merialdo; Simone Scardapane
Venue: Interspeech 2025 (main)

Abstract: The remarkable performance achieved by Large Language Models (LLM) has driven research efforts to leverage them for a wide range of tasks and input modalities. In speech-to-text (S2T) tasks, the emerging solution consists of projecting the output of the encoder of a Speech Foundational Model (SFM) into the LLM embedding space through an adapter module. However, no work has yet investigated how much the downstream-task performance depends on each component (SFM, adapter, LLM) nor whether the best design of the adapter depends on the chosen SFM and LLM. To fill this gap, we evaluate the combination of 5 adapter modules, 2 LLMs (Mistral and Llama), and 2 SFMs (Whisper and SeamlessM4T) on two widespread S2T tasks, namely Automatic Speech Recognition and Speech Translation. Our results demonstrate that the SFM plays a pivotal role in downstream performance, while the adapter choice has moderate impact and depends on the SFM and LLM.