GenAI4ELab/papercli-papers huggingface dataset:
Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model
Authors: Yong Ren; Chenxing Li; Le Xu; Hao Gu; Duzhen Zhang; Yujie Chen; Manjie Xu; Ruibo Fu; Shan Yang; Dong Yu
Venue: Interspeech 2025 (main)

Abstract: Humans can intuitively infer sounds from silent videos, but whether multimodal large language models can perform modal-mismatch reasoning without accessing target modalities remains relatively unexplored. Current text-assisted-video-to-audio (VT2A) methods excel in video foley tasks but struggle to acquire audio descriptions during inference. We introduce the task of Reasoning Audio Descriptions from Silent Videos (SVAD) to address this challenge and investigate vision-language models' (VLMs) capabilities on this task. To further enhance the VLMs' reasoning capacity for the SVAD task, we construct a CoT-AudioCaps dataset and propose a Chain-of-Thought-based supervised fine-tuning strategy. Experiments on SVAD and subsequent VT2A tasks demonstrate our method's effectiveness in two key aspects: significantly improving VLMs' modal-mismatch reasoning for SVAD and effectively addressing the challenge of acquiring audio descriptions during VT2A inference.
Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning
Authors: Le Xu; Chenxing Li; Yong Ren; Yujie Chen; Yu Gu; Ruibo Fu; Shan Yang; Dong Yu
Venue: Interspeech 2025 (main)

Abstract: Current vision-guided audio captioning systems frequently fail to address audiovisual misalignment in real-world scenarios, such as dubbed content or off-screen sounds. To bridge this critical gap, we present an entropy-aware gated fusion framework that dynamically modulates visual information flow through cross-modal uncertainty quantification. Our novel approach employs attention entropy analysis in cross-attention layers to automatically identify and suppress misleading visual cues during modal fusion. Complementing this architecture, we develop a batch-wise audiovisual shuffling technique that generates synthetic mismatched training pairs, greatly enhancing model resilience against alignment noise. Evaluations on the AudioCaps benchmark demonstrate our system's superior performance over existing baselines, especially in mismatched modality scenarios. Furthermore, our solution demonstrates an approximately 6x improvement in inference speed compared to the baseline.
ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition
Authors: Thai-Binh Nguyen; Thi Van Nguyen; Quoc Truong Do; Chi Mai Luong
Venue: Interspeech 2025 (main)

Abstract: Audio-Visual Speech Recognition (AVSR) has gained significant attention recently due to its robustness against noise, which often challenges conventional speech recognition systems that rely solely on audio features. Despite this advantage, AVSR models remain limited by the scarcity of extensive datasets, especially for most languages beyond English. Automated data collection offers a promising solution. This work presents a practical approach to generate AVSR datasets from raw video, refining existing techniques for improved efficiency and accessibility. We demonstrate its broad applicability by developing a baseline AVSR model for Vietnamese. Experiments show the automatically collected dataset enables a strong baseline, achieving competitive performance with robust ASR in clean conditions and significantly outperforming them in noisy environments like cocktail parties. This efficient method provides a pathway to expand AVSR to more languages, particularly under-resourced ones.
GALAXY: A Large-Scale Open-Domain Dataset for Multimodal Learning
Authors: Yihan Wu; Yichen Lu; Yijing Chen; Jiaqi Song; William Chen; Ruihua Song; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: Humans naturally use multimodal information, with vision, speech, and text working together to understand the world and solve problems. For artificial intelligence to achieve human-level capability, it must process multimodal information in a similar manner. However, there is a lack of large-scale open-domain datasets that support all three modalitiesâvision, speech, and textâwith high-quality speech transcriptions. To address this gap, we introduce GALAXY, a large-scale, open-domain dataset designed for multimodal learning, containing 8,270 hours of videos, speech, and transcriptions across 16 diverse domains. We describe the data creation pipeline and provide detailed statistics and analyses of the dataset. Using multimodal speech recognition as a case study, we validate GALAXYâs effectiveness and evaluate baseline modelsâ performance across different data volumes and domains. The results highlight GALAXYâs potential as a valuable resource for advancing multimodal understanding.
FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems
Authors: Yizhou Peng; Yi-Wen Chao; Dianwen Ng; Yukun Ma; Chongjia Ni; Bin Ma; Eng Siong Chng
Venue: Interspeech 2025 (main)

Abstract: Full-duplex spoken dialogue systems (FDSDS) enable more natural humanâmachine interactions by allowing real-time user interruptions and backchanneling, compared to traditional SDS that rely on turn-taking. However, existing benchmarks lack metrics for FD scenes, e.g., evaluating model performance during user interruptions. In this paper, we present a comprehensive FD benchmarking pipeline utilizing LLMs, TTS, and ASR to address this gap. It assesses FDSDSâs ability to handle user interruptions, manage delays, and maintain robustness in challenging scenarios with diverse novel metrics. We applied our benchmark to three open-source FDSDS (Moshi, Freeze-omni, and VITA-1.5) using over 40 hours of generated speech, with 293 simulated conversations and 1,200 interruptions. The results show that all models continue to face challenges, such as failing to respond to user interruptions, under frequent disruptions and noisy conditions. Demonstrations, data, and code will be released.
PersonaTAB: Predicting Personality Traits using Textual, Acoustic, and Behavioral Cues in Fully-Duplex Speech Dialogs
Authors: Sho Inoue; Shuai Wang; Haizhou Li
Venue: Interspeech 2025 (main)

Abstract: Despite significant progress in neural spoken dialog systems, personality-aware conversation agents---capable of adapting behavior based on personalities---remain underexplored due to the absence of personality annotations in speech datasets. We propose a pipeline that preprocesses raw audio recordings to create a dialogue dataset annotated with timestamps, response types, and emotion/sentiment labels.  We employ an automatic speech recognition (ASR) system to extract transcripts and timestamps, then generate conversation-level annotations. Leveraging these annotations, we design a system that employs large language models to predict conversational personality. Human evaluators were engaged to identify conversational characteristics and assign personality labels. Our analysis demonstrates that the proposed system achieves stronger alignment with human judgments compared to existing approaches.
FFD: Fine-Finger Diffusion Model for Music to Fine-grained Finger Dance Generation
Authors: Boya Dong; Wentao Lei; Li Liu
Venue: Interspeech 2025 (main)

Abstract: Finger dance is an emerging social media trend using finger gesture motions for expression. Music to finger dance generation is challenging due to its fine-grained movements. Existing music-driven methods often fail to model subtle finger motions, yielding poor performances. We propose Fine-Finger Diffusion (FFD), the first end-to-end framework for music to finger dance generation. Our method employs a diffusion model to create rhythmically aligned finger movements while ensuring motion stability. A novel detail-aware loss (DAL) enhances temporal coherence by constraining inter-frame motion fluctuations. We introduce DanceFingers-4K, the first large-scale finger dance dataset containing 4007 video clips with music-motion pairs. Comprehensive evaluations demonstrate FFD's superiority over existing approaches across objective metrics and user study.
Towards Diverse and Efficient Audio Captioning via Diffusion Models
Authors: Manjie Xu; Chenxing Li; Yong Ren; Xinyi Tu; Ruibo Fu; Wei Liang; Dong Yu
Venue: Interspeech 2025 (main)

Abstract: We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive diffusion model tailored for diverse and efficient audio captioning. Although existing captioning models relying on language backbones have achieved remarkable success in various captioning tasks, their insufficient performance in terms of generation speed and diversity impedes progress in audio understanding and multimedia applications. Our diffusion-based framework offers unique advantages stemming from its inherent stochasticity and holistic context modeling in captioning. Through rigorous evaluation, we demonstrate that DAC not only achieves superior performance levels compared to existing benchmarks in the caption quality, but also significantly outperforms them in terms of generation speed and diversity.
Pull It Together: Reducing the Modality Gap in Contrastive Learning
Authors: Amit Sofer; Yoav Goldman; Shlomo E. Chazan
Venue: Interspeech 2025 (main)

Abstract: Contrastive learning has become a powerful strategy for aligning different modalities in a shared embedding space. Contrastive LanguageâImage Pre-training (CLIP) has achieved remarkable performance across various downstream tasks. This methodology has been extended to the audio-text domain through Contrastive LanguageâAudio Pre-training (CLAP), demonstrating strong performance in related tasks. However, recent work highlights a modality gap in CLIPâs embedding space, where embeddings from different modalities remain partially separated rather than fully integrated. In this paper, we begin by analyzing the CLAP embedding space and identify a similar modality gap. Furthermore, we propose a novel solution combining a modality classifier with a Gradient Reverse Layer (GRL) to reduce this gap. Our experiments on CLIP and CLAP confirm that our approach reduces the modality gap while improving performance, and even achieving new State Of The Art (SOTA) results in text-audio retrieval.
EnvSDD: Benchmarking Environmental Sound Deepfake Detection
Authors: Han Yin; Yang Xiao; Rohan Kumar Das; Jisheng Bai; Haohe Liu; Wenwu Wang; Mark D Plumbley
Venue: Interspeech 2025 (main)

Abstract: Audio generation systems now create very realistic soundscapes that can enhance media production, but also pose potential risks. Several studies have examined deepfakes in speech or singing voice. However, environmental sounds have different characteristics, which may make methods for detecting speech and singing deepfakes less effective for real-world sounds. In addition, existing datasets for environmental sound deepfake detection are limited in scale and audio types. To address this gap, we introduce EnvSDD, the first large-scale curated dataset designed for this task, consisting of 45.25 hours of real and 316.74 hours of fake audio. The test set includes diverse conditions to evaluate the generalizability, such as unseen generation models and unseen datasets. We also propose an audio deepfake detection system, based on a pre-trained audio foundation model. Results on EnvSDD show that our proposed system outperforms the state-of-the-art systems from speech and singing domains.
Echoes of Phonetics: Unveiling Relevant Acoustic Cues for ASR via Feature Attribution
Authors: Dennis Fucci; Marco Gaido; Matteo Negri; Mauro Cettolo; Luisa Bentivogli
Venue: Interspeech 2025 (main)

Abstract: Despite significant advances in ASR, the specific acoustic cues models rely on remain unclear. Prior studies have examined such cues on a limited set of phonemes and outdated models. In this work, we apply a feature attribution technique to identify the relevant acoustic cues for a modern Conformer-based ASR system. By analyzing plosives, fricatives, and vowels, we assess how feature attributions align with their acoustic properties in the time and frequency domains, also essential for human speech perception. Our findings show that the ASR model relies on vowelsâ full time spans, particularly their first two formants, with greater saliency in male speech. It also better captures the spectral characteristics of sibilant fricatives than non-sibilants and prioritizes the release phase in plosives, especially burst characteristics. These insights enhance the interpretability of ASR models and highlight areas for future research to uncover potential gaps in model robustness.
Benchmarking Time-localized Explanations for Audio Classification Models
Authors: Cecilia BolaÃ±os; Leonardo Pepino; Martin Meza; Luciana Ferrer
Venue: Interspeech 2025 (main)

Abstract: Most modern approaches for audio processing are opaque, in the sense that they do not provide an explanation for their decisions. For this reason, various methods have been proposed to explain the outputs generated by these models. Good explanations can result in interesting insights about the data or the model, as well as increase trust in the system. Unfortunately, evaluating the quality of explanations is far from trivial since, for most tasks, there is no clear ground truth explanation to use as reference. In this work, we propose a benchmark for time-localized explanations for audio classification models that uses time annotations of target events as a proxy for ground truth explanations. We use this benchmark to systematically optimize and compare various approaches for model-agnostic post-hoc explanation, obtaining, in some cases, close to perfect explanations. Finally, we illustrate the utility of the explanations for uncovering spurious correlations.
Spectrotemporal Modulation: Efficient and Interpretable Feature Representation for Classifying Speech, Music, and Environmental Sounds
Authors: Andrew Chang; Yike Li; Iran R. Roman; David Poeppel
Venue: Interspeech 2025 (main)

Abstract: Audio DNNs have demonstrated impressive performance on various machine listening tasks; however, most of their representations are computationally costly and uninterpretable, leaving room for optimization. Here, we propose a novel approach centered on spectrotemporal modulation (STM) features, a signal processing method that mimics the neurophysiological representation in the human auditory cortex. The classification performance of our STM-based model, without any pretraining, is comparable to that of pretrained audio DNNs across diverse naturalistic speech, music, and environmental sounds, which are essential categories for both human cognition and machine perception. These results show that STM is an efficient and interpretable feature representation for audio classification, advancing the development of machine listening and unlocking exciting new possibilities for basic understanding of speech and auditory sciences, as well as developing audio BCI and cognitive computing.
Discrete Tokens Exhibit Interlanguage Speech Intelligibility Benefit: an Analytical Study Towards Accent-robust ASR Only with Native Speech Data
Authors: Kentaro Onda; Keisuke Imoto; Satoru Fukayama; Daisuke Saito; Nobuaki Minematsu
Venue: Interspeech 2025 (main)

Abstract: In this study, we gained insight that contributes to achieving accent-robust ASR using only native speech data. In human perception of non-native speech, the phenomenon known as "interlanguage speech intelligibility beneï¬t" (ISIB) is observed, where non-native listeners who share the native language with the speaker understand the speech better compared even to native listeners. Based on the idea that discrete tokens extracted from self-supervised learning (SSL) models represent the human perception of speech, we conducted an analytical study on the robustness of discrete token-based ASR to non-native speech, varying the language used for training the tokenization, which is viewed as a technical implementation of ISIB. The results showed that ISIB actually occurred in the discrete token-based ASR. Since our approach relies only on native speech data to simulate the behavior of human perception, it is expected to be applicable to a wide range of accents for which speech data is scarce.
Analysis of Semantic and Acoustic Token Variability Across Speech, Music, and Audio Domains
Authors: Takanori Ashihara; Marc Delcroix; Tsubasa Ochiai; Kohei Matsuura; Shota Horiguchi
Venue: Interspeech 2025 (main)

Abstract: Techniques for discrete audio representation, which convert an audio signal into a sequence of audio tokens using neural audio codecs or self-supervised speech models, have gained attention for offering the possibility of modeling audio with large language models (LM) efficiently. While these audio tokens have been studied in various domains (e.g., speech, music, and general sound), their encoding properties across domains remain unclear. This paper examines several audio token types to analyze cross-domain variations. Our major findings include that audio tokens exhibit consistent statistical structures and probabilistic predictability deduced from rank-frequency distribution and perplexity, regardless of the domain. However, the token usage pattern is somewhat domain-dependent. This result underpins the steady success of the versatile audio LM, while also suggesting that domain-aware LM could further optimize performance by better capturing domain-specific token usage distributions.
Is your model big enough? Training and interpreting large-scale monolingual speech foundation models
Authors: Yaroslav Getman; TamÃ¡s GrÃ³sz; Tommi Lehtonen; Mikko Kurimo
Venue: Interspeech 2025 (main)

Abstract: Self-supervised learning has been widely used in developing speech foundation models. Most languages, however, are only represented in multilingual foundations. We introduce monolingual self-supervised foundation models pre-trained on more than 150,000 hours of Finnish speech and propose a new interpretation technique to understand their capabilities. To our knowledge, this is the largest monolingual data used for self-supervised non-English speech representation learning. Our models demonstrate superior downstream low-resource ASR performance and improved generalization compared to prior work, with absolute WER reductions of up to 14%. Moreover, our proposed interpretation technique, Layer Utilization Rate (LUR), enables us to assess the percentage of neurons in each layer highly contributing towards the output. Empirical results show that the proposed LUR metric can be used to indicate the potential of the fine-tuned model's size and architecture to generalize to unseen domains.
Semantic-Aware Interpretable Multimodal Music Auto-Tagging
Authors: Andreas Patakis; Vassilis Lyberatos; Spyridon Kantarelis; Edmund Dervakos; Giorgos Stamou
Venue: Interspeech 2025 (main)

Abstract: Music auto-tagging is essential for organizing and discovering music in extensive digital libraries. While foundation models achieve exceptional performance in this domain, their outputs often lack interpretability, limiting trust and usability for researchers and end-users alike. In this work, we present an interpretable framework for music auto-tagging that leverages groups of musically meaningful multimodal features, derived from signal processing, deep learning, ontology engineering, and natural language processing. To enhance interpretability, we cluster features semantically and employ an expectation maximization algorithm, assigning distinct weights to each group based on its contribution to the tagging process. Our method achieves competitive tagging performance while offering a deeper understanding of the decision-making process, paving the way for more transparent and user-centric music tagging systems.
From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Models
Authors: Asim Ersoy; Basel Ahmad Mousi; Shammur Absar Chowdhury; Firoj Alam; Fahim I Dalvi; Nadir Durrani
Venue: Interspeech 2025 (main)

Abstract: The emergence of large language models has demonstrated that systems trained solely on text can acquire extensive world knowledge, develop reasoning capabilities, and internalize abstract semantic concepts - showcasing properties that can be associated with general intelligence. This raises an intriguing question: Do such concepts emerge in models trained on other modalities, such as speech? Furthermore, when models are trained jointly on multiple modalities: Do they develop a richer, more structured semantic understanding? To explore this, we analyze the conceptual structures learned by speech and textual models both individually and jointly. We employ Latent Concept Analysis, an unsupervised method for uncovering and interpreting latent representations in neural networks, to examine how semantic abstractions form across modalities. To support reproducibility, we have released our code along with a curated audio version of the SST-2 dataset for public access.
Effective Context in Neural Speech Models
Authors: Yen Meng; Sharon Goldwater; Hao Tang
Venue: Interspeech 2025 (main)

Abstract: Modern neural speech models benefit from having longer context, and many approaches have been proposed to increase the maximum context a model can use. However, few have attempted to measure how much context these models actually use, i.e., the effective context. Here, we propose two approaches to measuring the effective context, and use them to analyze different speech Transformers. For supervised models, we find that the effective context correlates well with the nature of the task, with fundamental frequency tracking, phone classification, and word classification requiring increasing amounts of effective context. For self-supervised models, we find that effective context increases mainly in the early layers, and remains relatively short---similar to the supervised phone model. Given that these models do not use a long context during prediction, we show that HuBERT can be run in streaming mode without modification to the architecture and without further fine-tuning.
Word stress in self-supervised speech models: A cross-linguistic comparison
Authors: Martijn Bentum; Louis ten Bosch; Tomas O. Lentz
Venue: Interspeech 2025 (main)

Abstract: In this paper we study word stress representations learned by self-supervised speech models (S3M), specifically the Wav2vec 2.0 model. We investigate the S3M representations of word stress for five different languages: Three languages with variable or lexical stress (Dutch, English and German) and two languages with fixed or demarcative stress (Hungarian and Polish). We train diagnostic stress classifiers on S3M embeddings and show that they can distinguish between stressed and unstressed syllables in read-aloud short sentences with high accuracy. We also tested language-specificity effects of S3M word stress.The results indicate that the word stress representations are language-specific, with a greater difference between the set of variable versus the set of fixed stressed languages.
What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-training
Authors: Marianne de Heer Kloots; Hosein Mohebbi; Charlotte Pouw; Gaofei Shen; Willem Zuidema; Martijn Bentum
Venue: Interspeech 2025 (main)

Abstract: How language-specific are speech representations learned by self-supervised models? Existing work has shown that a range of linguistic features can be successfully decoded from end-to-end models trained only on speech recordings. However, it's less clear to what extent pre-training on specific languages improves language-specific linguistic information. Here we test the encoding of Dutch phonetic and lexical information in internal representations of self-supervised Wav2Vec2 models. Pretraining exclusively on Dutch improves the representation of Dutch linguistic features as compared to pre-training on similar amounts of English or larger amounts of multilingual data. This language-specific advantage is well-detected by trained clustering or classification probes, and partially observable using zero-shot metrics. Furthermore, the language-specific benefit on linguistic feature encoding aligns with downstream performance on Automatic Speech Recognition.