GenAI4ELab/papercli-papers huggingface dataset:
Language-Agnostic Speech Tokenizer for Spoken Term Detection with Efficient Retrieval
Authors: Anup Singh; Kris Demuynck; Vipul Arora
Venue: Interspeech 2025 (main)

Abstract: The surge in multilingual and code-switched spoken content demands efficient Query-by-Example Spoken Term Detection (STD) systems capable of handling diverse languages. Existing STD systems are monolingual; they typically require large labeled datasets for training and use costly DTW-based matching during inference, limiting their practicality. This paper proposes a novel speech tokenizer that converts speech into language-agnostic tokens. Furthermore, a multi-stage search algorithm enables fast and efficient retrieval from large datasets. In experimental evaluations, the tokens from the proposed tokenizer demonstrate strong speaker invariance, consistent performance across languages, and a capability to generalize effectively to unseen languages, outperforming the baselines significantly.
H-QuEST: Accelerating Query-by-Example Spoken Term Detection with Hierarchical Indexing
Authors: Akanksha Singh; Yi-Ping Phoebe Chen; Vipul Arora
Venue: Interspeech 2025 (main)

Abstract: Query-by-example spoken term detection (QbE-STD) searches for matching words or phrases in an audio dataset using a sample spoken query. When annotated data is limited or unavailable, QbE-STD is often done using template matching methods like dynamic time warping (DTW), which are computationally expensive and canât scale well. To address this, we propose H-QuEST (Hierarchical Query-by-Example Spoken Term Detection), a novel framework that accelerates spoken term retrieval by utilizing Term Frequency and Inverse Document Frequency (TF-IDF)-based sparse representations obtained through advanced audio representation learning techniques and Hierarchical Navigable Small World (HNSW) indexing with further refinement to perform search. Experimental results show that H-QuEST delivers substantial improvements in retrieval speed, without sacrificing accuracy compared to existing methods.
Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval
Authors: Ruofan Hu; Yan Xia; Minjie Hong; Jieming Zhu; Bo Chen; Xiaoda Yang; Minghui Fang; Tao Jin
Venue: Interspeech 2025 (main)

Abstract: Multimodal large language models (MLLMs) have seen substantial progress in recent years. However, their ability to represent multimodal information in the acoustic domain remains underexplored. In this work, we introduce Vela, a novel framework designed to adapt MLLMs for the generation of universal multimodal embeddings. By leveraging MLLMs with specially crafted prompts and selected in-context learning examples, Vela effectively bridges the modality gap across various modalities. We then propose a single-modality training approach, where the model is trained exclusively on text pairs. Our experiments show that Vela outperforms traditional CLAP models in standard text-audio retrieval tasks. Furthermore, we introduce new benchmarks that expose CLAP models' limitations in handling long texts and complex retrieval tasks. In contrast, Vela, by harnessing the capabilities of MLLMs, demonstrates robust performance in these scenarios. Our code will soon be available.
Adversarial Deep Metric Learning for Cross-Modal Audio-Text Alignment in Open-Vocabulary Keyword Spotting
Authors: Youngmoon Jung; Yong-Hyeok Lee; Myunghun Jung; Jaeyoung Roh; Chang Woo Han; Hoon-Young Cho
Venue: Interspeech 2025 (main)

Abstract: For text enrollment-based open-vocabulary keyword spotting (KWS), acoustic and text embeddings are typically compared at either the phoneme or utterance level. To facilitate this, we optimize acoustic and text encoders using deep metric learning (DML), enabling direct comparison of multi-modal embeddings in a shared embedding space. However, the inherent heterogeneity between audio and text modalities presents a significant challenge. To address this, we propose Modality Adversarial Learning (MAL), which reduces the domain gap in heterogeneous modality representations. Specifically, we train a modality classifier adversarially to encourage both encoders to generate modality-invariant embeddings. Additionally, we apply DML to achieve phoneme-level alignment between audio and text, and conduct extensive comparisons across various DML objectives. Experiments on the Wall Street Journal (WSJ) and LibriPhrase datasets demonstrate the effectiveness of the proposed approach.
GTA: Towards Generative Text-To-Audio Retrieval via Multi-Scale Tokenizer
Authors: Minghui Fang; Shengpeng Ji; Jialong Zuo; Xize Cheng; Wenrui Liu; Xiaoda Yang; Ruofan Hu; Jieming Zhu; Zhou Zhao
Venue: Interspeech 2025 (main)

Abstract: Text-to-audio retrieval is a fundamental task in acoustic signal processing. Currently, mainstream approaches primarily employ a dual-tower architecture, independently encoding text and audio while performing similarity score matching. However, these methods struggle to maintain a uniform embedding space and the latency of score matching increases with corpus size. In this light, we propose GTA to move towards a generative text-to-audio retrieval paradigm. Specifically, we utilize a multi-scale audio tokenizer to embed audio semantics into identifiers, while incorporating a dual-alignment strategy to ensure consistency between audio and text semantics. Furthermore, we implement curriculum learning to bridge the gap between training and inference, guiding the model toward precise token-level generation. Extensive experiments demonstrate the effectiveness and robustness of GTA, further validating the feasibility of the generative text-to-audio retrieval paradigm.
Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning
Authors: Changin Choi; Sungjun Lim; Wonjong Rhee
Venue: Interspeech 2025 (main)

Abstract: Retrieval-augmented generation can improve audio captioning by incorporating relevant audio-text pairs from a knowledge base. Existing methods typically rely solely on the input audio as a unimodal retrieval query. In contrast, we propose Generation-Assisted Multimodal Querying, which generates a text description of the input audio to enable multimodal querying. This approach aligns the query modality with the audio-text structure of the knowledge base, leading to more effective retrieval. Furthermore, we introduce a novel progressive learning strategy that gradually increases the number of interleaved audio-text pairs to enhance the training process. Our experiments on AudioCaps, Clotho, and Auto-ACD demonstrate that our approach achieves state-of-the-art results across these benchmarks.
On Retrieval of Long Audios with Complex Text Queries
Authors: Ruochu Yang; Milind Rao; Harshavardhan Sundar; Anirudh Raju; Aparna Khare; Srinath Tankasala; Di He; Venkatesh Ravichandran
Venue: Interspeech 2025 (main)

Abstract: We consider a challenging problem of LARCQ (Long Audio Retrieval with Complex Queries), where audios to retrieve may be arbitrarily long and queries span multiple events. To solve it, we propose a novel pipeline systematically integrating multi-modal retrieval and ALM/LLM refining. At Steps 1 and 2, we introduce a chunking-aggregation method to retrieve candidate audios by constructing a similarity matrix. At Steps 3 and 4, audio captions are generated for retrieved candidates using ALMs, and the final audio is selected by comparing the query with generated captions through text LLMs/classifiers. Due to lack of benchmarks for LARCQ, we introduce Clotho-LARCQ and SoundDescs-LARCQ featuring long audios and complex queries. Our chunking-aggregation method achieves up to 67% R@1 and 40% R@5 gains. Incorporating ALM/LLM refining, our full pipeline achieves 21% R@1 on the original Clotho benchmark and up to 100% R@1 improvement on new LARCQ benchmarks.
SIDC-KWS: Efficient Spiking Inception-Dilated Conformer with Self-Attention for Keyword Spotting
Authors: Jin Gyo Lim; Seong Eun Kim
Venue: Interspeech 2025 (main)

Abstract: Recent deep learning advances have improved keyword spotting (KWS). However, as KWS is deployed on edge devices, energy efficiency remains a key challenge. Conventional deep neural networks offer high accuracy but require heavy computation, making them unsuitable for low-power use. To address this, we propose the Spiking Inception-Dilated Conformer for Keyword Spotting (SIDC-KWS), an energy-efficient transformer based on spiking neural networks (SNNs). By integrating an Inception-Dilated (ID) block and spike-based self-attention, SIDC-KWS maintains high accuracy while significantly reducing power consumption. Experiments on the Google Speech Commands V2 (GSC V2) dataset show that SIDC-KWS achieves 96.8% and 94.7% accuracy on 12-class and 35-class tasks, respectively. On the 35-class task, SIDC-KWS consumes 75.59% less energy than its ANN counterpart. These results underscore SNNs as a scalable, low-power alternative for real-time KWS in resource-limited environments.
Multichannel Keyword Spotting for Noisy Conditions
Authors: Dzmitry Saladukha; Ivan Koriabkin; Kanstantsin Artsiom; Aliaksei Rak; Nikita Ryzhikov
Venue: Interspeech 2025 (main)

Abstract: This article presents a method for improving a keyword spotter (KWS) algorithm in noisy environments. Although beamforming (BF) and adaptive noise cancellation (ANC) techniques are robust in some conditions, they may degrade the performance of the activation system by distorting or suppressing useful signals. The authors propose a neural network architecture that uses several input channels and an attention mechanism that allows the network to determine the most useful channel or their combination. The improved quality of the algorithm was demonstrated on two datasets: from a laboratory with controlled conditions and from smart speakers in natural conditions. The proposed algorithm was compared against several baselines in terms of the quality of noise reduction metrics, KWS metrics, and computing resources in comparison with existing solutions.
LLM-Synth4KWS: Scalable Automatic Generation and Synthesis of Confusable Data for Custom Keyword Spotting
Authors: Pai Zhu; Quan Wang; Dhruuv Agarwal; Kurt Partridge
Venue: Interspeech 2025 (main)

Abstract: Custom keyword spotting (KWS) allows detecting user-defined spoken keywords from streaming audio. This is achieved by comparing the embeddings from voice enrollments and input audio. State-of-the-art custom KWS models are typically trained contrastively using  utterances whose keywords are randomly sampled from training dataset. These KWS models often struggle with confusing keywords, such as ``blue'' versus ``glue''. This paper introduces an effective way to augment the training with confusable utterances where keywords are generated and grouped from large language models (LLMs), and speech signals are synthesized with diverse speaking styles from text-to-speech (TTS) engines. To better measure user experience on confusable KWS, we define a new northstar metric using the average area under DET curve from confusable groups (c-AUC). Featuring high scalability and zero labor cost, the proposed method improves AUC by 3.7% and c-AUC by 11.3% on the Speech Commands testing set.
GraphemeAug: A Systematic Approach to Synthesized Hard Negative Keyword Spotting Examples
Authors: Harry Zhang; Kurt Partridge; Pai Zhu; Neng Chen; Hyun Jin Park; Dhruuv Agarwal; Quan Wang
Venue: Interspeech 2025 (main)

Abstract: Spoken Keyword Spotting (KWS) is the task of distinguishing between the presence and absence of a keyword in audio. The accuracy of a KWS model hinges on its ability to correctly classify examples close to the keyword and non-keyword boundary. These boundary examples are often scarce in training data, limiting model performance. In this paper, we propose a method to systematically generate adversarial examples close to the decision boundary by making insertion/deletion/substitution edits on the keyword's graphemes. We evaluate this technique on held-out data for a popular keyword and show that the technique improves AUC on a dataset of synthetic hard negatives by 61% while maintaining quality on positives and ambient negative audio data.
SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs
Authors: Firoj Alam; Md Arid Hasan; Shammur Absar Chowdhury
Venue: Interspeech 2025 (main)

Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across various disciplines and tasks. However, benchmarking their capabilities with multilingual spoken queries remains largely unexplored. In this study, we introduce SpokenNativQA, the first multilingual and culturally aligned spoken question-answering (SQA) dataset designed to evaluate LLMs in real-world conversational settings. The dataset comprises approximately 33,000 naturally spoken questions and answers in multiple languages, including low-resource and dialect-rich languages, providing a robust benchmark for assessing LLM performance in speech-based interactions. SpokenNativQA addresses the limitations of text-based QA datasets by incorporating speech variability, accents, and linguistic diversity. We benchmark different ASR systems and LLMs for SQA and present our findings. We released the data and the experimental scripts for the research community.
CAMER: Contribution-Aware Multimodal Emotion Recognition
Authors: Sun-Kyung Lee; Jong-Hwan Kim
Venue: Interspeech 2025 (main)

Abstract: Multimodal emotion recognition (MER) integrates data from speech, facial expressions, and text to enhance emotion prediction, with applications in various human-computer interaction scenarios. While recent end-to-end models improved performance, they typically lack interpretability regarding the role of each modality in predicting emotions. This paper introduces a contribution-aware MER (CAMER) with a novel adaptive weighting mechanism that dynamically adjusts the contribution of each modality based on the emotional content using cross-attention between modality features and emotion embeddings. This approach not only enhances the performance of MER by optimizing modality integration but also significantly improves the interpretability of the model's predictions. Evaluations on the IEMOCAP and CMU-MOSEI datasets demonstrate the superiority of our method. Additionally, we provide an interactive demo that allows users to explore and visualize the model's decision-making process.
GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints
Authors: Jiajun He; Jinyi Mi; Tomoki Toda
Venue: Interspeech 2025 (main)

Abstract: Multimodal emotion recognition (MER) extracts emotions from multimodal data, including visual, speech, and text inputs, playing a key role in human-computer interaction. Attention-based fusion methods dominate MER research, achieving strong classification performance. However, two key challenges remain: effectively extracting modality-specific features and capturing cross-modal similarities despite distribution differences caused by modality heterogeneity. To address these, we propose a gated interactive attention mechanism to adaptively extract modality-specific features while enhancing emotional information through pairwise interactions. Additionally, we introduce a modality-invariant generator to learn modality-invariant representations and constrain domain shifts by aligning cross-modal similarities. Experiments on IEMOCAP demonstrate that our method outperforms state-of-the-art MER approaches, achieving WA 80.7% and UA 81.3%.
SNIFR : Boosting Fine-Grained Child Harmful Content Detection Through Audio-Visual Alignment with Cascaded Cross-Transformer
Authors: Orchid Chetia Phukan; Mohd Mujtaba Akhtar; Girish; Swarup Ranjan Behera; Abu Osama Siddiqui; Sarthak Jain; Priyabrata Mallick; Jaya Sai Kiran Patibandla; Pailla Balakrishna Reddy; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2025 (main)

Abstract: As video-sharing platforms have grown over the past decade, child viewership has surged, increasing the need for precise detection of harmful content like violence or explicit scenes. Malicious users exploit moderation systems by embedding unsafe content in minimal frames to evade detection. While prior research has focused on visual cues and advanced such fine-grained detection, audio features remain underexplored. In this study, we embed audio cues with visual for fine-grained child harmful content detection and introduce SNIFR, a novel framework for effective alignment. SNIFR employs a transformer encoder for intra-modality interaction, followed by a cascaded cross-transformer for inter-modality alignment. Our approach achieves superior performance over unimodal and baseline fusion methods, setting a new state-of-the-art.
CNVSRC 2024: The Second Chinese Continuous Visual Speech Recognition Challenge
Authors: Zehua Liu; Xiaolou Li; Chen Chen; Lantian Li; Dong Wang
Venue: Interspeech 2025 (main)

Abstract: This paper presents the second Chinese Continuous Visual Speech Recognition Challenge (CNVSRC 2024), which builds on CNVSRC 2023 to advance research in Chinese Large Vocabulary Continuous Visual Speech Recognition (LVC-VSR). The challenge evaluates two test scenarios: reading in recording studios and Internet speech. CNVSRC 2024 uses the same datasets as its predecessor CNVSRC 2023, which involves CN-CVS for training and CNVSRC-Single/Multi for development and evaluation. However, CNVSRC 2024 introduced two key improvements: (1) a stronger baseline system, and (2) an additional dataset, CN-CVS2-P1, for open tracks to improve data volume and diversity. The new challenge has demonstrated several important innovations in data preprocessing, feature extraction, model design, and training strategies, further pushing the state-of-the-art in Chinese LVC-VSR. More details and resources are available at the official website.
PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association
Authors: Abdul Hannan; Muhammad Arslan Manzoor; Shah Nawaz; Muhammad Irzam Liaqat; Markus Schedl; Mubashir Noman
Venue: Interspeech 2025 (main)

Abstract: We study the task of learning association between faces and voices, which is gaining interest in the multimodal community lately. These methods suffer from the deliberate crafting of negative mining procedures as well as the reliance on the distant margin parameter. These issues are addressed by learning a joint embedding space in which orthogonality constraints are applied to the fused embeddings of faces and voices. However, embedding spaces of faces and voices possess different characteristics and require spaces to be aligned before fusing them. To this end, we propose a method that accurately aligns the embedding spaces and fuses them with an enhanced gated fusion thereby improving the performance of face-voice association. Extensive experiments on the VoxCeleb dataset reveals the merits of the proposed approach.
Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model
Authors: Ke Hu; Ehsan Hosseini-Asl; Chen Chen; Edresson Casanova; Subhankar Ghosh; Piotr Å»elasko; Zhehuai Chen; Jason Li; Jagadeesh Balam; Boris Ginsburg
Venue: Interspeech 2025 (main)

Abstract: Spoken dialogue is an intuitive form of human-computer interaction, yet current speech language models often remain constrained to turn-based exchanges, lacking real-time adaptability such as user barge-in. We propose a novel duplex speech to speech (S2S) architecture featuring continuous user inputs and codec agent outputs with channel fusion that directly models simultaneous user and agent streams. Using a pretrained streaming encoder for user input enables the first duplex S2S model without requiring speech pretrain. Separate architectures for agent and user modeling facilitate codec fine-tuning for better agent voices and halve the bitrate (0.6 kbps) compared to previous works. Experimental results show that the proposed model outperforms previous duplex models in reasoning, turn-taking, and barge-in abilities. The model requires significantly less speech data, as speech pretrain is skipped, which markedly simplifies the process of building a duplex S2S model from any LLMs. Finally, it is the first openly available duplex S2S model with training and inference code to foster reproducibility.
U-SAM: An Audio Language Model for Unified Speech, Audio, and Music Understanding
Authors: Ziqian Wang; Xianjun Xia; Xinfa Zhu; Lei Xie
Venue: Interspeech 2025 (main)

Abstract: The text generation paradigm for audio tasks has opened new possibilities for unified audio understanding. However, existing models face significant challenges in achieving a comprehensive understanding across diverse audio types, such as speech, general audio events, and music. Furthermore, their exclusive reliance on cross-entropy loss for alignment often falls short, as it treats all tokens equally and fails to account for redundant audio features, leading to weaker cross-modal alignment. To deal with the above challenges, this paper introduces U-SAM, an advanced audio language model that integrates specialized encoders for speech, audio, and music with a pretrained large language model (LLM). U-SAM employs a Mixture of Experts (MoE) projector for task-aware feature fusion, dynamically routing and integrating the domain-specific encoder outputs. Additionally, U-SAM incorporates a Semantic-Aware Contrastive Loss Module, which explicitly identifies redundant audio features under language supervision and rectifies their semantic and spectral representations to enhance cross-modal alignment. Extensive experiments demonstrate that USAM consistently outperforms both specialized models and existing audio language models across multiple benchmarks. Moreover, it exhibits emergent capabilities on unseen tasks, showcasing its generalization potential. Code is available.
Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Data
Authors: Yun Tang; Eesung Kim; Vijendra Raj Apsingekar
Venue: Interspeech 2025 (main)

Abstract: A joint speech and text optimization method is proposed for hybrid transducer and attention-based encoder decoder (TAED) modeling to leverage large amounts of text corpus and enhance ASR accuracy. The joint TAED (J-TAED) is trained with both speech and text input modalities together, while it only takes speech data as input during inference. The trained model can unify the internal representations from different modalities, and be further extended to text based domain adaptation.  It can effectively alleviate data scarcity for mismatch domain tasks since no speech data is required.  Our experiments show J-TAED successfully integrates speech and linguistic information into one model, and reduce the WER by 5.8 ~ 12.8% on the Librispeech dataset. The model is also evaluated on two out-of-domain datasets: one is finance and another is named entity focused. The text based domain adaptation brings 15.3%  and 17.8% WER reduction on those two datasets respectively.
The role of audio-visual integration in the time course of phonetic encoding in self-supervised speech models
Authors: Yi Wang; Oli Danyi Liu; Peter Bell
Venue: Interspeech 2025 (main)

Abstract: Human speech perception is multimodal. In natural speech, lip movements can precede corresponding voicing by a non-negligible gap of 100-300 ms, especially for specific consonants, affecting the time course of neural phonetic encoding in human listeners. However, it remains unexplored whether self-supervised learning models, which have been used to simulate audio-visual integration in humans, can capture this asynchronicity between audio and visual cues. We compared AV-HuBERT, an audio-visual model, with audio-only HuBERT, by using linear classifiers to track their phonetic decodability over time. We found that phoneme information becomes available in AV-HuBERT embeddings only about 20 ms before HuBERT, likely due to AV-HuBERT's lower temporal resolution and feature concatenation process. It suggests AV-HuBERT does not adequately capture the temporal dynamics of multimodal speech perception, limiting its suitability for modeling the multimodal speech perception process.