GenAI4ELab/papercli-papers huggingface dataset:
MMLoRA: Multitask Memory Parameter-Efficient Fine-Tuning for Multimodal SER
Authors: Yuanbo Fang; Xiaofen Xing; Xueru Li; Weibin Zhang; Xiangmin Xu
Venue: Interspeech 2025 (main)

Abstract: The differences in emotional expression present significant challenges for the development of effective emotion recognition systems. Although large language models (LLMs) have demonstrated strong performance, their generalization in emotion recognition tasks is often limited by variations in individual emotional expression, which remain inadequately addressed even by parameter-efficient fine-tuning techniques such as LoRA. To address these challenges, we proposes a multitask memory parameter-efficient fine-tuning method (MMLoRA) that enhances multimodal SER by incorporating gender as an auxiliary task. The method leverages shared LoRA experts to facilitate information exchange between tasks and employs a mixture of LoRA experts to process task-specific information. Additionally, a memory mechanism propagates task-specific information across layers. Experimental results demonstrate that the proposed MMLoRA significantly improves emotion recognition performance compared to vanilla LoRA.
Robust Unsupervised Adaptation of a Speech Recogniser Using Entropy Minimisation and Speaker Codes
Authors: Rogier C. van Dalen; Shucong Zhang; Titouan Parcollet; Sourav Bhattacharya
Venue: Interspeech 2025 (main)

Abstract: Speech recognisers usually perform optimally only in a specific environment and need to be adapted to work well in another. For adaptation to a new speaker, there is often too little data for fine-tuning to be robust, and that data is usually unlabelled. This paper proposes a combination of approaches to make adaptation to a single minute of data robust. First, instead of estimating the adaptation parameters with cross-entropy on a single error-prone hypothesis or "pseudo-label", this paper proposes a novel loss function, the conditional entropy over complete hypotheses. Using multiple hypotheses makes adaptation more robust to errors in the initial recognition. Second, a "speaker code" characterises a speaker in a vector short enough that it requires little data to estimate. On a far-field noise-augmented version of Common Voice, the proposed scheme yields a 20 % relative improvement in word error rate on one minute of adaptation data, increasing on 10 minutes to 29 %.
SCRIBAL: A Digital Transcription Tool in Higher Education
Authors: Javier RomÃ¡n; Pol Pastells; Mauro VÃ¡zquez; Clara PuigventÃ³s; Montserrat Nofre; Mariona TaulÃ©; Mireia FarrÃºs
Venue: Interspeech 2025 (main)

Abstract: SCRIBAL is a digital transcription and translation tool for university teaching, covering Catalan transcription and translation to the main foreign languages in class. Based on Whisper, SCRIBAL is fine-tuned for specific Catalan dialectal varieties and specialized academic terminology. It becomes an essential tool for accessibility, as well as for breaking language barriers and preserving the national languages in higher education.
From Static to Dynamic: Enhancing AAC with Generative Imagery and Zero-Shot TTS
Authors: Juliana Francis; Joakim Gustafsson; Ãva SzÃ©kely
Venue: Interspeech 2025 (main)

Abstract: This paper presents an Augmentative and Alternative Communication (AAC) approach for minimally verbal children with Autism Spectrum Disorder. Traditional AAC systems use fixed symbol sets and pre-defined Text-to-Speech (TTS) voices, this proposed method leverages text-to-image generation and zero-shot TTS to expand expressive capabilities. Users can create visual symbols for concepts and interests, enabling richer communication. Further, zero-shot TTS allows users to upload or record personalized voices, enabling users to have individualized output. By minimizing reliance on static symbols and voices, this approach aims to increase communicative agency, personal relevance, and social validity, areas often neglected in traditional interventions. Future research will explore long-term effects on communicative skills, user satisfaction, social engagement, and adaptability across various cultural and linguistic settings, aiming to develop more dynamic and personalized AAC solutions.
Concurrent Speech and Auditory Tag Clouds for Non-Visual Web Interaction
Authors: Dhia Eddine Merzougui; Nilesh Tete; Fabrice Maurel; GaÃ«l Dias; Mohammed Hasanuzzaman; AurÃ©lien Bournonville; Edgar Madelaine; Thomas Berthelin Le Tellier; FranÃ§ois Ledoyen; Laure Poutrain-Lejeune; FranÃ§ois Rioult; JÃ©rÃ©mie Pantin
Venue: Interspeech 2025 (main)

Abstract: Millions of visually impaired individuals face significant barriers to independent information access, limiting their autonomy and self-determination. Despite advancements in assistive technologies, challenges remain, particularly in handling web document structure and multi-channel information transposition. This paper addresses these issues by introducing the TagThunder experimental framework, which applies the "cocktail party effect" metaphor to transpose the morpho-dispositional semantics of web documents into an auditory tag cloud, enabling rapid non-visual skimming. Additionally, it explores interactive stimuli for structured information scanning through discrete and continuous guiding exploration strategies.
Towards Domain-Specific Spoken Language Understanding for a Catalan Voice-Controlled Video Game
Authors: Alex PeirÃ³-Lilja; Rodolfo Zevallos; Carme Armentano-Oller; Jose Giraldo; Cristina EspaÃ±a-Bonet; Mireia FarrÃºs
Venue: Interspeech 2025 (main)

Abstract: We design a voice-controlled video game to integrate Catalan into gaming using speech technologies developed under the Aina project. The game is designed to elicit natural speech commands from players. However, a significant challenge in this endeavor is the limited availability of Catalan-language Spoken Language Understanding (SLU) datasets, especially those covering specialized linguistic domains relevant to interactive gaming environments. To address this, we implement a cascading SLU system that combines automatic speech recognition (ASR) with roBERTa-based models previously trained in Catalan. The latter was finetuned as a multi-task classifier by generating synthetic transcriptions from a small set of human-written examples. With acceptable accuracy and time inference, our goal is to evaluate its performance in-game and gather feedback from users.
Accessible Delivery of Visual-Acoustic Biofeedback for Speech Sound Disorder
Authors: Tara McAllister; Peter Traver; Amanda Eads; William Haack; Helen Carey; Yi Shan; Wendy Liang; Tae Hong Park
Venue: Interspeech 2025 (main)

Abstract: This paper presents staRt, an iOS and web application for visual-acoustic biofeedback aiming to facilitate widespread uptake of technology-enhanced speech training. staRt provides a real-time linear predictive coding (LPC) spectrum that can be compared to an adjustable slider representing a target frequency for vocal tract resonance. The software currently presents exercises for children with speech sound disorder who have difficulty pronouncing "r" sounds in English, but it shows promise for adaptation to a number of other contexts.
End-to-End Indian Language Dubbing with Zero-Shot Speaker Preservation
Authors: Giri Raju; Sandeep Konam
Venue: Interspeech 2025 (main)

Abstract: This paper presents an end-to-end AI-driven dubbing platform designed specifically for Indian languages. The system leverages state-of-the-art speech models (ASR, MT, TTS) to streamline the dubbing process, minimizing manual effort while enabling precise output control. It generates high-quality, natural-sounding speech, preserving speaker characteristics through zero-shot synthesis irrespective of accent, age, or gender. Already successfully deployed with creators and educators, the platform enhances content accessibility and cross-lingual adaptation in education and entertainment for India's diverse linguistic communities.
Band-SCNet: A Causal, Lightweight Model for High-Performance Real-Time Music Source Separation
Authors: Junqi Yang; Yuhong Yang; Weiping Tu; Xin Zhao; Cedar Lin
Venue: Interspeech 2025 (main)

Abstract: Music source separation (MSS) for real-time applications faces challenges due to significant performance loss. In this paper, we propose Band-SCNet, a lightweight real-time model that bridges the performance gap between real-time and non-real-time models. Band-SCNet combines Sparse Compression with Cross-band and Narrow-band Blocks to reduce model size and complexity while improving performance. We also introduce the Compressed Self-Attention (CSA) Fusion Module, which enhances efficiency by reducing parameters. Experiments on the MUSDB18-HQ dataset show that Band-SCNet outperforms existing real-time models with an SDR of 7.79 dB, 2.59 million parameters by relaxing to a 92 ms latency, confirming its suitability for real-time MSS. The model achieves a balance between performance, latency, and model size, offering a promising solution for real-time music source separation.
CabinSep: IR-Augmented Mask-Based MVDR for Real-Time In-car Speech Separation with Distributed Heterogeneous Arrays
Authors: Runduo Han; Yanxin Hu; Yihui Fu; Zihan Zhang; Yukai Jv; Li Chen; Lei Xie
Venue: Interspeech 2025 (main)

Abstract: Separating overlapping speech from multiple speakers is crucial for effective human-vehicle interaction. This paper proposes CabinSep, a lightweight neural mask-based minimum variance distortionless response (MVDR) speech separation approach, to reduce speech recognition errors in back-end automatic speech recognition (ASR) models. Our contributions are threefold: First, we utilize channel information to extract spatial features, which improves the estimation of speech and noise masks. Second, we employ MVDR during inference, reducing speech distortion to make it more ASR-friendly. Third, we introduce a data augmentation method combining simulated and real-recorded impulse responses (IRs), improving speaker localization at zone boundaries and further reducing speech recognition errors. With a computational complexity of only 0.4 GMACs, CabinSep achieves a 17.5% relative reduction in speech recognition error rate in a real-recorded dataset compared to the state-of-the-art DualSep model.
DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization
Authors: Geonyoung Lee; Geonhee Han; Paul Hongsuck Seo
Venue: Interspeech 2025 (main)

Abstract: Language-queried Audio Source Separation (LASS) enables open-vocabulary sound separation via natural language queries. While existing methods rely on task-specific training, we explore whether pretrained diffusion models, originally designed for audio generation, can inherently perform separation without further training. In this study, we introduce a training-free framework leveraging generative priors for zero-shot LASS. Analyzing naÃ¯ve adaptations, we identify key limitations arising from modality-specific challenges. To address these issues, we propose Diffusion-Guided Mask Optimization (DGMO), a test-time optimization framework that refines spectrogram masks for precise, input-aligned separation. Our approach effectively repurposes pretrained diffusion models for source separation, achieving competitive performance without task-specific supervision. This work expands the application of diffusion models beyond generation, establishing a new paradigm for zero-shot audio separation.
Cross-Attention-Based Target Sound Extraction by Fully Leveraging Enrollment in a Shared Latent Space
Authors: Xue Yang; Guiru Shen; Yu Yang
Venue: Interspeech 2025 (main)

Abstract: Target sound extraction aims to isolate the target sound event (SE) from a mixture containing undesired interference. The enrollment of the target SE class is often utilized to derive an embedding for guidance. However, this embedding is highly summarized and discards important details, resulting in suboptimal performance. To address this problem, the local dynamics and temporal structures of the enrollment are leveraged in this paper. Specifically, the enrollment and mixture are first processed separately along the frequency and time axes, mapping them into a shared latent space. The processed features are then interacted via a cross-attention mechanism, with the resulting feature serving as guidance. This approach effectively leverages the contextual information in the enrollment and allows for the extraction of SE class that are unseen during training. Experimental results show that our proposed method attains high performance and exhibits certain zero-shot generalization ability.
DnR-nonverbal: Cinematic Audio Source Separation DatasetContaining Non-Verbal Sounds
Authors: Takuya Hasumi; Yusuke Fujita
Venue: Interspeech 2025 (main)

Abstract: We propose a new dataset for cinematic audio source separation (CASS) that handles non-verbal sounds. Existing CASS datasets only contain reading-style sounds as a speech stem. These datasets differ from actual movie audio, which is more likely to include acted-out voices. Consequently, models trained on conventional datasets tend to have issues where emotionally heightened voices, such as laughter and screams, are more easily separated as an effect, not speech. To address this problem, we build a new dataset, DnR-nonverbal. The proposed dataset includes non-verbal sounds like laughter and screams in the speech stem. From the experiments, we reveal the issue of non-verbal sound extraction by the current CASS model and show that our dataset can effectively address the issue in the synthetic and actual movie audio.
Neural Speech Extraction with Human Feedback
Authors: Malek Itani; Ashton Graves; Sefik Emre Eskimez; Shyamnath Gollakota
Venue: Interspeech 2025 (main)

Abstract: We present the first neural target speech extraction (TSE) system that uses human feedback for iterative refinement. Our approach allows users to mark specific segments of the TSE output, generating an edit mask. The refinement system then improves the marked sections while preserving unmarked regions. Since large-scale datasets of human-marked errors are difficult to collect, we generate synthetic datasets using various automated masking functions and train models on each. Evaluations show that models trained with noise power-based masking (in dBFS) and probabilistic thresholding perform best, aligning with human annotations. In a study with 22 participants, users showed a preference for refined outputs over baseline TSE. Our findings demonstrate that human-in-the-loop refinement is a promising approach for improving the performance of neural speech extraction.
Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate
Authors: Hanglei Zhang; Yiwei Guo; Zhihan Li; Xiang Hao; Xie Chen; Kai Yu
Venue: Interspeech 2025 (main)

Abstract: Most neural speech codecs achieve bitrate adjustment through intra-frame mechanisms, such as codebook dropout, at a Constant Frame Rate (CFR). However, speech segments inherently have time-varying information density (e.g., silent intervals versus voiced regions). This property makes CFR not optimal in terms of bitrate and token sequence length, hindering efficiency in real-time applications. In this work, we propose a Temporally Flexible Coding (TFC) technique, introducing variable frame rate (VFR) into neural speech codecs for the first time. TFC enables seamlessly tunable average frame rates and dynamically allocates frame rates based on temporal entropy. Experimental results show that a codec with TFC achieves optimal reconstruction quality with high flexibility, and maintains competitive performance even at lower frame rates. Our approach is promising for the integration with other efforts to develop low-frame-rate neural speech codecs for more efficient downstream tasks.
SPCODEC: Split and Prediction for Neural Speech Codec
Authors: Liang Wen; Lizhong Wang; Yuxing Zheng; Weijing Shi; Kwang Pyo Choi
Venue: Interspeech 2025 (main)

Abstract: Recent advancements in time-domain end-to-end neural speech codecs have significantly improved performance. However, existing codecs fail to fully exploit the correlations across different frequency bands in speech, leading to inefficiencies and reduced interpretability. In this paper, we introduce SPCODEC, a time-domain end-to-end neural speech codec featuring a latent split-and-prediction scheme. The model consists of a fully convolutional encoder-decoder and a group residual vector quantization module enhanced with a split-and-prediction mechanism. This mechanism disentangles low- and high-frequency representations and employs prediction to effectively reduce feature redundancy. SPCODEC achieves state-of-the-art MOS-POLQA scores of 4.0 at 6/8 kbps and 4.5 at 10.66/16 kbps for wideband and super-wideband speech, significantly outperforming both neural and traditional codecs.
Probing the Robustness Properties of Neural Speech Codecs
Authors: Wei-Cheng Tseng; David Harwath
Venue: Interspeech 2025 (main)

Abstract: Neural speech codecs have revolutionized speech coding, achieving higher compression while preserving audio fidelity. Beyond compression, they have emerged as tokenization strategies, enabling language modeling on speech and driving paradigm shifts across various speech processing tasks. Despite these advancements, their robustness in noisy environments remains underexplored, raising concerns about their generalization to real-world scenarios. In this work, we systematically evaluate neural speech codecs under various noise conditions, revealing non-trivial differences in their robustness. We further examine their linearity properties, uncovering non-linear distortions which partly explain observed variations in robustness. Lastly, we analyze their frequency response to identify factors affecting audio fidelity. Our findings provide critical insights into codec behavior and future codec design, as well as emphasizing the importance of noise robustness for their real-world integration1.
LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec
Authors: Yiwei Guo; Zhihan Li; Chenpeng Du; Hankun Wang; Xie Chen; Kai Yu
Venue: Interspeech 2025 (main)

Abstract: Although discrete speech tokens have exhibited strong potential for language model-based speech generation, their high bitrates and redundant timbre information restrict the development of such models. In this work, we propose LSCodec, a discrete speech codec that has both low bitrate and speaker decoupling ability. LSCodec adopts a multi-stage unsupervised training framework with a speaker perturbation technique. A continuous information bottleneck is first established, followed by vector quantization that produces a discrete speaker-decoupled space. A discrete token vocoder finally refines acoustic details from LSCodec. By reconstruction evaluations, LSCodec demonstrates superior intelligibility and audio quality with only a single codebook and smaller vocabulary size than baselines. Voice conversion and speaker probing experiments prove the excellent speaker disentanglement of LSCodec, and ablation study verifies the effectiveness of the proposed training framework.
Bringing Interpretability to Neural Audio Codecs
Authors: Samir Sadok; Julien Hauret; Ãric Bavu
Venue: Interspeech 2025 (main)

Abstract: The advent of neural audio codecs has increased in popularity due to their potential for efficiently modeling audio with transformers. Such advanced codecs represent audio from a highly continuous waveform to low-sampled discrete units. In contrast to semantic units, acoustic units may lack interpretability because their training objectives primarily focus on reconstruction performance. This paper proposes a two-step approach to explore the encoding of speech information within the codec tokens. The primary goal of the analysis stage is to gain deeper insight into how speech attributes such as content, identity, and pitch are encoded. The synthesis stage then trains an AnCoGen network for post-hoc explanation of codecs to extract speech attributes from the respective tokens directly.
NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference
Authors: Edresson Casanova; Paarth Neekhara; Ryan Langman; Shehzeen Hussain; Subhankar Ghosh; Xuesong Yang; Ante Jukic; Jason Li; Boris Ginsburg
Venue: Interspeech 2025 (main)

Abstract: Large Language Models (LLMs) have significantly advanced audio processing by leveraging audio codecs to discretize audio into tokens, enabling the application of language modeling techniques to speech data. However, existing audio codecs often operate at high frame rates, leading to slow training and inference, particularly for autoregressive models. To address this, there is growing interest in low frame-rate audio codecs, which reduce the number of autoregressive steps required to generate one second of audio. In this paper, we conduct ablation studies to examine the impact of frame rate, bitrate, and causality on codec reconstruction quality. Based on our findings, we introduce NanoCodec, a state-of-the-art audio codec that achieves high-quality compression at just 12.5 frames per second (FPS). NanoCodec outperforms related works across various bitrate ranges, establishing a new benchmark for low-latency and efficient Speech LLM training and inference.
Beat gestures made by human-like avatars affect speech perception
Authors: Matteo Maran; Renske RÃ¶tjes; Anna R. E. Schreurs; Hans Rutger Bosker
Venue: Interspeech 2025 (main)

Abstract: In face-to-face communication, several visual cues support speech perception. Even the timing of simple up-and-down flicks of the hand, called beat gestures, can convey word stress, changing what individuals hear (e.g., CONtent vs. conTENT). While beat gestures have been traditionally investigated in human-to-human communications, nowadays individuals increasingly interact with computer-controlled avatars (e.g., virtual assistants). The present study tested whether beat gestures produced by an avatar affect word stress perception, similarly to human gestures. Furthermore, this study tested whether a minimal visual cue such as a 2D moving disc can also affect speech perception. Beat gestures made by the avatar significantly affected speech perception, albeit slightly less than human-made gestures. The disc condition did not affect speech perception. The present work lays the foundation for the application of (beat) gesturing avatars, which could be used to boost speech intelligibility.
The mutual exclusivity bias of bilingual visually grounded speech models
Authors: Dan Oneata; Leanne Nortje; Yevgen Matusevych; Herman Kamper
Venue: Interspeech 2025 (main)

Abstract: Mutual exclusivity (ME) is a strategy where a novel word is associated with a novel object rather than a familiar one, facilitating language learning in children. Recent work has found an ME bias in a visually grounded speech (VGS) model trained on English speech with paired images. But ME has also been studied in bilingual children, who may employ it less due to cross-lingual ambiguity. We explore this pattern computationally using bilingual VGS models trained on combinations of English, French, and Dutch. We find that bilingual models generally exhibit a weaker ME bias than monolingual models, though exceptions exist. Analyses show that the combined visual embeddings of bilingual models have a smaller variance for familiar data, partly explaining the increase in confusion between novel and familiar concepts. We also provide new insights into why the ME bias exists in VGS models in the first place.