GenAI4ELab/papercli-papers huggingface dataset:
SonarGuard2: Ultrasonic Face Liveness Detection Based on Adaptive Doppler Effect Feature Extraction
Authors: Xiaoming Zhang; Ke-Yue Zhang; Taiping Yao; Songjun Cao; Shouhong Ding; Long Ma
Venue: Interspeech 2025 (main)

Abstract: Face anti-spoofing is crucial to security of face recognition systems. Vision-based face liveness detection algorithms often fail when faced with video attacks, such as video replay. However, liveness detection based on sound waves can effectively detect such attacks by relying on the Doppler effect. In order to improve the robustness of liveness detection, we propose a novel framework, named SonarGuard2, adaptively selecting the ultrasonic signals and analyse the Doppler effect. Specifically, we introduce acoustic echo cancellation to filter to get the Doppler effect features, then utilize complex convolutional neural networks to enhance the abilitity to model the Doppler effect features. Furthemore, we propose a novel selection strategy to determine the usability of ultrasonic signals on mobile devices. The performance and visualizaiton on the collected data demonstrate the effectiveness of our framework.
Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioning
Authors: Hien Ohnaka; Yuma Shirahata; Byeongseon Park; Ryuichi Yamamoto
Venue: Interspeech 2025 (main)

Abstract: We propose a model to obtain phonemic and prosodic labels of speech that are coherent with graphemes. Unlike previous methods that simply fine-tune a pre-trained ASR model with the labels, the proposed model conditions the label generation on corresponding graphemes by two methods: 1) Add implicit grapheme conditioning through prompt encoder using pre-trained BERT features. 2) Explicitly prune the label hypotheses inconsistent with the grapheme during inference. These methods enable obtaining parallel data of speech, the labels, and graphemes, which is applicable to various downstream tasks such as text-to-speech and accent estimation from text. Experiments showed that the proposed method significantly improved the consistency between graphemes and the predicted labels. Further, experiments on accent estimation task confirmed that the created parallel data by the proposed method effectively improve the estimation accuracy.
Non-Standard Accent TTS Support via Large Multi-Accent Frontend Pronunciation Knowledge Transfer
Authors: Noe Berger; Siqi Sun; Korin Richmond
Venue: Interspeech 2025 (main)

Abstract: Mainstream text-to-speech (TTS) applications rarely offer non-standard accented voice options, perhaps in part due to practical challenges associated with the acquisition of prerequisite training data. Building on prior research, this work demonstrates that a large multi-accent neural frontend model can reduce pronunciation training data requirements by 95% for robust performance in low-resource accents. We further show that pronunciation knowledge transfer is weakly influenced by accent similarity, quantified using a Levenshtein distance-based metric. The large multi-accent paradigm thus emerges as an effective strategy for improving non-standard accent voice-building accessibility, provided source accents are selected to maximize similarity with target accents where possible.
Speech-guided Grapheme-to-Phoneme Conversion for Cantonese Text-to-Speech
Authors: Timothy Shin Heng Mak; King Yiu Suen; Albert Y.S. Lam
Venue: Interspeech 2025 (main)

Abstract: Grapheme-to-Phoneme (G2P) conversion is a crucial component in neural Text-to-Speech. Cantonese G2P is especially difficult and suffers from two pain points. First, a large number of commonly used characters have multiple correct pronunciations (jyutpings) that cannot be distinguished based on textual context. Secondly, there is a lack of accurate jyutping-labelled data that can be used for the training of character-to-jyutping (C2J) models. In this study, we propose a speech-guided C2J method based on augmenting an off-the-shelf Automatic Speech Recognition model with a speech-guided C2J module. To overcome the data scarcity problem, we trained the model on speech generated from a commercial Text-to-Speech model. We show that this simple approach achieved a jyutping error rate (JER) of 2% in unseen, clean, speech, improving the best text-based C2J method by 2.6% absolutely. For common polyphonic characters, the improvement was even greater, reducing the JER from 10.6% to 2.0%.
Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotation
Authors: Rui Hu; Xiaolong Lin; Jiawang Liu; Shixi Huang; Zhenpeng Zhan
Venue: Interspeech 2025 (main)

Abstract: In this paper, we propose a method for annotating phonemic and prosodic labels on a given audio-transcript pair, aimed at constructing Japanese text-to-speech (TTS) datasets. Our approach involves fine-tuning a large-scale pre-trained automatic speech recognition (ASR) model, conditioned on ground truth transcripts, to simultaneously output phrase-level graphemes and annotation labels. To further correct errors in phonemic labeling, we employ a decoding strategy that utilizes dictionary prior knowledge. The objective evaluation results demonstrate that our proposed method outperforms previous approaches relying solely on text or audio. The subjective evaluation results indicate that the naturalness of speech synthesized by the TTS model, trained with labels annotated using our method, is comparable to that of a model trained with manual annotations.
Enabling the replicability of speech synthesis perceptual evaluations
Authors: SÃ©bastien Le Maguer; GwÃ©nolÃ© LecorvÃ©; Damien Lolive; Naomi Harte; Juraj Å imko
Venue: Interspeech 2025 (main)

Abstract: How speech synthesis is evaluated is nowadays questioned. Not only have conventional listening tests as a whole been proven a poor match for modern synthesis, but more fundamentally, important information (e.g., the question asked to the listener) is frequently missing in the report of the outcome of the evaluation despite the impact on the interpretation of the test results. This can lead to uncertainty about the validity of these evaluations. To address this issue, we propose standardising the structure of any evaluation report. To facilitate this standardisation, our contribution is twofold: an open-source subjective evaluation platform; and a set of reporting guidelines. The platform is designed to enable the development of easily shareable evaluation recipes. The set of guidelines complements the platform to support researchers in reporting their evaluation choices and analysis in more detail while relying on the recipe to describe the actual evaluation process.
When The MOS Predictor Asks For Training Annotation In Cross Lingual/Domain Adaptation
Authors: Natacha Miniconi; Meysam Shamsi; Anthony Larcher
Venue: Interspeech 2025 (main)

Abstract: The Mean Opinion Score (MOS) is widely used to assess speech synthesis quality, but requires costly human evaluation. Automatic MOS predictors have been developed to estimate MOS. Training and generalizing these predictors across different languages and domains remain difficult due to the high cost of labeled data. To optimize MOS prediction while minimizing human annotation efforts, we explore active learning. As far as we know, this is the first study to investigate the use of active learning as training sample selection strategies for enhancing MOS prediction. We investigate its effectiveness on two tasks: cross-domain and cross-lingual adaptation, comparing multiple selection strategies that rely on uncertainty or diversity measures. Our sample selection strategies have been compared with random selection. Among these strategies, Monte Carlo (MC) Dropout proved effective for cross-lingual adaptation, while perturbation noise performed well for cross-domain adaptation.
Assessment of the synthetic quality and controllability of laughing onset in speech-laugh synthesis
Authors: Ryo Setoguchi; Yoshiko Arimoto
Venue: Interspeech 2025 (main)

Abstract: This study is the first challenge of building a synthetic speech-laugh model via a deep learning technique. To maintain the phonetic intelligibility of synthesized speech-laugh, the model was trained with nonlaughing read speech material for both phones of speech-laugh (SL) and of speech (SP). To control laughing onset in SL, the model was also trained using SL material only for the phones of SL instances. The listening tests revealed that the naturalness score for synthesized female SL was as high as that for human SL and that the laughter-likeness score for synthesized SL was higher than that for synthesized SP in almost all conditions. The dictation test revealed that the training for phonetic intelligibility in SL synthesis was highly effective for synthesized SL. However, the difference between segmented SL onset and correct onset was greater for synthesized SL with phonetic intelligibility training than for that without training.
Running Conventional Automatic Speech Recognition on Memristor Hardware: A Simulated Approach
Authors: Nick Rossenbach; Benedikt Hilmes; Leon Brackmann; Moritz Gunz; Ralf SchlÃ¼ter
Venue: Interspeech 2025 (main)

Abstract: Memristor-based hardware offers new possibilities for energy-efficient machine learning (ML) by providing analog in-memory matrix multiplication. Current hardware prototypes cannot fit large neural networks, and related literature covers only small ML models for tasks like MNIST or single word recognition. Simulation can be used to explore how hardware properties affect larger models, but existing software assumes simplified hardware. We propose a PyTorch-based library based on "Synaptogen" to simulate neural network execution with accurately captured memristor hardware properties. For the first time, we show how an ML system with millions of parameters would behave on memristor hardware, using a Conformer trained on the speech recognition task TED-LIUMv2 as example. With adjusted quantization-aware training, we limit the relative degradation in word error rate to 25% when using a 3-bit weight precision to execute linear operations via simulated analog computation.
Word Level Timestamp Generation for Automatic Speech Recognition and Translation
Authors: Ke Hu; Krishna Puvvada; Elena Rastorgueva; Zhehuai Chen; He Huang; Shuoyang Ding; Kunal Dhawan; Hainan Xu; Jagadeesh Balam; Boris Ginsburg
Venue: Interspeech 2025 (main)

Abstract: We introduce a data-driven approach for enabling word-level timestamp prediction in the Canary model. Accurate timestamp information is crucial for a variety of downstream tasks such as speech content retrieval and timed subtitles. While traditional hybrid systems and end-to-end (E2E) models may employ external modules for timestamp prediction, our approach eliminates the need for separate alignment mechanisms. By leveraging the NeMo Forced Aligner (NFA) as a teacher model, we generate word-level timestamps and train the Canary model to predict timestamps directly. We introduce a new 
 token, enabling the Canary model to predict start and end timestamps for each word. Our method demonstrates precision and recall rates between 80% and 90%, with timestamp prediction errors ranging from 20 to 120 ms across four languages, with minimal WER degradation. Additionally, we extend our system to automatic speech translation (AST) tasks, achieving timestamp prediction errors around 200 milliseconds. Our code is open-sourced through NeMo, and the checkpoint is released at Hugging Face.
Directional Speech Recognition with Full-Duplex Capability
Authors: Ju Lin; Yiteng Huang; Ming Sun; Frank Seide; Florian Metze
Venue: Interspeech 2025 (main)

Abstract: Recent work on directional automatic speech recognition (DASR) has enabled automatic transcription of a conversation partner several feet away via smart glasses. DASR leverages multiple microphones in the glasses by using multiple beamformers simultaneously. We aim to make the DASR insensitive to scenarios that also involve text-to-speech (TTS) playback. This could enable additional future scenarios like simultaneous speech translation. How to prevent ASR from capturing the system's own TTS output, while maintaining optimal clarity of the captured conversation partner's speech? We experiment with two modern linear acoustic echo cancellation (AEC) algorithms. To remedy accuracy regressions from echo residuals, we propose AEC-aware model training. While AEC alone eliminates most TTS loopback, dramatically improving the word-error rate (WER) by over 70%, AEC-aware model training provides further relative WER boosts of 13% or more.
CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Models
Authors: Jiajun He; Naoki Sawada; Koichi Miyazaki; Tomoki Toda
Venue: Interspeech 2025 (main)

Abstract: In real-world applications, automatic speech recognition (ASR) systems must handle overlapping speech from multiple speakers and recognize rare words like technical terms. Traditional methods address multi-talker ASR and contextual biasing separately, limiting performance in complex scenarios. We propose a unified framework that combines multi-talker overlapping speech recognition and contextual biasing into a single task. Our ASR method integrates pretrained speech encoders and large language models (LLMs), using optimized finetuning strategies. We also introduce a two-stage filtering algorithm to efficiently identify relevant rare words from large biasing lists and incorporate them into the LLMâs prompt input, enhancing rare word recognition. Experiments show that our approach outperforms traditional contextual biasing methods, achieving a WER of 7.9% on LibriMix and 32.9% on AMI SDM when the biasing size is 1,000, demonstrating its effectiveness in complex speech scenarios.
Selective Invocation for Multilingual ASR: A Cost-effective Approach Adapting to Speech Recognition Difficulty
Authors: Hongfei Xue; Yufeng Tang; Jun Zhang; Xuelong Geng; Lei Xie
Venue: Interspeech 2025 (main)

Abstract: Although multilingual automatic speech recognition (ASR) systems have significantly advanced, enabling a single model to handle multiple languages, inherent linguistic differences and data imbalances challenge SOTA performance across all languages. While language identification (LID) models can route speech to the appropriate ASR model, they incur high costs from invoking SOTA commercial models and suffer from inaccuracies due to misclassification. To overcome these, we propose SIMA, a selective invocation for multilingual ASR that adapts to the difficulty level of the input speech. Built on a spoken large language model (SLLM), SIMA evaluates whether the input is simple enough for direct transcription or requires the invocation of a SOTA ASR model. Our approach reduces word error rates by 18.7% compared to the SLLM and halves invocation costs compared to LID-based methods. Tests on three datasets show that SIMA is a scalable, cost-effective solution for multilingual ASR applications.
Improving Audio Classification by Transitioning from Zero- to Few-Shot
Authors: James Taylor; Wolfgang Mack
Venue: Interspeech 2025 (main)

Abstract: State-of-the-art audio classification often employs a zero-shot approach, which involves comparing audio embeddings with embeddings from text describing the respective audio class. These embeddings are usually generated by neural networks trained through contrastive learning to align audio and text representations. Identifying the optimal text description for an audio class is challenging, particularly when the class comprises a wide variety of sounds. This paper examines few-shot methods designed to improve classification accuracy beyond the zero-shot approach. Specifically, audio embeddings are grouped by class and processed to replace the inherently noisy text embeddings. Our results demonstrate that few-shot classification typically outperforms the zero-shot baseline.
Zero-Shot Learning for Acoustic Event Classification Using an Attribute Vector and Conditional GAN
Authors: Kohei Uehara; Ryoichi Takashima; Tetsuya Takiguchi
Venue: Interspeech 2025 (main)

Abstract: This paper presents a zero-shot acoustic event classification (ZS-AEC) method to classify acoustic events for which there is no training data. A previous study proposed a method to classify unseen events by estimating attribute information instead of acoustic event labels, where each acoustic event is associated with attribute information such as sound source material and the pitch (high or low) of the sound. However, this method often leads to the misclassification of unseen acoustic events as seen events. In this paper, we propose a generative-based ZS-AEC method to reduce the bias of prediction toward seen acoustic events. The proposed method generates latent features of unseen acoustic events from their attribute information using a conditional GAN, and a classifier is trained using the generated latent features. Experimental results demonstrated that the proposed method achieved higher classification accuracies than the conventional method based on attribute information.
Leveraging Multi-Level Features of ATST with Conformer-Based Dual-Branch Network for Sound Event Detection
Authors: Lipeng Dai; Qing Wang; Jie Zhang; Shengyu Peng; Yu Guan; Wu Guo
Venue: Interspeech 2025 (main)

Abstract: In this paper, we propose a Conformer-based dual-branch framework to fully exploit the multi-layer features of pre-trained model (PTM) for sound event detection (SED). The proposed model follows the mainstream framework, consisting of a front-end encoder built upon the pre-trained Audio Teacher-Student Transformer (ATST) model and a back-end context network. For the front-end, we evenly divide the Transformer layers of ATST into shallow and deep parts, each of which is fused using weighted integration to effectively incorporate multi-layer features for SED. In the back-end, dual-branch Conformer is used to extract both high-level and low-level clues from the aggregated features. Furthermore, we adopt adapter modules instead of full fine-tuning the ATST model in the training stage, achieving comparable performance with far fewer parameters. We carry out experiments on the validation set for Task 4 of DCASE 2024 Challenge, and results demonstrate the efficacy of the proposed method.
Leveraging Unlabeled Audio for Audio-Text Contrastive Learning via Audio-Composed Text Features
Authors: Tatsuya Komatsu; Hokuto Munakata; Yuchi Ishikawa
Venue: Interspeech 2025 (main)

Abstract: We propose a novel approach to audio-text contrastive learning that leverages unlabeled audio by introducing audio-composed text features. First, we generate composed audio by additively combining labeled and unlabeled audio. To obtain a text feature aligned with this newly composed audio, we introduce an audio-to-text (a2t) module that transforms the features of unlabeled audio into the corresponding text feature. The newly generated text feature is then concatenated with the original text of the labeled audio and passed through a text encoder to produce the audio-composed text features. By pairing these features with the composed audio for contrastive learning, our approach effectively integrates information from both labeled and unlabeled data. In audio-text retrieval experiments on Clotho and AudioCaps, the proposed method achieves notable improvements in Recall@1, with relative gains of 9.3% and 13.6%, respectively, compared to those trained solely with labeled audio.
Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos
Authors: Yuchi Ishikawa; Shota Nakada; Hokuto Munakata; Kazuhiro Saito; Tatsuya Komatsu; Yoshimitsu Aoki
Venue: Interspeech 2025 (main)

Abstract: In this paper, we propose Language-Guided Contrastive Audio-Visual Masked Autoencoders (LG-CAV-MAE) to improve audio-visual representation learning. LG-CAV-MAE integrates a pretrained text encoder into contrastive audio-visual masked autoencoders, enabling the model to learn across audio, visual and text modalities. To train LG-CAV-MAE, we introduce an automatic method to generate audio-visual-text triplets from unlabeled videos. We first generate frame-level captions using an image captioning model and then apply CLAP-based filtering to ensure strong alignment between audio and captions. This approach yields high-quality audio-visual-text triplets without requiring manual annotations. We evaluate LG-CAV-MAE on audio-visual retrieval tasks, as well as an audio-visual classification task. Our method significantly outperforms existing approaches, achieving up to a 5.6% improvement in recall@10 for retrieval tasks and a 3.2% improvement for the classification task.
AC/DC: LLM-based Audio Comprehension via Dialogue Continuation
Authors: Yusuke Fujita; Tomoya Mizumoto; Atsushi Kojima; Lianbo Liu; Yui Sudo
Venue: Interspeech 2025 (main)

Abstract: We propose an instruction-following audio comprehension model that leverages the dialogue continuation ability of large language models (LLMs). Instead of directly generating target captions in training data, the proposed method trains a model to produce responses as if the input caption triggered a dialogue. This dialogue continuation training mitigates the caption variation problem. Learning to continue a dialogue effectively captures the caption's meaning beyond its surface-level words. As a result, our model enables zero-shot instruction-following capability without multitask instruction tuning, even trained solely on audio captioning datasets. Experiments on AudioCaps, WavCaps, and Clotho datasets with AudioBench audio-scene question-answering tests demonstrate our model's ability to follow various unseen instructions.
Anomalous Sound Detection Based Feature Fusion and Dual-path Non-linear Independent Components Estimation
Authors: Yawei Wang; Qiaoling Zhang; Yi Zhang; Junyao Hu
Venue: Interspeech 2025 (main)

Abstract: Anomalous sound detection (ASD) that relies on annotated information during training has achieved excellent performances in benchmarks such as DCASE2020 Task 2. However, annotated data may not always be available in many scenarios. The Non-linear Independent Components Estimation (NICE) has been a promising technique for anomaly detection, as it does not rely on annotated information and is capable of exact likelihood estimation. In this paper, based on the classic NICE, a dual-path NICE model (dpNICE) is proposed, which allows for two input features feeding to two pathways and provides interactive channels for interaction and fusion between the two pathways. Then, a multiple feature fusion and dpNICE-based approach is developed, which leverages information of multiple audio features and learns to detect anomalies without annotated data. Experimental results show that the AUC and pAUC performances of our method are better than other competing annotation-free methods.
An Effective Anomalous Sound Detection Method Based on Global and Local Attribute Mining
Authors: Nan Jiang; Yan Song; Qing Gu; Haoyu Song; Lirong Dai; Ian McLoughlin
Venue: Interspeech 2025 (main)

Abstract: Recent anomalous sound detection (ASD) methods have achieved impressive performance by leveraging attribute information. However, domain shift and imbalance issues caused by inconsistent attributes between source and target domains remain a key challenge. In this paper, we propose a novel global and local attribute mining method for ASD (GLAM-ASD) with a Transformer based encoder consisting of teacher and student branches to exploit semantic-rich information. The student mines contextual information via masked patch reconstruction while the teacher obtains weights from the student via exponential moving average. Global and local attributes are then derived in an unsupervised way. The student branch is further optimized by global self-distillation and pseudo-label prediction via learned local attributes. Evaluations on the DCASE2023 Task2 benchmark demonstrate the superiority of GLAM-ASD, especially in its target domain generalization performance.