GenAI4ELab/papercli-papers huggingface dataset:
PPGs-BERT: Leveraging Phoneme Sequence and BERT for Alzheimerâs Disease Detection from Spontaneous Speech
Authors: Qi Sun; Ziyue Qiu; Yu Pu; Jinpeng Li; Xuchu Chen; Wei-Qiang Zhang
Venue: Interspeech 2025 (main)

Abstract: Alzheimer's Disease (AD) is a neurodegenerative condition characterized by linguistic impairments. While ASR and LLMs show promise in AD detection, ASR often normalizes key AD-related speech patterns and faces cross-lingual challenges due to language dependencies. Besides, ASR training demands extensive matched data. In our paper, however, we employ a phoneme recognizer as a frontend tokenizer. Provided it has comprehensive phoneme coverage, a multitude of linguistic phenomena can be represented via phoneme sequences, including hesitations, repetitions, pauses, mispronunciations, and even distinctions between different language identities that are crucial for AD detection. Furthermore, the BERT model is employed to extract high-dimensional features from the Phonetic PosteriorGrams (PPGs), which are ultimately used to diagnose Alzheimer's disease. Our approach offers cross-lingual applicability, achieves competitive accuracy, and maintains computational efficiency.
LLM-based phoneme-to-grapheme for phoneme-based speech recognition
Authors: Te Ma; Min Bi; Saierdaer Yusuyin; Hao Huang; Zhijian Ou
Venue: Interspeech 2025 (main)

Abstract: In automatic speech recognition (ASR), phoneme-based multilingual pre-training and crosslingual fine-tuning is attractive for its high data efficiency and competitive results compared to subword-based models. However, Weighted Finite State Transducer (WFST) based decoding is limited by its complex pipeline and inability to leverage large language models (LLMs). Therefore, we propose LLM-based phoneme-to-grapheme (LLM-P2G) decoding for phoneme-based ASR, consisting of speech-to-phoneme (S2P) and phoneme-to-grapheme (P2G). A challenge is that there seems to have information loss in cascading S2P and P2G. To address this challenge, we propose two training strategies: data augmentation with noisy phonemes (DANP), and randomized top-K marginalized (TKM) training and decoding. Our experimental results show that LLM-P2G outperforms WFST-based systems in crosslingual ASR for Polish and German, by relative WER reductions of 3.6% and 6.9% respectively.
Pinyin-Guided Chinese Speech Recognition with Large Language Model
Authors: Jie Zhengjie; Gaofeng Cheng
Venue: Interspeech 2025 (main)

Abstract: While LLM-based automatic speech recognition (LLM-ASR) has demonstrated efficacy through direct acoustic-to-text mapping, its implicit alignment often fails to capture phonetic relationships in Chinese, leading to pronunciation confusion and homophone errors. This paper proposes Pinyin-Guided ASR (PYG-ASR), which innovatively modifies the LLM-ASR to simultaneously map acoustic features to both Pinyin and text tokens, enhancing linguistic representation. PYG-ASR leverages the generated Pinyin alongside text for error correction, prompting text LLM to refine transcriptions without finetuning. Furthermore, error correction phase inherently enables context biasing by filtering bias phrases through Pinyin matching and incorporating them into the prompt. Experiments show that PYG-ASR reduces CER by 25% on the AISHELL-1 test set. Additionally, our approach shows a 49.2% CER reduction relatively for bias phrases on the AISHELL-1 test set after contextual bias.
Text-Enhanced Audio Encoder for Large Language Model based Speech Recognition via Cross-Modality Pre-training with Unpaired Audio-Text Data
Authors: Hang Su; Yuxiang Kong; Lichun Fan; Jian Luan
Venue: Interspeech 2025 (main)

Abstract: Recent advances in Large Language Models (LLM) have brought a new architecture for Automatic Speech Recognition (ASR) tasks, where an audio encoder is followed by a powerful LLM. Refining audio embeddings from the audio encoder to better align textual embeddings can enhance performance of LLM-based ASR. However, current LLM-based ASR research mainly focuses on aligning textual and audio features via paired audio-text data. The use of unpaired audio-text data for such alignment remains under-explored. This paper proposes a cross-modality pre-training method using readily available unpaired audio-text data to better align the audio embeddings to text modality. Experimental results show that using this text-enhanced audio encoder in LLM-based ASR significantly outperforms using the audio encoder pre-trained only with audio data. This method has great potential for further improvement with plentiful easily accessible unpaired audio-text data.
Towards atypical speech transcription using LLM-based ASR
Authors: Jinda Zhang; Aanchan Mohan
Venue: Interspeech 2025 (main)

Abstract: Training a linear transformation between speech encoders and LLMs enable LLMs to transcribe speech. SLAM-ASR is one such recently proposed architecture. This paper examines its adaptability across three domains of varying difficulty: read speech (Librispeech, easiest), meeting speech (AMI, medium) and post-stroke aphasic speech (AphasiaBank, most difficult) for both word- and phoneme-level transcription. After studying cross-domain adaptability, our work explores the use of transfer learning to seed model fine-tuning for the target domain from a source domain. Results show that transferring from an easier to a harder domain offers little benefit, while the reverse seems to improve model robustness in the easier target domain. Our work also looks at the impact of a phoneme encoder at the input and multiple single-task instruction fine tuning on phoneme and word transcription tasks. This work advances the adaptation of LLM-based ASR for atypical speech transcription.
Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLM
Authors: Jeena Prakash; Blessingh Kumar; Kadri Hacioglu; Bidisha Sharma; Sindhuja Gopalan; Malolan Chetlur; Shankar Venkatesan; Andreas Stolcke
Venue: Interspeech 2025 (main)

Abstract: Automatic speech recognition (ASR) models rely on high-quality transcribed data for effective training. Generating pseudo-labels for large unlabeled audio datasets often relies on complex pipelines that combine multiple ASR outputs through multi-stage processing, leading to error propagation, information loss and disjoint optimization. We propose a unified multi-ASR prompt-driven framework using postprocessing by either textual or speech-based large language models (LLMs), replacing voting or other arbitration logic for reconciling the ensemble outputs. We perform a comparative study of multiple architectures with and without LLMs, showing significant improvements in transcription accuracy compared to traditional methods. Furthermore, we use the pseudo-labels generated by the various approaches to train semi-supervised ASR models for different datasets, again showing improved performance with textual and speechLLM transcriptions compared to baselines.
Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis
Authors: Tianyi Xu; Hongjie Chen; Qing Wang; Lv Hang; Jian Kang; Jie Li; Zhennan Lin; Yongxiang Li; Lei Xie
Venue: Interspeech 2025 (main)

Abstract: Large-scale training corpora have significantly improved the performance of ASR models. Unfortunately, due to the relative scarcity of data, Chinese accents and dialects remain a challenge for most ASR models. Recent advancements in self-supervised learning have shown that self-supervised pretraining, combined with large language models (LLM), can effectively enhance ASR performance in low-resource scenarios. We aim to investigate the effectiveness of this paradigm for Chinese dialects. Specifically, we pre-train a Data2vec2 model on 300,000 hours of unlabeled dialect and accented speech data and do alignment training on a supervised dataset of 40,000 hours. Then, we systematically examine the impact of various projectors and LLMs on Mandarin, dialect, and accented speech recognition performance under this paradigm. Our method achieved SOTA results on multiple dialect datasets, including Kespeech. We will open-source our work to promote reproducible research.
Synonymity-Based Semantic Coding for Efficient Speech Compression
Authors: Shanhui Gan; Zijian Liang; Kai Niu; Ping Zhang
Venue: Interspeech 2025 (main)

Abstract: Recent neural speech coding methods optimize the coding rates for perceptual performance in an end-to-end manner. In this paper, we build the relationship between perceptual-oriented compression and the concept of "semantic" compression. We propose a synonymity-based semantic speech coding framework, in which synonymous representations corresponding to the extracted latent features serve as the input of the semantic compression. This framework is designed to approach the compression limits established by recent semantic information theory while preserving perceptual qualities. We provide an implementation of our proposed framework using a K-means algorithm to determine the synonymous representations and a nonlinear transform coding model as the semantic compression method to approach the compression limits. Experimental results show that our method outperforms both traditional and neural speech coding schemes, achieving superior compression efficiency and better perceptual quality.
Towards an Ultra-Low-Delay Neural Audio Coding with Computational Efficiency
Authors: Byeong Hyeon Kim; Hyungseob Lim; Inseon Jang; Hong-Goo Kang
Venue: Interspeech 2025 (main)

Abstract: Recent studies on neural audio codecs (NACs) have primarily focused on improving audio quality in extremely low bit-rate scenarios. However, they have not thoroughly explored the impact of latency. In this work, we first demonstrate that NACs can achieve high-quality reconstruction with an algorithmic delay below 1 ms, albeit at substantial computational costs. To address this challenge, we propose DualStream, a novel framework designed to significantly reduce computational costs in ultra-low-delay settings. DualStream integrates a lightweight encoding module with a small down-sampling ratio to maintain low algorithmic delay, combined with a larger module with a higher down-sampling ratio that processes time-delayed inputs to improve efficiency without introducing additional delay. Experimental results demonstrate that DualStream, with an algorithmic delay of 0.7 ms, achieves comparable performance to conventional NACs while reducing computational costs by approximately 40%.
SpecTokenizer: A Lightweight Streaming Codec in the Compressed Spectrum Domain
Authors: Zixiang Wan; Guochang Zhang; Yifeng He; Jianqiang Wei
Venue: Interspeech 2025 (main)

Abstract: Neural Audio Codecs (NACs) have gained growing attention in recent years as technologies for audio compression and audio representation in speech language models. While mainstream NACs typically require G-level computation and M-level parameters, the performance of lightweight and streaming NACs remains underexplored. This paper proposes SpecTokenizer, a lightweight streaming codec that operates in the compressed spectral domain. Composed solely of alternating CNN and RNN layers, SpecTokenizer achieves greater efficiency and better representational capability through multi-scale modeling in the compressed spectrum domain. At 4 kbps, the proposed SpecTokenizer achieves comparable or superior performance compared to the codec with state-of-the-art lightweight architecture while requiring only 20% of the computation and 10% of the parameters. Furthermore, it significantly outperforms the codec when using similar computational and storage resources.
TS3-Codec: Transformer-Based Simple Streaming Single Codec
Authors: Haibin Wu; Naoyuki Kanda; Sefik Emre Eskimez; Jinyu Li
Venue: Interspeech 2025 (main)

Abstract: Neural audio codecs (NACs) have garnered significant attention as key technologies for audio compression as well as audio representation for speech language models. While mainstream NACs are predominantly convolution-based, the performance of NACs with a purely transformer-based, and convolution-free architecture remains unexplored. This paper introduces TS3-Codec, a Transformer-Based Simple Streaming Single Codec. TS3-Codec consists of only a stack of transformer layers and linear layers, offering greater simplicity and expressiveness by fully eliminating convolution layers that require careful hyperparameter tuning and large computations. Under the streaming setup, TS3-Codec achieves comparable or superior performance compared to the codec with state-of-the-art convolution-based architecture while requiring only 12% of the computation and 77% of bitrate. Furthermore, it significantly outperforms the convolution-based codec when using similar computational resources.
Towards Bitrate-Efficient and Noise-Robust Speech Coding with Variable Bitrate RVQ
Authors: Yunkee Chae; Kyogu Lee
Venue: Interspeech 2025 (main)

Abstract: Residual Vector Quantization (RVQ) has become a dominant approach in neural speech and audio coding, providing high-fidelity compression. However, speech coding presents additional challenges due to real-world noise, which degrades compression efficiency. Standard codecs allocate bits uniformly, wasting bitrate on noise components that do not contribute to intelligibility. This paper introduces a Variable Bitrate RVQ (VRVQ) framework for noise-robust speech coding, dynamically adjusting bitrate per frame to optimize rate-distortion trade-offs. Unlike constant bitrate (CBR) RVQ, our method prioritizes critical speech components while suppressing residual noise. Additionally, we integrate a feature denoiser to further improve noise robustness. Experimental results show that VRVQ improves rate-distortion trade-offs over conventional methods, achieving better compression efficiency and perceptual quality in noisy conditions. Samples are available at our project page.
LSPnet: an ultra-low bitrate hybrid neural codec
Authors: Bowen Zhang; Ian McLoughlin; Xiaoxiao Miao; AS Madhukumar
Venue: Interspeech 2025 (main)

Abstract: This paper presents an ultra-low bitrate speech codec that achieves high-fidelity speech coding at 1.2kbps while maintaining low computational complexity. Building upon the LPCNet framework, combined with a parametric encoder, we introduce several key improvements by incorporating line spectral pairs (LSP) to improve quantization error performance and eliminate explicit LPC estimation by directly predicting the probability distribution of audio samples using a deep neural network, and employing a joint time-frequency training strategy combining short-time Fourier transform (STFT) loss with cross-entropy (CE) loss. The codec is suitable for real-time applications in resource-constrained environments. Experimental results show that the proposed codec not only outperforms traditional speech codecs but also achieves superior speech quality compared to state-of-the-art end-to-end codecs, offering a compelling balance between quality and computational cost.
Vision-Integrated High-Quality Neural Speech Coding
Authors: Yao Guo; Yang Ai; Rui-Chen Zheng; Hui-Peng Du; Xiao-Hang Jiang; Zhen-Hua Ling
Venue: Interspeech 2025 (main)

Abstract: This paper proposes a novel vision-integrated neural speech codec (VNSC), which aims to enhance speech coding quality by leveraging visual modality information. In VNSC, the image analysis-synthesis module extracts visual features from lip images, while the feature fusion module facilitates interaction between the image analysis-synthesis module and the speech coding module, transmitting visual information to assist the speech coding process. Depending on whether visual information is available during the inference stage, the feature fusion module integrates visual features into the speech coding module using either explicit integration or implicit distillation strategies. Experimental results confirm that integrating visual information effectively improves the quality of the decoded speech and enhances the noise robustness of the neural speech codec, without increasing the bitrate.
Neural Spectral Band Generation for Audio Coding
Authors: Woongjib Choi; Byeong Hyeon Kim; Hyungseob Lim; Inseon Jang; Hong-Goo Kang
Venue: Interspeech 2025 (main)

Abstract: Spectral band replication (SBR) enables bit-efficient coding by generating high-frequency bands from the low-frequency ones. However, it only utilizes coarse spectral features upon a subband-wise signal replication, limiting adaptability to diverse acoustic signals. In this paper, we explore the efficacy of a deep neural network (DNN)-based generative approach for coding the high-frequency bands, which we call neural spectral band generation (n-SBG). Specifically, we propose a DNN-based encoder-decoder structure to extract and quantize the side information related to the high-frequency components and generate the components given both the side information and the decoded core-band signals. The whole coding pipeline is optimized with generative adversarial criteria to enable the generation of perceptually plausible sound. From experiments using AAC as the core codec, we show that the proposed method achieves a better perceptual quality than HE-AAC-v1 with much less side information.
Multi-Channel Acoustic Echo Cancellation Based on Direction-of-Arrival Estimation
Authors: Fei Zhao; Xueliang Zhang; Zhong-Qiu Wang
Venue: Interspeech 2025 (main)

Abstract: Acoustic echo cancellation (AEC) is an important speech signal processing technology that can remove echoes from microphone signals to enable natural-sounding full-duplex speech communication. While single-channel AEC is widely adopted, multi-channel AEC can leverage spatial cues afforded by multiple microphones to achieve better performance. Existing multi-channel AEC approaches typically combine beamforming with deep neural networks (DNN). This work proposes a two-stage algorithm that enhances multi-channel AEC by incorporating sound source directional cues. Specifically, a lightweight DNN is first trained to predict the sound source directions, and then the predicted directional information, multi-channel microphone signals, and single-channel far-end signal are jointly fed into an AEC network to estimate the near-end signal. Evaluation results show that the proposed algorithm outperforms baseline approaches and exhibits robust generalization across diverse acoustic environments.
Simultaneous Masked and Unmasked Decoding with Speculative Decoding Masking for Fast ASR without Accuracy Loss
Authors: Koji Okabe; Hitoshi Yamamoto
Venue: Interspeech 2025 (main)

Abstract: In this paper, we introduce two methods, Simultaneous Masked and Unmasked Decoding (SMUD) and speculative decoding masking, into Partially autoregressive (PAR) decoding. These methods achieve the same recognition accuracy as Autoregressive (AR) decoding while maintaining higher computational efficiency than AR in Automatic Speech Recognition (ASR). SMUD and speculative decoding masking can accurately identify hypotheses where decoder score computation can be omitted. By omitting these computations, they achieve faster processing while obtaining the same search results as AR decoding. In TED-LIUM2 evaluations, SMUD with speculative decoding masking achieved a WER of 7.3% and an RTF of 0.41, as compared to AR's WER of 7.3% and RTF of 0.59, showcasing the methodâs ability to maintain the same high accuracy as AR while enhancing computational efficiency.
WIND: Accelerated RNN-T Decoding with Windowed Inference for Non-blank Detection
Authors: Hainan Xu; Vladimir Bataev; Lilit Grigoryan; Boris Ginsburg
Venue: Interspeech 2025 (main)

Abstract: We propose Windowed Inference for Non-blank Detection (WIND), a novel strategy that significantly accelerates RNN-T inference without compromising model accuracy. During model inference, instead of processing frames sequentially, WIND processes multiple frames simultaneously within a window in parallel, allowing the model to quickly locate non-blank predictions during decoding, resulting in significant speed-ups. We implement WIND for greedy decoding, batched greedy decoding with label-looping techniques, and also propose a novel beam-search decoding method. Experiments on multiple datasets with different conditions show that our method, when operating in greedy modes, speeds up as much as 2.4X compared to the baseline sequential approach while maintaining identical Word Error Rate (WER) performance. Our beam-search algorithm achieves slightly better accuracy than alternative methods, with significantly improved speed. We will open-source our WIND implementation.
NGPU-LM: GPU-Accelerated N-Gram Language Model for Context-Biasing in Greedy ASR Decoding
Authors: Vladimir Bataev; Andrei Andrusenko; Lilit Grigoryan; Aleksandr Laptev; Vitaly Lavrukhin; Boris Ginsburg
Venue: Interspeech 2025 (main)

Abstract: Statistical n-gram language models are widely used for context-biasing tasks in Automatic Speech Recognition (ASR). However, existing implementations lack computational efficiency due to poor parallelization, making context-biasing less appealing for industrial use. This work rethinks data structures for statistical n-gram language models to enable fast and parallel operations for GPU-optimized inference. Our approach, named NGPU-LM, introduces customizable greedy decoding for all major ASR model types - including transducers, attention encoder-decoder models, and CTC - with less than 7% computational overhead. The proposed approach can eliminate more than 50% of the accuracy gap between greedy and beam search for out-of-domain scenarios while avoiding significant slowdown caused by beam search. The implementation of the proposed NGPU-LM will be open-sourced.
Pushing the Limits of Beam Search Decoding for Transducer-based ASR models
Authors: Lilit Grigoryan; Vladimir Bataev; Andrei Andrusenko; Hainan Xu; Vitaly Lavrukhin; Boris Ginsburg
Venue: Interspeech 2025 (main)

Abstract: Transducer models have emerged as a promising choice for end-to-end ASR systems, offering a balanced trade-off between recognition accuracy, streaming capabilities, and inference speed in greedy decoding. However, beam search significantly slows down Transducers due to repeated evaluations of key network components, limiting practical applications. This paper introduces a universal method to accelerate beam search for Transducers, enabling the implementation of two optimized algorithms: ALSD++ and AES++. The proposed method utilizes batch operations, a tree-based hypothesis structure, novel blank scoring for enhanced shallow fusion, and CUDA graph execution for efficient GPU inference. This narrows the speed gap between beam and greedy modes to only 10-20% for the whole system, achieves 14-30% relative improvement in WER compared to greedy decoding, and improves shallow fusion for low-resource up to 11% compared to existing implementations. All the algorithms are open sourced.
Skip-Salsa: Skip Synchronous Fusion of ASR LLM Decoders
Authors: Ashish Mittal; Darshan Prabhu; Sunita Sarawagi; Preethi Jyothi
Venue: Interspeech 2025 (main)

Abstract: The integration of large language models (LLMs) with ASR is increasingly explored, but remains challenging for low-resource languages. Loose coupling via N-best lists fails due to high ASR errors, while tight coupling using audio tokens requires too much data. A promising middle ground SALSA enables synchronous decoding by cascading ASR and LLM decoders via projection layers, overcoming differing tokenizations. In this work, we show that SALSA fails when the ASR and LLM tokenizations have a large token fertility gap. This problem particularly plagues low-resource languages; the ASR decoder overtokenizes LLM tokens starving the LLM decoder of sufficient audio context. To address this, we propose SKIP-SALSA, that adaptively skips ahead and advances the ASR decoder states to synchronize with the LLM. The skip size is learned via a lightweight skip predictor. SKIP-SALSA significantly improves ASR performance on multiple low-resource languages yielding up to 20% over a strong baseline.