GenAI4ELab/papercli-papers huggingface dataset:
Transmitted and Aggregated Self-Attention for Automatic Speech Recognition
Authors: Tian-Hao Zhang; Xinyuan Qian; Feng Chen; Xu-Cheng Yin
Venue: Interspeech 2024 (main)

Abstract: Transformer based models have recently achieved outstanding progress in ASR system. The attention maps are generated in self-attention to capture temporal relationships among input tokens and heavily influence transformer performance. Many works demonstrate that attention maps of different layers incorporate various contextual scopes of information. We believe that the information from diverse attention maps is valuable and complementary. This inspires us with a novel proposal, namely Transmitted and Aggregated Self-Attention (TASA), which leverages the information of attention maps in each layer to improve the overall performance. In particular, we design Residual-TASA and Dense-TASA which are distinguished by using attention maps of the previous layer or all previous layers, respectively. Extensive experiments demonstrate that the proposed method achieving up to 10.62% CER and 7.36% WER relative reduction conducted on AISHELL-1 and LibriSpeech datasets, respectively.
MULTI-CONVFORMER: Extending Conformer with Multiple Convolution Kernels
Authors: Darshan Prabhu; Yifan Peng; Preethi Jyothi; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: Convolutions have become essential in state-of-the-art end-to-end Automatic Speech Recognition (ASR) systems due to their efficient modelling of local context. Notably, its use in Conformers has led to superior performance compared to vanilla Transformer-based ASR systems. While components other than the convolution module in the Conformer have been reexamined, altering the convolution module itself has been far less explored. Towards this, we introduce MULTI-CONVFORMER that uses multiple convolution kernels within the convolution module of the Conformer in conjunction with gating. This helps in improved modeling of local dependencies at varying granularities. Our model rivals existing Conformer variants such as CgMLP and E-Branchformer in performance, while being more parameter efficient. We empirically compare our approach with Conformer and its variants across four different datasets and three different modelling paradigms and show up to 8% relative word error rate (WER) improvements.
Exploring the Capability of Mamba in Speech Applications
Authors: Koichi Miyazaki; Yoshiki Masuyama; Masato Murata
Venue: Interspeech 2024 (main)

Abstract: This paper explores the capability of Mamba, a recently proposed architecture based on state space models (SSMs), as a competitive alternative to Transformer-based models. In the speech domain, well-designed Transformer-based models, such as the Conformer and E-Branchformer, have become the de facto standards. Extensive evaluations have demonstrated the effectiveness of these Transformer-based models across a wide range of speech tasks. In contrast, the evaluation of SSMs has been limited to a few tasks, such as automatic speech recognition (ASR) and speech synthesis. In this paper, we compared Mamba with state-of-the-art Transformer variants in various speech applications, including ASR, text-to-speech, spoken language understanding, and speech summarization. Experimental evaluations revealed that Mamba achieves comparable or better performance than Transformer-based models, and demonstrated its efficiency in long-form speech processing.
Lightweight Transducer Based on Frame-Level Criterion
Authors: Genshun Wan; Mengzhi Wang; Tingzhi Mao; Hang Chen; Zhongfu Ye
Venue: Interspeech 2024 (main)

Abstract: The transducer model trained based on sequence-level criterion requires a lot of memory due to the generation of the large probability matrix. We proposed a lightweight transducer model based on frame-level criterion, which uses the results of the CTC forced alignment algorithm to determine the label for each frame. Then the encoder output can be combined with the decoder output at the corresponding time, rather than adding each element output by the encoder to each element output by the decoder as in the transducer. This significantly reduces memory and computation requirements. To address the problem of imbalanced classification caused by excessive blanks in the label, we decouple the blank and non-blank probabilities and truncate the gradient of the blank classifier to the main network. This enables the lightweight transducer achieving similar results to transducer. Additionally, we use richer information to predict the probability of blank, achieving superior results to transducer.
Exploring the limits of decoder-only models trained on public speech recognition corpora
Authors: Ankit Gupta; George Saon; Brian Kingsbury
Venue: Interspeech 2024 (main)

Abstract: The emergence of industrial-scale automatic speech recognition (ASR) models such as Whisper and USM, trained on 1M hours of weakly labelled and 12M hours of audio only proprietary data respectively, has led to a stronger need for large scale public ASR corpora and competitive open source pipelines. Unlike the said models, large language models are typically based on Transformer decoders, and it remains unclear if decoder-only models trained on public data alone can deliver competitive performance. In this work, we investigate factors such as choice of training datasets and modeling components necessary for obtaining the best performance using only public English ASR corpora. Our Decoder-Only Transformer for ASR (DOTA) model comprehensively outperforms the encoder-decoder open source replication of Whisper (OWSM) on nearly all English ASR benchmarks and outperforms Whisper large-v3 on 6 out of 15 test sets. We release our codebase and model checkpoints under permissive license.
Contextual Biasing Speech Recognition in Speech-enhanced Large Language Model
Authors: Xun Gong; Anqi Lv; Zhiming Wang; Yanmin Qian
Venue: Interspeech 2024 (main)

Abstract: Recently, the rapid advancements in audio- and speech-enhanced large language models (SpeechLLMs), such as Qwen-Audio and SALMONN, have significantly propelled automatic speech recognition (ASR) forward. However, despite the improvements in universal recognition capabilities, bias word recognition persists as a prominent challenge for SpeechLLM, and is not extensively studied. In this study, we introduce two contextual biasing strategies aimed at improving the bias word recognition of SpeechLLM. Firstly, we explored two types of biasing prompts for SpeechLLMs, achieving 10% relative reduction in bias word error rate (WER). However, as the size of the bias list increased, performance significantly declined due to hallucination. Subsequently, we built the biasing fusion network for SpeechLLM that integrates high-level bias embeddings with the SpeechLLM framework. Our experiments conducted on the LibriSpeech test-clean/-other datasets demonstrate that our method achieves up to 10%/35% relative reduction in overall/bias WER compared to our baseline.
Towards Effective and Efficient Non-autoregressive Decoding Using Block-based Attention Mask
Authors: Tianzi Wang; Xurong Xie; Zhaoqing Li; Shoukang Hu; Zengrui Jin; Jiajun Deng; Mingyu Cui; Shujie Hu; Mengzhe Geng; Guinan Li; Helen Meng; Xunying Liu
Venue: Interspeech 2024 (main)

Abstract: This paper proposes a novel non-autoregressive (NAR) block-based Attention Mask Decoder (AMD) that flexibly balances performance-efficiency trade-offs for Conformer ASR systems. AMD performs parallel NAR inference within contiguous blocks of output labels that are concealed using attention masks, while conducting left-to-right AR prediction and history context amalgamation between blocks. A beam search algorithm is designed to leverage a dynamic fusion of CTC, AR Decoder, and AMD probabilities. Experiments on the LibriSpeech-100hr corpus suggest the tripartite Decoder incorporating the AMD module produces a maximum decoding speed-up ratio of 1.73x over the baseline CTC+AR decoding, while incurring no statistically significant word error rate (WER) increase on the test sets. When operating with the same decoding real time factors,  statistically significant WER reductions of up to 0.7% and 0.3% absolute (5.3% and 6.1% relative) were obtained over the CTC+AR baseline.
E-Paraformer: A Faster and Better Parallel Transformer for Non-autoregressive End-to-End Mandarin Speech Recognition
Authors: Kun Zou; Fengyun Tan; Ziyang Zhuang; Chenfeng Miao; Tao Wei; Shaodan Zhai; Zijian Li; Wei Hu; Shaojun Wang; Jing Xiao
Venue: Interspeech 2024 (main)

Abstract: Paraformer is a powerful non-autoregressive (NAR) model for Mandarin speech recognition. It relies on Continuous Integrate-and-Fire (CIF) to implement parallel decoding. However, the CIF mechanism needs to recursively obtain the acoustic boundary of the emitted token, which will lead to inefficiency. In this paper, we introduce a novel monotonic alignment mechanism as an alternative to CIF that can convert frame-level embeddings into token-level embeddings in parallel. Combining this method with other improvements to the model structure, we design a faster and better parallel transformer called the Efficient Paraformer (E-Paraformer). Experiments are performed on the AISHELL-1 benchmark. Compared to Paraformer baseline, the E-Paraformer achieves character error rates (CER) of 4.36%/4.79% on the AISHELL-1 dev/test dataset, representing 7.8% and 6.3% (relative) reductions, respectively. Moreover, it achieves about 2x inference speedup and 1.35x training speedup.
Beam-search SIEVE for low-memory speech recognition
Authors: Martino Ciaperoni; Athanasios Katsamanis; Aristides Gionis; Panagiotis Karras
Venue: Interspeech 2024 (main)

Abstract: A capacity to recognize speech offline eliminates privacy concerns and the need for an internet connection. Despite efforts to reduce the memory demands of speech recognition systems, these demands remain formidable and thus popular tools such as Kaldi run best via cloud computing. The key bottleneck arises form the fact that a bedrock of such tools, the Viterbi algorithm, requires memory that grows linearly with utterance length even when contained via beam search. A recent recasting of the Viterbi algorithm, SIEVE, eliminates the path length factor from space complexity, but with a significant practical runtime overhead. In this paper, we develop a variant of SIEVE that lessens this runtime overhead via beam search, retains the decoding quality of standard beam search, and waives its linearly growing memory bottleneck. This space-complexity reduction is orthogonal to decoding quality and complementary to memory savings in model representation and training.
Speed of Light Exact Greedy Decoding for RNN-T Speech Recognition Models on GPU
Authors: Daniel Galvez; Vladimir Bataev; Hainan Xu; Tim Kaldewey
Venue: Interspeech 2024 (main)

Abstract: The vast majority of inference time for RNN Transducer (RNN-T) models today is spent on decoding. Current state-of-the-art RNN-T decoding implementations leave the GPU idle 80% of the time. Leveraging a new CUDA 12.4 feature, CUDA graph conditional nodes, we present an exact GPU-based implementation of greedy decoding for RNN-T models that eliminates this idle time. Our optimizations speed up a 1.1 billion parameter RNN-T model end-to-end by a factor of 2.5x. This technique can applied to the "label looping" alternative greedy decoding algorithm as well, achieving 1.7x and 1.4x end-to-end speedups when applied to 1.1 billion parameter RNN-T and Token and Duration Transducer models respectively. This work enables a 1.1 billion parameter RNN-T model to run only 16% slower than a similarly sized CTC model, contradicting the common belief that RNN-T models are not suitable for high throughput inference. The implementation is available in NVIDIA NeMo.
Contextual Biasing with the Knuth-Morris-Pratt Matching Algorithm
Authors: Weiran Wang; Zelin Wu; Diamantino Caseiro; Tsendsuren Munkhdalai; Khe Chai Sim; Pat Rondon; Golan Pundak; Gan Song; Rohit Prabhavalkar; Zhong Meng; Ding Zhao; Tara Sainath; Yanzhang He; Pedro Moreno Mengibar
Venue: Interspeech 2024 (main)

Abstract: We propose a GPU/TPU-friendly implementation for contextual biasing based on the Knuth-Morris-Pratt (KMP) pattern matching algorithm. Our algorithms simulate classical search-based biasing approaches which are often implemented in the weighted finite state transducer (WFST) framework, with careful considerations on memory footprint and efficiency by vectorization. We design scoring mechanisms such that, during beam search, a token extension receives a bonus if it extends matching into a biasing phrase, and receives a penalty to cancel previously received bonus otherwise. Our methods could be incorporated in either the shallow fusion or on-the-fly rescoring manner, to trade off accuracy with efficiency. On a large-scale voice search dataset, our method achieves significant word error rate (WER) reductions on biasing test sets without introducing additional model parameters, and yields further performance gain when combined with a model-based biasing method.
Text-only Domain Adaptation for CTC-based Speech Recognition through Substitution of Implicit Linguistic Information in the Search Space
Authors: Tatsunari Takagi; Yukoh Wakabayashi; Atsunori Ogawa; Norihide Kitaoka
Venue: Interspeech 2024 (main)

Abstract: Domain adaptation using only language models in Automatic Speech Recognition (ASR) has been widely studied because of its practicality. Still, it remains challenging for non-autoregressive ASR models such as Connectionist Temporal Classification (CTC)-based ones. Against this background, this study addresses a text-only domain adaptation method for CTC-based ASR models by leveraging the Density Ratio Approach (DRA). Our method combines a beam search algorithm for substituting linguistic information in DRA, accommodated to the CTC decoding procedure, and a language model adaptation method considered the conditional independence assumption of CTC. We conducted domain adaptation experiments for character-level ASR with the Corpus of Spontaneous Japanese (CSJ) and sub-word ASR with the English-language LibriSpeech and GigaSpeech corpora. The experimental results confirmed that our proposed method achieved improved accuracy in Japanese and English compared to the Shallow Fusion method.
Pitch-Aware RNN-T for Mandarin Chinese Mispronunciation Detection and Diagnosis
Authors: Xintong Wang; Mingqian Shi; Ye Wang
Venue: Interspeech 2024 (main)

Abstract: Mispronunciation Detection and Diagnosis (MDD) systems, leveraging Automatic Speech Recognition (ASR), face two main challenges in Mandarin Chinese: 1) The two-stage models create an information gap between the phoneme or tone classification stage and the MDD stage. 2) The scarcity of Mandarin MDD datasets limits model training. In this paper, we introduce a stateless RNN-T model for Mandarin MDD, utilizing HuBERT features with pitch embedding through a Pitch Fusion Block. Our model, trained solely on native speaker data, shows a 3% improvement in Phone Error Rate and a 7% increase in False Acceptance Rate over the state-of-the-art baseline in non-native scenarios.
MultiPA: A Multi-task Speech Pronunciation Assessment Model for Open Response Scenarios
Authors: Yu-Wen Chen; Zhou Yu; Julia Hirschberg
Venue: Interspeech 2024 (main)

Abstract: Pronunciation assessment models designed for open response scenarios enable users to practice language skills in a manner similar to real-life communication. However, previous open-response pronunciation assessment models have predominantly focused on a single pronunciation task, such as sentence-level accuracy, rather than offering a comprehensive assessment in various aspects. We propose MultiPA, a Multitask Pronunciation Assessment model that provides sentence-level accuracy, fluency, prosody, and word-level accuracy assessment for open responses. We examined the correlation between different pronunciation tasks and showed the benefits of multi-task learning. Our model reached the state-of-the-art performance on existing in-domain data sets and effectively generalized to an out-of-domain dataset that we newly collected. The experimental results demonstrate the practical utility of our model in real-world applications.
A Framework for Phoneme-Level Pronunciation Assessment Using CTC
Authors: Xinwei Cao; Zijian Fan; TorbjÃ¸rn Svendsen; Giampiero Salvi
Venue: Interspeech 2024 (main)

Abstract: Traditional phoneme-level goodness of pronunciation (GOP) methods require phoneme to speech alignment. The drawback is that these methods, by their definitions, are prone to alignment errors and preclude the possibility of deletion and insertion errors in pronunciation. We produce experimental evidence that CTC-based methods can be used in traditional GOP estimation in spite of their âpeakyâ output behaviour and may be less prone to alignment errors than traditional methods. We also propose a new framework for GOP estimation based on CTC-trained model that is independent of speech-phoneme alignment. By accounting for deletion and insertions as well as substitution errors, we show that our framework outperform alignment-based method. Our experimental results are based on the CMU-kids dataset for child speech and on the Speechocean762 containing both child and adult speech speakers. Our best method achieves 29.02% relative improvement over the baseline GOP methods.
Phonological-Level Mispronunciation Detection and Diagnosis
Authors: Mostafa Shahin; Beena Ahmed
Venue: Interspeech 2024 (main)

Abstract: The automatic identification and analysis of pronunciation errors, known as mispronunciation detection and diagnosis (MDD), is vital in computer-aided pronunciation learning (CAPL) tools for second-language (L2) learning. Existing MDD methods focus on analyzing phonemes, but they can only detect categorical errors for phonemes with sufficient training data. Due to the unpredictable nature of non-native speakersâ pronunciation errors and limited training datasets, modelling all mispronunciations becomes impractical. Additionally, phoneme-level MDD approaches provide limited diagnostic information. In our proposed approach, we detect phonological features, breaking down phoneme production into elementary components related to the articulatory system, offering more informative feedback to learners. Applied to L2 English speech data, it outperformed traditional phoneme-level methods, reducing false acceptance rate (FAR), false rejection rate (FRR), and diagnostic error rate (DER).
Acoustic Feature Mixup for Balanced Multi-aspect Pronunciation Assessment
Authors: Heejin Do; Wonjun Lee; Gary Geunbae Lee
Venue: Interspeech 2024 (main)

Abstract: In automated pronunciation assessment, recent emphasis progressively lies on evaluating multiple aspects to provide enriched feedback. However, acquiring multi-aspect-score labeled data for non-native language learners' speech poses challenges; moreover, it often leads to score-imbalanced distributions. In this paper, we propose two Acoustic Feature Mixup strategies, linearly and non-linearly interpolating with the in-batch averaged feature, to address data scarcity and score-label imbalances. Primarily using goodness-of-pronunciation as an acoustic feature, we tailor mixup designs to suit pronunciation assessment. Further, we integrate fine-grained error-rate features by comparing speech recognition results with the original answer phonemes, giving direct hints for mispronunciation. Effective mixing of the acoustic features notably enhances overall scoring performances on the speechocean762 dataset, and detailed analysis highlights our potential to predict unseen distortions.
Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task
Authors: Nhan Phan; Anna von Zansen; Maria Kautonen; Ekaterina Voskoboinik; Tamas Grosz; Raili Hilden; Mikko Kurimo
Venue: Interspeech 2024 (main)

Abstract: We propose a framework to address several unsolved challenges in second language (L2) automatic speaking assessment (ASA) and feedback. The challenges include: 1. ASA of visual task completion, 2. automated content grading and explanation of spontaneous L2 speech, 3. corrective feedback generation for L2 learners, and 4. all the above for a language that has minimal speech data of L2 learners. The proposed solution combines visual natural language generation (NLG), automatic speech recognition (ASR) and prompting a large language model (LLM) for low-resource L2 learners. We describe the solution and the outcomes of our case study for a picture description task in Finnish. Our results indicate substantial agreement with human experts in grading, explanation and feedback. This framework has the potential for a significant impact in constructing next-generation computer-assisted language learning systems to provide automatic scoring with feedback for learners of low-resource languages.
Query-by-Example Keyword Spotting Using Spectral-Temporal Graph Attentive Pooling and Multi-Task Learning
Authors: Zhenyu Wang; Shuyu Kong; Li Wan; Biqiao Zhang; Yiteng Huang; Mumin Jin; Ming Sun; Xin Lei; Zhaojun Yang
Venue: Interspeech 2024 (main)

Abstract: Existing keyword spotting (KWS) systems primarily rely on predefined keyword phrases. However, the ability to recognize customized keywords is crucial for tailoring interactions with intelligent devices. In this paper, we present a novel Query-by-Example (QbyE) KWS system that employs spectral-temporal graph attentive pooling and multi-task learning. This framework aims to effectively learn speaker-invariant and linguistic-informative embeddings for QbyE KWS tasks. Within this framework, we investigate three distinct network architectures for encoder modeling: LiCoNet, Conformer and ECAPA_TDNN. The experimental results on a substantial internal dataset of 629 speakers have demonstrated the effectiveness of the proposed QbyE framework in maximizing the potential of simpler models such as LiCoNet. Particularly, LiCoNet, which is 13x more efficient, achieves comparable performance to the computationally intensive Conformer model (1.98% vs. 1.63% FRR at 0.3 FAs/Hr).
Relational Proxy Loss for Audio-Text based Keyword Spotting
Authors: Youngmoon Jung; Seungjin Lee; Joon-Young Yang; Jaeyoung Roh; Chang Woo Han; Hoon-Young Cho
Venue: Interspeech 2024 (main)

Abstract: In recent years, there has been an increasing focus on user convenience, leading to increased interest in text-based keyword enrollment systems for keyword spotting (KWS). Since the system utilizes text input during the enrollment phase and audio input during actual usage, we call this task audio-text based KWS. To enable this task, both acoustic and text encoders are typically trained using deep metric learning loss functions, such as triplet- and proxy-based losses. This study aims to improve existing methods by leveraging the structural relations within acoustic embeddings and within text embeddings. Unlike previous studies that only compare acoustic and text embeddings on a point-to-point basis, our approach focuses on the relational structures within the embedding space by introducing the concept of Relational Proxy Loss (RPL). By incorporating RPL, we demonstrated improved performance on the Wall Street Journal (WSJ) corpus.