GenAI4ELab/papercli-papers huggingface dataset:
TfCleanformer: A streaming, array-agnostic, full- and sub-band modeling front-end for robust ASR
Authors: Jens Heitkaemper; Joe Caroselli; Arun Narayanan; Nathan Howard
Venue: Interspeech 2024 (main)

Abstract: Multiple recent publications have demonstrated the benefits of neural network based enhancement in the time-frequency domain. This paper builds on those findings to improve upon a recently published streaming, array agnostic multi-channel enhancement system called Cleanformer. The proposed streaming enhancement system achieves competitive results against a non-causal state-of-the-art model on a source separation task, outperforming Cleanformer. Additionally, the presented model improves upon Cleanformer enhancement results in multiple challenging environments without introducing further latency.  A short ablation study is performed to evaluate the influence of the proposed changes on the improved performance.
Improving Streaming Speech Recognition With Time-Shifted Contextual Attention And Dynamic Right Context Masking
Authors: Khanh Le; Duc Chau
Venue: Interspeech 2024 (main)

Abstract: Chunk-based inference stands out as a popular approach in developing real-time streaming speech recognition, valued for its simplicity and efficiency. However, because it restricts the model's focus to only the history and current chunk context, it may result in performance degradation in scenarios that demand consideration of future context. Addressing this, we propose a novel approach featuring Time-Shifted Contextual Attention (TSCA) and Dynamic Right Context (DRC) masking. Our method shows a relative word error rate reduction of 10 to 13.9\% on the Librispeech dataset with the inclusion of in-context future information provided by TSCA. Moreover, we present a streaming automatic speech recognition pipeline that facilitates the integration of TSCA with minimal user-perceived latency, while also enabling batch processing capability, making it practical for various applications.
Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection
Authors: Haoyu Wang; Guoqiang Hu; Guodong Lin; Wei-Qiang Zhang; Jian Li
Venue: Interspeech 2024 (main)

Abstract: As a robust and large-scale multilingual speech recognition model, Whisper has demonstrated impressive results in many low-resource and out-of-distribution scenarios. However, its encoder-decoder structure hinders its application to streaming speech recognition. In this paper, we introduce Simul-Whisper, which uses the time alignment embedded in Whisper's cross-attention to guide auto-regressive decoding and achieve chunk-based streaming ASR without any fine-tuning of the pre-trained model. Furthermore, we observe the negative effect of the truncated words at the chunk boundaries on the decoding results and propose an integrate-and-fire-based truncation detection model to address this issue.  Experiments on multiple languages and Whisper architectures show that Simul-Whisper achieves an average absolute word error rate degradation of only 1.46% at a chunk size of 1 second, which significantly outperforms the current state-of-the-art baseline.
Dynamic Data Pruning for Automatic Speech Recognition
Authors: Qiao Xiao; Pingchuan Ma; Adriana Fernandez-Lopez; Boqian Wu; Lu Yin; Stavros Petridis; Mykola Pechenizkiy; Maja Pantic; Decebal Constantin Mocanu; Shiwei Liu
Venue: Interspeech 2024 (main)

Abstract: The recent success of Automatic Speech Recognition (ASR) is largely attributed to the ever-growing amount of training data. However, this trend has made model training prohibitively costly and imposed computational demands. While data pruning has been proposed to mitigate this issue by identifying a small subset of relevant data, its application in ASR has been barely explored, and existing works often entail significant overhead to achieve meaningful results. To fill this gap, this paper presents the first investigation of dynamic data pruning for ASR, finding that we can reach the full-data performance by dynamically selecting 70% of data. Furthermore, we introduce Dynamic Data Pruning for ASR (DDP-ASR), which offers several fine-grained pruning granularities specifically tailored for speech-related datasets, going beyond the conventional pruning of entire time sequences. Our intensive experiments show that DDP-ASR can save up to 1.6Ã training time with negligible performance loss.
Mitigating Overfitting in Structured Pruning of ASR Models with Gradient-Guided Parameter Regularization
Authors: Dong-Hyun Kim; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Recent advancements in automatic speech recognition such as Wav2vec 2.0 and Whisper, confront deployment challenges due to their substantial model parameters. Model compression through joint distillation and structured pruning emerges as an effective solution but still faces overfitting and catastrophic forgetting, exacerbated by domain shifts or limited data availability. To address this issue, we propose the gradient-guided parameter regularization method aimed at maintaining the model's generality. Our approach employs gradient values to detect overfit-prone parameters in the student model and subsequently regularize these parameters to align closely with their counterparts in the teacher model. Through extensive experiments, we demonstrate the efficacy of our approach in reducing overfitting and enhancing performance, especially in scenarios characterized by domain shifts and limited data availability.
SparseWAV: Fast and Accurate One-Shot Unstructured Pruning for Large Speech Foundation Models
Authors: Tianteng Gu; Bei Liu; Hang Shao; Yanmin Qian
Venue: Interspeech 2024 (main)

Abstract: Self-supervised speech representation learning has shown remarkable capability in automatic speech recognition. However, it requires substantial computations and storage capacity. Pruning is an effective method for model compression. In this work, we propose SparseWAV, a fast and accurate unstructured pruning framework designed for large speech foundation models, which can efficiently remove unimportant parameters without sacrificing performance. It adaptively determines the sparsity ratio for each weight matrix within pre-trained models and updates the remaining parameters to compensate for the eliminated ones. Experiments on Librispeech demonstrate the proposed method can remove 80% of the parameters of pre-trained large speech foundation models with negligible performance loss. Compared to previous works, our resulting models achieves up to 30% improvement in performance under similar parameters. Meanwhile, the compression algorithm's time consumption is reduced by up to 1080x.
One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model
Authors: Zhaoqing Li; Haoning Xu; Tianzi Wang; Shoukang Hu; Zengrui Jin; Shujie Hu; Jiajun Deng; Mingyu Cui; Mengzhe Geng; Xunying Liu
Venue: Interspeech 2024 (main)

Abstract: We propose a novel one-pass multiple ASR systems joint compression and quantization approach using an all-in-one neural model. A single compression cycle allows multiple nested systems with varying Encoder depths, widths, and quantization precision settings to be simultaneously constructed without the need to train and store individual target systems separately. Experiments consistently demonstrate the multiple ASR systems compressed in a single all-in-one model produced a word error rate (WER) comparable to, or lower by up to 1.01% absolute (6.98% relative) than individually trained systems of equal complexity. A 3.4x overall system compression and training time speed-up was achieved. Maximum model size compression ratios of 12.8x and 3.93x were obtained over the baseline Switchboard-300hr Conformer and LibriSpeech-100hr fine-tuned wav2vec2.0 models, respectively, incurring no statistically significant WER increase.
USM RNN-T model weights binarization
Authors: Oleg Rybakov; Dmitriy Serdyuk; Chengjian Zheng
Venue: Interspeech 2024 (main)

Abstract: Large-scale universal speech models (USM) are already used in production. However, as the model size grows, the serving cost grows too. Serving cost of large models is dominated by model size that is why model size reduction is an important research topic. In this work we are focused on model size reduction using weights only quantization. We present the weights binarization of USM Recurrent Neural Network Transducer (RNN-T) and show that its model size can be reduced by 15.9x times at cost of word error rate (WER) increase by only 1.9% in comparison to the float32 model. It makes it attractive for practical applications.
DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Models
Authors: Tzu-Quan Lin; Hung-yi Lee; Hao Tang
Venue: Interspeech 2024 (main)

Abstract: Self-supervised speech models have shown to be useful for various tasks, but their large size limits the use in devices with low computing power and memory. In this work, we explore early exit, an approach for reducing latency by exiting the forward process of a network early. Most approaches of early exit need a separate early exit model for each task, with some even requiring fine-tuning of the entire pretrained model. We introduce Data Adaptive Self-Supervised Early Exit (DAISY), an approach that decides when to exit based on the self-supervised loss, eliminating the need for multiple round of training and fine-tuning. DAISY matches the performance of HuBERT on the MiniSUPERB benchmark, but with much faster inference times. Our analysis on the adaptivity of DAISY shows that the model exits early (using fewer layers) on clean data while exits late (using more layers) on noisy data, dynamically adjusting the computational cost of inference based on the noise level of each sample.
RepTor: Re-parameterizable Temporal Convolution for Keyword Spotting via Differentiable Kernel Search
Authors: Eunik Park; Daehyun Ahn; Hyungjun Kim
Venue: Interspeech 2024 (main)

Abstract: Keyword spotting (KWS) is an essential in speech recognition, particularly for smart device interfaces. However, it is challenging to design KWS models that are both efficient and accurate, especially for mobile devices with limited resources. In this paper, we propose RepTor-family models, where a structural re-parameterization technique was applied to temporal convolutions in order to improve the efficiency of KWS systems in various environments. The re-parameterized RepTor models have a single-branch plain feed-forward architecture without any residual connections and, therefore, can be efficiently computed on mobile devices. We further propose a latency-aware differentiable kernel search method to find the optimal kernel size for each convolution layer. The proposed RepTor-A~E models achieve 97.9% top-1 accuracy on the Google Speech Command dataset v2 with a latency of 183ms on the Galaxy S10. They outperform state-of-the-art KWS models regarding the accuracy-latency trade-off.
Global-Local Convolution with Spiking Neural Networks for Energy-efficient Keyword Spotting
Authors: Shuai Wang; Dehao Zhang; Kexin Shi; Yuchen Wang; Wenjie Wei; Jibin Wu; Malu Zhang
Venue: Interspeech 2024 (main)

Abstract: Thanks to Deep Neural Networks (DNNs), the accuracy of Keyword Spotting (KWS) has made substantial progress. However, as KWS systems are usually implemented on edge devices, energy efficiency becomes a critical requirement besides performance. Here, we take advantage of spiking neural networks' energy efficiency and propose an end-to-end lightweight KWS model. The model consists of two innovative modules: 1) Global-Local Spiking Convolution (GLSC) module and 2) Bottleneck-PLIF module. Compared to the hand-crafted feature extraction methods, the GLSC module achieves speech feature extraction that is sparser, more energy-efficient, and yields better performance. The Bottleneck-PLIF module further processes the signals from GLSC with the aim to achieve higher accuracy with fewer parameters. Extensive experiments are conducted on the Google Speech Commands Dataset (V1 and V2). The results show our method achieves competitive performance among SNN-based KWS models with fewer parameters.
ED-sKWS: Early-Decision Spiking Neural Networks for Rapid, and Energy-Efficient Keyword Spotting
Authors: Zeyang Song; Qianhui Liu; Qu Yang; Yizhou Peng; Haizhou Li
Venue: Interspeech 2024 (main)

Abstract: Keyword Spotting (KWS) is essential in edge computing requiring rapid and energy-efficient responses. Spiking Neural Networks (SNNs) are well-suited for KWS for their efficiency and temporal capacity for speech. To further reduce the latency and energy consumption, this study introduces ED-sKWS, an SNN-based KWS model with an early-decision mechanism that can stop speech processing and output the result before the end of speech utterance. Furthermore, we introduce a Cumulative Temporal (CT) loss that can enhance prediction accuracy at both the intermediate and final timesteps. To evaluate early-decision performance, we present the SC-100 dataset including 100 speech commands with beginning and end timestamp annotation. Experiments on the Google Speech Commands v2 and our SC-100 datasets show that ED-sKWS maintains competitive accuracy with 61% timesteps and 52% energy consumption compared to SNN models without early-decision mechanism, ensuring rapid response and energy efficiency.
A Small and Fast BERT for Chinese Medical Punctuation Restoration
Authors: Tongtao Ling; Yutao Lai; Lei Chen; Shilei Huang; Yi Liu
Venue: Interspeech 2024 (main)

Abstract: In clinical dictation, utterances after automatic speech recognition (ASR) without explicit punctuation marks may lead to the misunderstanding of dictated reports. To provide a precise and understandable clinical report with ASR, automatic punctuation restoration (APR) is required. Considering a practical scenario, we propose a fast and lightweight pre-trained model for Chinese medical punctuation restoration based on the âpre-training and fine-tuningâ paradigm. In this work, we distill pre-trained models by incorporating supervised contrastive learning and a novel auxiliary pre-training task (Punctuation Mark Prediction) to make it well-suited for punctuation restoration. We then reformulate APR as a slot tagging problem in the fine-tuning stage to bridge the gap between pre-training and fine-tuning. Our experiments on various distilled models reveal that our model can achieve 95% performance with a 10% model size relative to the state-of-the-art Chinese RoBERTa.
Quantification of stylistic differences in human- and ASR-produced transcripts of African American English
Authors: Annika Heuser; Tyler Kendall; Miguel del Rio; Quinn McNamara; Nishchal Bhandari; Corey Miller; MigÃ¼el JettÃ©
Venue: Interspeech 2024 (main)

Abstract: Common measures of accuracy used to assess the performance of automatic speech recognition (ASR) systems, as well as human transcribers, conflate multiple sources of error. Stylistic differences, such as verbatim vs non-verbatim, can play a significant role in ASR performance evaluation when differences exist between training and test datasets. The problem is compounded for speech from underrepresented varieties, where the speech to orthography mapping is not as standardized. We categorize the kinds of stylistic differences between 6 transcription versions, 4 human- and 2 ASR-produced, of 10 hours of African American English (AAE) speech. Focusing on verbatim features and AAE morphosyntactic features, we investigate the interactions of these categories with how well transcripts can be compared via word error rate (WER). The results, and overall analysis, help clarify how ASR outputs are a function of the decisions made by the training dataâs human transcribers.
Beyond Levenshtein: Leveraging Multiple Algorithms for Robust Word Error Rate Computations And Granular Error Classifications
Authors: Korbinian Kuhn; Verena Kersken; Gottfried Zimmermann
Venue: Interspeech 2024 (main)

Abstract: The Word Error Rate (WER) is the common measure of accuracy for Automatic Speech Recognition (ASR). Transcripts are usually pre-processed by substituting specific characters to account for non-semantic differences. As a result of this normalisation, information on the accuracy of punctuation or capitalisation is lost. We present a non-destructive, token-based approach using an extended Levenshtein distance algorithm to compute a robust WER and additional orthographic metrics. Transcription errors are also classified more granularly by existing string similarity and phonetic algorithms. An evaluation on several datasets demonstrates the practical equivalence of our approach compared to common WER computations. We also provide an exemplary analysis of derived use cases, such as a punctuation error rate, and a web application for interactive use and visualisation of our implementation. The code is available open-source.
Comparing ASR Systems in the Context of Speech Disfluencies
Authors: Maria Teleki; Xiangjue Dong; Soohwan Kim; James Caverlee
Venue: Interspeech 2024 (main)

Abstract: In this work, we evaluate the disfluency capabilities of two automatic speech recognition systems - Google ASR and WhisperX - through a study of 10 human-annotated podcast episodes and a larger set of 82,601 podcast episodes. We employ a state-of-the-art disfluency annotation model to perform a fine-grained analysis of the disfluencies in both the scripted and non-scripted podcasts. We find, on the set of 10 podcasts, that while WhisperX overall tends to perform better, Google ASR outperforms in WIL and BLEU scores for non-scripted podcasts. We also find that Google ASRâs transcripts tend to contain closer to the ground truth number of edited-type disfluent nodes, while WhisperXâs transcripts are closer for interjection-type disfluent nodes. This same pattern is present in the larger set. Our findings have implications for the choice of an ASR model when building a larger system, as the choice should be made depending on the distribution of disfluent nodes present in the data.
Deep Prosodic Features in Tandem with Perceptual Judgments of Word Reduction for Tone Recognition in Conversed Speech
Authors: Xiang-Li Lu; Yi-Fen Liu
Venue: Interspeech 2024 (main)

Abstract: To tackle the tone classification problem in conversational speech, we propose a transformer-based encoding network to classify tones in an utterance on a syllable-by-syllable basis. Using just F0 and rhythmic information, the interaction encoder consolidates contour representations first. By jointly predicting word tones using perceived judgments on reduction degrees, the learning architecture improves automatic recognition of the underlying syllable tones. Leveraging these enhancements, the experiments show that the proposed model is very robust and achieved a 12% increase in tone classification accuracy.
SeMaScore: A new evaluation metric for automatic speech recognition tasks
Authors: Zitha Sasindran; Harsha Yelchuri; T. V. Prabhakar
Venue: Interspeech 2024 (main)

Abstract: In this study, we present SeMaScore, generated using a segment-wise mapping and scoring algorithm that serves as an evaluation metric for automatic speech recognition tasks. SeMaScore leverages both the error rate and a more robust similarity score. We show that our algorithmâs score generation improves upon the state-of-the-art BERTScore. Our experimental results show that SeMaScore corresponds well with expert human assessments, signal-to-noise ratio levels, and other natural language metrics. We outperform BERTScore by 41x in metric computation speed. Overall, we demonstrate that SeMaScore serves as a more dependable evaluation metric, particularly in real-world situations involving atypical speech patterns.
Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition
Authors: Jingjing Xu; Wei Zhou; Zijian Yang; Eugen Beck; Ralf SchlÃ¼ter
Venue: Interspeech 2024 (main)

Abstract: Varying-size models are often required to deploy ASR systems under different hardware and/or application constraints such as memory and latency. To avoid redundant training and optimization efforts for individual models of different sizes, we present the dynamic encoder size approach, which jointly trains multiple performant models within one supernet from scratch. These subnets of various sizes are layer-wise pruned from the supernet, and thus, enjoy full parameter sharing. By combining score-based pruning with supernet training, we propose two novel methods, Simple-Top-k and Iterative-Zero-Out, to automatically select the best-performing subnets in a data-driven manner, avoiding resource-intensive search efforts. Our experiments using CTC on both Librispeech and TED-LIUM-v2 corpora show that our methods can achieve on-par performance as individually trained models of each size category. Also, our approach consistently brings small performance improvements for the full-size supernet.
Revisiting Convolution-free Transformer for Speech Recognition
Authors: Zejiang Hou; Goeric Huybrechts; Anshu Bhatia; Daniel Garcia-Romero; Kyu J. Han; Katrin Kirchhoff
Venue: Interspeech 2024 (main)

Abstract: Convolution augmented Transformer architectures have dominated the field of automatic speech recognition by showing better WER results when the models are trained on relatively smaller training data. In this work, we revisit the necessity of convolution modules in the ASR encoder architecture, given that the inductive bias brought by the convolution modules may only boost performance in a low training data regime. We show that with architectural improvements to the Transformer block, a convolution-free Transformer architecture (namely, Transformer++) can catch up with the best Conformer WER results as we scale up the training data. Moreover, we demonstrate that with large scale unsupervised pre-training, the proposed Transformer++ can achieve even better WER than the best Conformer results. Importantly, Transformer++ achieves state-of- the-art performance with top efficiency, where we show 40% CPU inference realtime factor (RTF) improvement and 25% GPU training speedup compared to Conformer.
Optimizing Large-Scale Context Retrieval for End-to-End ASR
Authors: Zhiqi Huang; Diamantino Caseiro; Kandarp Joshi; Christopher Li; Pat Rondon; Zelin Wu; Petr Zadrazil; Lillian Zhou
Venue: Interspeech 2024 (main)

Abstract: Contextual Automatic Speech Recognition (ASR) requires scalable and accurate retrieval of content relevant to the userâs context. This paper presents a comparative study of two independent context retrieval methods: sequence and segment level scoring. Evaluated on datasets with up to 100k phrases, all methods exhibit excellent retrieval recall. Notably, the segment-level scoring achieves an outstanding 75.6% recall over 100k entities. When each method is further integrated with ASR through joint training, significant improvements over nonbiased ASR are observed, with WER reduction of up to 36% with 2k entities and 28% with 100k entities. This comparative analysis provides valuable insights for selecting the optimal context retrieval technique to achieve scalable and accurate performance in contextual ASR applications.