GenAI4ELab/papercli-papers huggingface dataset:
Language Model Personalization for Improved Touchscreen Typing
Authors: Jiban Adhikary; Keith Vertanen
Venue: Interspeech 2023 (main)

Abstract: Touchscreen keyboards rely on language modeling to auto-correct noisy typing and to offer word predictions. While language models can be pre-trained on huge amounts of text, they may fail to capture a user's unique writing style. Using a recently released email personalization dataset, we show improved performance compared to a unigram cache by adapting to a user's text via language models based on prediction by partial match (PPM) and recurrent neural networks. On simulated noisy touchscreen typing of 44 users, our best model increased keystroke savings by 9.9% relative and reduced word error rate by 36% relative compared to a static background language model.
Blank Collapse: Compressing CTC Emission for the Faster Decoding
Authors: Minkyu Jung; Ohhyeok Kwon; Seunghyun Seo; Soonshin Seo
Venue: Interspeech 2023 (main)

Abstract: Connectionist Temporal Classification (CTC) model is a very efficient method for modeling sequences, especially for speech data. In order to use the CTC model as an Automatic Speech Recognition (ASR) task, the beam search decoding with an external language model like n-gram LM is necessary to obtain reasonable results. In this paper, we analyze the blank label in CTC beam search deeply and propose a very simple method to reduce the amount of calculation resulting in faster beam search decoding speed. With this method, we can get up to 78% faster decoding speed than ordinary beam search decoding with a very small loss of accuracy in LibriSpeech datasets. We prove this method is effective not only practically by experiments but also theoretically by mathematical reasoning. We also observe that this reduction is more obvious if the accuracy of the model is higher.
Improving Joint Speech-Text Representations Without Alignment
Authors: Cal Peyser; Zhong Meng; Rohit Prabhavalkar; Andrew Rosenberg; Tara Sainath; Michael Picheny; Kyunghyun Cho; Ke Hu
Venue: Interspeech 2023 (main)

Abstract: The last year has seen astonishing progress in text-prompted image generation premised on the idea of a cross-modal representation space in which the text and image domains are represented jointly. In ASR, this idea has found application as joint speech-text encoders that can scale to the capacities of very large parameter models by being trained on both unpaired speech and text. While these methods show promise, they have required special treatment of the sequence-length mismatch inherent in speech and text, either by up-sampling heuristics or an explicit alignment model. In this work, we offer evidence that joint speech-text encoders naturally achieve consistent representations across modalities by disregarding sequence length, and argue that consistency losses could forgive length differences and simply assume the best alignment. We show that such a loss improves downstream WER in both a large-parameter monolingual and multilingual system.
Leveraging Cross-Utterance Context For ASR Decoding
Authors: Robert Flynn; Anton Ragni
Venue: Interspeech 2023 (main)

Abstract: While external language models (LMs) are often incorporated into the decoding stage of automated speech recognition systems, these models usually operate with limited context. Cross utterance information has been shown to be beneficial during second pass re-scoring, however this limits the hypothesis space based on the local information available to the first pass LM. In this work, we investigate the incorporation of long-context transformer LMs for cross-utterance decoding of acoustic models via beam search, and compare against results from n-best rescoring. Results demonstrate that beam search allows for an improved use of cross-utterance context. When evaluating on the long-format dataset AMI, results show a 0.7% and 0.3% absolute reduction on dev and test sets compared to the single-utterance setting, with improvements when including up to 500 tokens of prior context. Evaluations are also provided for Tedlium-1 with less significant improvements of around 0.1% absolute.
Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation
Authors: Minglun Han; Feilong Chen; Jing Shi; Shuang Xu; Bo Xu
Venue: Interspeech 2023 (main)

Abstract: Large-scale pre-trained language models (PLMs) have shown great potential in natural language processing tasks. Leveraging the capabilities of PLMs to enhance automatic speech recognition (ASR) systems has also emerged as a promising research direction. However, previous works may be limited by the inflexible structures of PLMs and the insufficient utilization of PLMs. To alleviate these problems, we propose the hierarchical knowledge distillation (HKD) on the continuous integrate-and-fire (CIF) based ASR models. To transfer knowledge from PLMs to the ASR models, HKD employs cross-modal knowledge distillation with contrastive loss at the acoustic level and knowledge distillation with regression loss at the linguistic level. Compared with the original CIF-based model, our method achieves 15% and 9% relative error rate reduction on the AISHELL-1 and LibriSpeech datasets, respectively.
Integration of Frame- and Label-synchronous Beam Search for Streaming Encoder-decoder Speech Recognition
Authors: Emiru Tsunoo; Hayato Futami; Yosuke Kashiwagi; Siddhant Arora; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: Although frame-based models, such as CTC and transducers, have an affinity for streaming automatic speech recognition, their decoding uses no future knowledge, which could lead to incorrect pruning. Conversely, label-based attention encoder--decoder mitigates this issue using soft attention to the input, while it tends to overestimate labels biased towards its training domain, unlike CTC. We exploit these complementary attributes and propose to integrate the frame- and label-synchronous (F-/L-Sync) decoding alternately performed within a single beam-search scheme. F-Sync decoding leads the decoding for block-wise processing, while L-Sync decoding provides the prioritized hypotheses using look-ahead future frames within a block. We maintain the hypotheses from both decoding methods to perform effective pruning. Experiments demonstrate that the proposed search algorithm achieves lower error rates compared to the other search methods, while being robust against out-of-domain situations.
A Neural Time Alignment Module for End-to-End Automatic Speech Recognition
Authors: Dongcheng Jiang; Chao Zhang; Philip C. Woodland
Venue: Interspeech 2023 (main)

Abstract: End-to-end trainable (E2E) automatic speech recognition (ASR) systems have low word error rates, but they do not model timings or silence by default unlike hidden Markov model (HMM)-based systems. In this paper, an extra neural aligner module is proposed for E2E ASR models, which labels the word timings in a post-processing stage. Pre-trained neural transducer and attention-based encoder-decoder models are adopted as the ASR backbones for experiments. The aligner module uses self-attention and cross-attention and takes the hidden representations from the backbone to predict the durations of each word and the possible silences. A novel loss is proposed for aligner training with the backbone frozen. Experimental results showed that when trained using the references from an existing HMM-based forced aligner, the proposed methods can make time predictions at accuracy about 95% for matched recognised words, and about 99% for utterances up to 10 s with reference text, with 200 ms tolerance.
Accelerating Transducers through Adjacent Token Merging
Authors: Yuang Li; Yu Wu; Jinyu Li; Shujie Liu
Venue: Interspeech 2023 (main)

Abstract: Recent end-to-end automatic speech recognition (ASR) systems often utilize a Transformer-based acoustic encoder that generates embedding at a high frame rate. However, this design is inefficient, particularly for long speech signals due to the quadratic computation of self-attention. To address this, we propose a new method, Adjacent Token Merging (A-ToMe), which gradually combines adjacent tokens with high similarity scores between their key values. In this way, the total time step could be reduced, and the inference of both the encoder and joint network is accelerated. Experiments on LibriSpeech show that our method can reduce 57% of tokens and improve the inference speed on GPU by 70% without any notable loss of accuracy. Additionally, we demonstrate that A-ToMe is also an effective solution to reduce tokens in long-form ASR, where the input speech consists of multiple utterances.
Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition
Authors: Siyuan Feng; Ming Tu; Rui Xia; Chuanzeng Huang; Yuxuan Wang
Venue: Interspeech 2023 (main)

Abstract: We improve low-resource ASR by integrating the ideas of multilingual training and self-supervised learning. Concretely, we leverage an International Phonetic Alphabet (IPA) multilingual model to create frame-level pseudo labels for unlabeled speech, and use these pseudo labels to guide hidden-unit BERT (HuBERT) based speech pretraining in a phonetically-informed manner. The experiments on the Multilingual Speech (MLS) Corpus show that the proposed approach consistently outperforms the standard HuBERT on all the target languages. Moreover, on 3 of the 4 languages, comparing to the standard HuBERT, the approach performs better, meanwhile is able to save supervised training data by 1.5k hours (75%) at most. Our approach outperforms most of the state of the arts, with much less pretraining data in terms of hours and language diversity. Compared to XLSR-53 and a retraining based multilingual method, our approach performs better with full and limited finetuning data scenarios.
Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition
Authors: Wenxuan Wang; Guodong Ma; Yuke Li; Binbin Du
Venue: Interspeech 2023 (main)

Abstract: Multilingual speech recognition for both monolingual and code-switching speech is a challenging task. Recently, based on the Mixture of Experts (MoE), many works have made good progress in multilingual and code-switching ASR, but present huge computational complexity with the increase of supported languages. In this work, we propose a computation-efficient network named Language-Routing Mixture of Experts (LR-MoE) for multilingual and code-switching ASR. LR-MoE extracts language-specific representations through the Mixture of Language Experts (MLE), which is guided to learn by a frame-wise language routing mechanism. The weight-shared frame-level language identification (LID) network is jointly trained as the shared pre-router of each MoE layer. Experiments show that the proposed method significantly improves multilingual and code-switching speech recognition performances over baseline with comparable computational efficiency.
Embedding Articulatory Constraints for Low-resource Speech Recognition Based on Large Pre-trained Model
Authors: Jaeyoung Lee; Masato Mimura; Tatsuya Kawahara
Venue: Interspeech 2023 (main)

Abstract: Knowledge about phonemes and their articulatory attributes can help improve automatic speech recognition (ASR) of low-resource languages. In this study, we propose a simple and effective approach to embed prior knowledge about phonemes into end-to-end ASR based on a large pre-trained model. An articulatory attribute prediction layer is constructed by embedding articulatory constraints in layer initialization, which allows for predicting articulatory attributes without the need for explicit training. The final ASR transcript is inferred by combining the output of this layer with encoded speech features. We apply our method to finetune a pre-trained XLS-R model using Ainu and Mboshi corpora, and achieve a 12% relative improvement when target data of only 1 hour is available. This demonstrates that the approach of incorporating phonetic prior knowledge is useful when combined with a large pre-trained model.
Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learning
Authors: Xuankai Chang; Brian Yan; Yuya Fujita; Takashi Maekaku; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning (SSL) of speech has shown impressive results in speech-related tasks, particularly in automatic speech recognition (ASR). While most methods employ the output of intermediate layers of the SSL model as real-valued features for downstream tasks, there is potential in exploring alternative approaches that use discretized token sequences. This approach offers benefits such as lower storage requirements and the ability to apply techniques from natural language processing. In this paper, we propose a new protocol that utilizes discretized token sequences in ASR tasks, which includes de-duplication and sub-word modeling to enhance the input sequence. It reduces computational cost by decreasing the length of the sequence. Our experiments on the LibriSpeech dataset demonstrate that our proposed protocol performs competitively with conventional ASR systems using continuous input features, while reducing computational and storage costs.
SpellMapper: A non-autoregressive neural spellchecker for ASR customization with candidate retrieval based on n-gram mappings
Authors: Alexandra Antonova; Evelina Bakhturina; Boris Ginsburg
Venue: Interspeech 2023 (main)

Abstract: Contextual spelling correction models are an alternative to shallow fusion to improve automatic speech recognition (ASR) quality given user vocabulary. To deal with large user vocabularies, most of these models include candidate retrieval mechanisms, usually based on minimum edit distance between fragments of ASR hypothesis and user phrases. However, the edit-distance approach is slow, non-trainable, and may have low recall as it relies only on common letters. We propose: 1) a novel algorithm for candidate retrieval, based on misspelled n-gram mappings, which gives up to 90% recall with just the top 10 candidates on Spoken Wikipedia; 2) a non-autoregressive neural model based on BERT architecture, where the initial transcript and ten candidates are combined into one input. The experiments on Spoken Wikipedia show 21.4% word error rate improvement compared to a baseline ASR system.
Text Injection for Capitalization and Turn-Taking Prediction in Speech Models
Authors: Shaan Bijwadia; Shuo-Yiin Chang; Weiran Wang; Zhong Meng; Hao Zhang
Venue: Interspeech 2023 (main)

Abstract: Text injection for automatic speech recognition (ASR), wherein unpaired text-only data is used to supplement paired audio-text data, has shown promising improvements for word error rate. This study examines the use of text injection for auxiliary tasks, which are the non-ASR tasks often performed by an E2E model. In this work, we use joint end-to-end and internal language model training (JEIT) as our text injection algorithm to train an ASR model which performs two auxiliary tasks. The first is capitalization, which is a de-normalization task. The second is turn-taking prediction, which attempts to identify whether a user has completed their conversation turn in a digital assistant interaction. We show results demonstrating that our text injection method boosts capitalization performance for long-tail data, and improves turn-taking detection recall.
Confidence-based Ensembles of End-to-End Speech Recognition Models
Authors: Igor Gitman; Vitaly Lavrukhin; Aleksandr Laptev; Boris Ginsburg
Venue: Interspeech 2023 (main)

Abstract: The number of end-to-end speech recognition models grows every year. These models are often adapted to new domains or languages resulting in a proliferation of expert systems that achieve great results on target data, while generally showing inferior performance outside of their domain of expertise. We explore combination of such experts via confidence-based ensembles: ensembles of models where only the output of the most-confident model is used. We assume that models' target data is not available except for a small validation set. We demonstrate effectiveness of our approach with two applications. First, we show that a confidence-based ensemble of 5 monolingual models outperforms a system where model selection is performed via a dedicated language identification block. Second, we demonstrate that it is possible to combine base and adapted models to achieve strong results on both original and target data. We validate all our results on multiple datasets and model architectures.
Unsupervised Code-switched Text Generation from Parallel Text
Authors: Jie Chi; Brian Lu; Jason Eisner; Peter Bell; Preethi Jyothi; Ahmed M. Ali
Venue: Interspeech 2023 (main)

Abstract: There has been great interest in developing automatic speech recognition (ASR) systems that can handle code-switched (CS) speech to meet the needs of a growing bilingual population. However, existing datasets are limited in size. It is expensive and difficult to collect real transcribed spoken CS data due to the challenges of finding and identifying CS data in the wild. As a result, many attempts have been made to generate synthetic CS data. Existing methods either require the existence of CS data during training, or are driven by linguistic knowledge. We introduce a novel approach of forcing a multilingual MT system that was trained on non-CS data to generate CS translations. Comparing against two prior methods, we show that simply leveraging the shared representations of two languages (Mandarin and English) yields better CS text generation and, ultimately, better CS ASR.
A Binary Keyword Spotting System with Error-Diffusion Based Feature Binarization
Authors: Dingyi Wang; Mengjie Luo; Lin Li; Xiaoqin Wang; Shushan Qiao; Yumei Zhou
Venue: Interspeech 2023 (main)

Abstract: Binary-neural-network based keyword spotting (KWS) for resource-constrained devices has gained much attention in recent years. Although several works proved their success, a fully binary KWS system is yet to come, considering high-precision speech feature maps are still required for satisfying accuracy. Such precision mismatch results in non-binary activation layers, thus leading to extra computational costs. In this paper, we present an extremely compact KWS system using a binary neural network and error-diffusion binarized speech features. The system eliminates all high-precision multiplications and requires only hardware-friendly bit-wise operations and additions for inference. Experiments on the Google speech commands show that our binary KWS system yields 98.54% accuracy on a 1-keyword task and 95.05% on a 2-keyword task, outperforming 8-bit KWS systems of bigger size. The result proves the feasibility of a fully binary KWS system and can be inspiring for hardware implementations.
Language-universal Phonetic Encoder for Low-resource Speech Recognition
Authors: Siyuan Feng; Ming Tu; Rui Xia; Chuanzeng Huang; Yuxuan Wang
Venue: Interspeech 2023 (main)

Abstract: Multilingual training is effective in improving low-resource ASR, which may partially be explained by phonetic representation sharing between languages. In end-to-end (E2E) ASR systems, graphemes are often used as basic modeling units, however graphemes may not be ideal for multilingual phonetic sharing. In this paper, we leverage International Phonetic Alphabet (IPA) based language-universal phonetic model to improve low-resource ASR performances, for the first time within the attention encoder-decoder architecture. We propose an adaptation method on the phonetic IPA model to further improve the proposed approach on extreme low-resource languages. Experiments carried out on the open-source MLS corpus and our internal databases show  our approach outperforms baseline monolingual models and most  state-of-the-art works. Our main approach and adaptation are effective on extremely low-resource languages, even within domain- and language-mismatched scenarios.
A Lexical-aware Non-autoregressive Transformer-based ASR Model
Authors: Chong-En Lin; Kuan-Yu Chen
Venue: Interspeech 2023 (main)

Abstract: Non-autoregressive automatic speech recognition (ASR) has become a mainstream of ASR modeling because of its fast decoding speed and satisfactory result. To further boost the performance, relaxing the conditional independence assumption and cascading large-scaled pre-trained models are two active research directions. In addition to these strategies, we propose a lexical-aware non-autoregressive Transformer-based (LA-NAT) ASR framework, which consists of an acoustic encoder, a speech-text shared encoder, and a speech-text shared decoder. The acoustic encoder is used to process the input speech features as usual, and the speech-text shared encoder and decoder are designed to train speech and text data simultaneously. By doing so, LA-NAT aims to make the ASR model aware of lexical information, so the resulting model is expected to achieve better results by leveraging the learned linguistic knowledge.
Improving Under-Resourced Code-Switched Speech Recognition: Large Pre-trained Models or Architectural Interventions
Authors: Joshua Jansen van VÃ¼ren; Thomas Niesler
Venue: Interspeech 2023 (main)

Abstract: We present three approaches to improve language modelling of under-resourced code-switched speech. First, we challenge the practice of fine-tuning large pre-trained language models on small datasets. Secondly, we investigate the advantages of sub-word encodings for our multilingual code-switched speech. Thirdly, we propose an architectural innovation to the RNN language model that is specifically designed for code-switched text. We show a clear reduction in absolute word error rate of 0.17% for the adapted LSTM language model compared to M-BERT when employed in n-best rescoring experiments. Further, the LSTM models afford a seven-fold reduction in total number of parameters and reduces runtime during rescoring 100-fold. Contrary to recent research trends, our LSTM models do not outperform the word-level models when using sub-word vocabularies. Finally, the new architectural mechanism applied to the LSTM improves language prediction for a span of several words following a code-switch.
Pragmatic Pertinence: A Learnable Confidence Metric to Assess the Subjective Quality of LM-Generated Text
Authors: Jerome R. Bellegarda
Venue: Interspeech 2023 (main)

Abstract: To be perceived as trustworthy, artificially generated text must be sufficiently congruent with the available discourse history. Pre-trained language models (LMs) operating in generative mode are capable of predicting locally coherent phrases, but those do not always reflect salient syntactic, semantic, or pragmatic facets of prior content. This paper introduces a learnable evaluation metric to assess the pragmatic pertinence of LM-generated text for a given history. Pertinence is closely aligned with qualitative human judgments of acceptability, thereby emerging as a blend of sensibleness and specificity. Experiments conducted across different domains using different learning architectures show that this approach circumvents the issue of multiple valid ground-truths, while providing a reliable quantitative ranking of generated text completion candidates in context. Pertinence scoring could thus prove useful for the detection of hallucinations.