GenAI4ELab/papercli-papers huggingface dataset:
N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space
Authors: Rao Ma; Mark J. F. Gales; Kate M. Knill; Mengjie Qian
Venue: Interspeech 2023 (main)

Abstract: Error correction models form an important part of Automatic Speech Recognition (ASR) post-processing to improve the readability and quality of transcriptions. Most prior works use the 1-best ASR hypothesis as input and therefore can only perform correction by leveraging the context within one sentence. In this work, we propose a novel N-best T5 model for this task, which is fine-tuned from a T5 model and utilizes ASR N-best lists as model input. By transferring knowledge from the pre-trained language model and obtaining richer information from the ASR decoding space, the proposed approach outperforms a strong Conformer-Transducer baseline. Another issue with standard error correction is that the generation process is not well-guided. To address this a constrained decoding process, either based on the N-best list or an ASR lattice, is used which allows additional information to be propagated.
SememeASR: Boosting Performance of End-to-End Speech Recognition against Domain and Long-Tailed Data Shift with Sememe Semantic Knowledge
Authors: Jiaxu Zhu; Changhe Song; Zhiyong Wu; Helen Meng
Venue: Interspeech 2023 (main)

Abstract: Recently, excellent progress has been made in speech recognition. However, pure data-driven approaches have struggled to solve the problem in domain-mismatch and long-tailed data. Considering that knowledge-driven approaches can help data-driven approaches alleviate their flaws, we introduce sememe-based semantic knowledge information to speech recognition (SememeASR). Sememe, according to the linguistic definition, is the minimum semantic unit in a language and is able to represent the implicit semantic information behind each word very well. Our experiments show that the introduction of sememe information can improve the effectiveness of speech recognition. In addition, our further experiments show that sememe knowledge can improve the model's recognition of long-tailed data and enhance the model's domain generalization ability.
miniStreamer: Enhancing Small Conformer with Chunked-Context Masking for Streaming ASR Applications on the Edge
Authors: Haris Gulzar; Monikka Roslianna Busto; Takeharu Eda; Katsutoshi Itoyama; Kazuhiro Nakadai
Venue: Interspeech 2023 (main)

Abstract: Real-time applications of Automatic Speech Recognition (ASR) on user devices on the edge require streaming processing. Conformer model has achieved state-of-the-art performance in ASR for the non-streaming task. Conventional approaches have tried to achieve streaming ASR with Conformer using causal operations, but it leads to quadratic increase in the computational cost as the utterance length increases. In this work, we propose a chunked-context masking approach to perform streaming ASR with Conformer, which limits the computational cost from quadratic to a constant value. Our approach allows self-attention in Conformer encoder to attend the limited past information in form of chunked context. It achieves close to the full context causal performance for Conformer-Transducer, while significantly reducing the computational cost and maintains a low Real Time Factor (RTF) which is highly desirable trait for resource-constrained low-power edge devices.
CoMFLP: Correlation Measure Based Fast Search on ASR Layer Pruning
Authors: Wei Liu; Zhiyuan Peng; Tan Lee
Venue: Interspeech 2023 (main)

Abstract: Transformer-based speech recognition (ASR) model with deep layers exhibited significant performance improvement. However, the model is inefficient for deployment on resource constrained devices. Layer pruning (LP) is a commonly used compression method to remove redundant layers. Previous studies on LP usually identify the redundant layers according to a task-specific evaluation metric. They are time-consuming for models with a large number of layers, even in a greedy search manner. To address this problem, we propose CoM-FLP, a fast search LP algorithm based on correlation measure. The correlation between layers is computed to generate a correlation matrix, which identifies the redundancy among layers. The search process is carried out in two steps: (1) coarse search: to determine top K candidates by pruning the most redundant layers based on the correlation matrix; (2) fine search: to select the best pruning proposal among K candidates using a task-specific evaluation metric. Experiments on an ASR task show that the pruning proposal determined by CoMFLP outperforms existing LP methods while only requiring constant time complexity. The code is publicly available at https://github.com/louislau1129/CoMFLP.
Exploration on HuBERT with Multiple Resolution
Authors: Jiatong Shi; Yun Tang; Hirofumi Inaguma; Hongyu Gong; Juan Pino; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: Hidden-unit BERT (HuBERT) is a widely-used self-supervised learning (SSL) model in speech processing. However, we argue that its fixed 20ms resolution for hidden representations would not be optimal for various speech-processing tasks since their attributes (e.g., speaker characteristics and semantics) are based on different time scales. To address this limitation, we propose utilizing HuBERT representations at multiple resolutions for downstream tasks. We explore two approaches, namely the parallel and hierarchical approaches, for integrating HuBERT features with different resolutions. Through experiments, we demonstrate that HuBERT with multiple resolutions outperforms the original model. This highlights the potential of utilizing multiple resolutions in SSL models like HuBERT to capture diverse information from speech signals.
Quantization-aware and Tensor-compressed Training of Transformers for Natural Language Understanding
Authors: Zi Yang; Samridhi Choudhary; Siegfried Kunzmann; Zheng Zhang
Venue: Interspeech 2023 (main)

Abstract: Fine-tuned transformer models have shown superior performances in many natural language tasks. However, the large model size prohibits deploying high-performance transformer models on resource-constrained devices. This paper proposes a quantization-aware tensor-compressed training approach to reduce the model size, arithmetic operations, and runtime latency of transformer-based models. We compress the embedding and linear layers of transformers into small low-rank tensor cores, significantly reducing model parameters. A quantization-aware training with learnable scales factors is used to further obtain low-precision representations of the tensor-compressed models. The developed approach can be used for both end-to-end training and distillation-based training. To improve the convergence, layer-by-layer distillation is applied to distill a quantized tensor-compressed student model from a pre-trained transformer. The performance is demonstrated in two natural language understanding tasks, showing up to 63 times compression ratio with little accuracy loss and remarkable inference and training speedup.
Word-level Confidence Estimation for CTC Models
Authors: Burin Naowarat; Thananchai Kongthaworn; Ekapol Chuangsuwanich
Venue: Interspeech 2023 (main)

Abstract: Measuring confidence in Automatic Speech Recognition (ASR) is important for ensuring the reliability of downstream applications. Previous works proposed Confidence Estimation Module (CEM) for predicting confidences for autoregressive attention-based and neural transducer architectures. However, CEM for connectionist temporal classification (CTC) models have not been explored. In this work, we expand the idea of CEM to CTC models and further propose considering surrounding words for estimating confidences. Our experiments on four test sets in two languages demonstrate that our proposed method significantly reduces calibration errors of both common and rare words compared to naive confidences from CTC softmax. Moreover, we show that the approach is also effective for hard words and out-of-domain test sets, indicating its potential to be used as a reliable trigger for human intervention.
Multilingual Contextual Adapters To Improve Custom Word Recognition In Low-resource Languages
Authors: Devang Kulshreshtha; Saket Dingliwal; Brady Houston; Sravan Bodapati
Venue: Interspeech 2023 (main)

Abstract: Connectionist Temporal Classification (CTC) models are popular for their balance between speed and performance for Automatic Speech Recognition (ASR). However, these CTC models still struggle in other areas, such as personalization towards custom words. A recent approach explores Contextual Adapters, wherein an attention-based biasing model for CTC is used to improve the recognition of custom entities. While this approach works well with enough data, we showcase that it isn't an effective strategy for low-resource languages. In this work, we propose a supervision loss for smoother training of the Contextual Adapters. Further, we explore a multilingual strategy to improve performance with limited training data. Our method achieves 48% F1 improvement in retrieving unseen custom entities for a low-resource language. Interestingly, as a by-product of training the Contextual Adapters, we see a 5-11% Word Error Rate (WER) reduction in the performance of the base CTC model as well.
Unsupervised Active Learning: Optimizing Labeling Cost-Effectiveness for Automatic Speech Recognition
Authors: Zhisheng Zheng; Ziyang Ma; Yu Wang; Xie Chen
Venue: Interspeech 2023 (main)

Abstract: In recent years, speech-based self-supervised learning (SSL) has made significant progress in various tasks, including automatic speech recognition (ASR). An ASR model with decent performance can be realized by fine-tuning an SSL model with a small fraction of labeled data. Reducing the demand for labeled data is always of great practical value. In this paper, we further extend the use of SSL to cut down labeling costs with active learning. Three types of units on different granularities are derived from speech signals in an unsupervised way, and their effects are compared by applying a contrastive data selection method. The experimental results show that our proposed data selection framework can effectively improve the word error rate (WER) by more than 11% with the same amount of labeled data, or halve the labeling cost while maintaining the same WER, compared to random selection.
4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders
Authors: Yui Sudo; Shakeel Muhammad; Brian Yan; Jiatong Shi; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: End-to-end (E2E) automatic speech recognition (ASR) can be classified into several models, including connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), attention mechanism, and mask-predict models. There are pros and cons to each of these architectures, and thus practitioners may switch between these different models depending on application requirements. Instead of building separate models, we propose a joint modeling scheme where four different decoders (CTC, attention, RNN-T, mask-predict) share an encoder - we refer to this as 4D modeling. Additionally, we propose to 1) train 4D models using a two-stage strategy which stabilizes multitask learning and 2) decode 4D models using a novel time-synchronous one-pass beam search. We demonstrate that jointly trained 4D models improve the performances of each individual decoder. Further, we show that our joint CTC/RNN-T/attention decoding surpasses the previously proposed CTC/attention decoding.
Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition
Authors: Hao Yen; Pin-Jui Ku; Chao-Han Huck Yang; Hu Hu; Sabato Marco Siniscalchi; Pin-Yu Chen; Yu Tsao
Venue: Interspeech 2023 (main)

Abstract: We propose a novel adversarial reprogramming (AR) approach for low-resource spoken command recognition (SCR) and build an AR-SCR system. The AR procedure aims at repurposing a pretrained SCR model (from the source domain) to modify the acoustic signals (from the target domain). To solve the label mismatches between source and target domains and further improve the stability of AR, we propose a novel similarity-based label mapping technique to align classes. In addition, the transfer learning (TL) technique is combined with the original AR process to improve the model adaptation capability. We evaluate the proposed AR-SCR system on three low-resource SCR datasets, including Arabic, Lithuanian, and dysarthric Mandarin speech. Experimental results show that with a pretrained acoustic model trained on a large-scale English dataset, the proposed AR-SCR system outperforms the current state-of-the-art results on Lithuanian and Arabic datasets, with only a limited amount of training data.
Language-specific Boundary Learning for Improving Mandarin-English Code-switching Speech Recognition
Authors: Zhiyun Fan; Linhao Dong; Chen Shen; Zhenlin Liang; Jun Zhang; Lu Lu; Zejun Ma
Venue: Interspeech 2023 (main)

Abstract: Code-switching speech recognition (CSSR) transcribes speech that switches between multiple languages or dialects within a single sentence. The main challenge in this task is that different languages often have similar pronunciations, making it difficult for models to distinguish between them. In this paper, we propose a method for solving the CSSR task from the perspective of language-specific acoustic boundary learning. We introduce language-specific weight estimators (LSWE) to model acoustic boundary learning in different languages separately. Additionally, a non-autoregressive (NAR) decoder and a language change detection (LCD) module are employed to assist in training. Evaluated on the SEAME corpus, our method achieves a state-of-the-art mixed error rate (MER) of 16.29% and 22.81% on the testman and testsge sets. We also demonstrate the effectiveness of our method on a 9000-hour in-house meeting code-switching dataset, where our method achieves a relatively 7.9% MER reduction.
Mixture-of-Expert Conformer for Streaming Multilingual ASR
Authors: Ke Hu; Bo Li; Tara Sainath; Yu Zhang; FranÃ§oise Beaufays
Venue: Interspeech 2023 (main)

Abstract: End-to-end models with large capacity have significantly improved multilingual automatic speech recognition, but their computation cost poses challenges for on-device applications. We propose a streaming truly multilingual Conformer incorporating mixture-of-expert (MoE) layers that learn to only activate a subset of parameters in training and inference. The MoE layer consists of a softmax gate which chooses the best two experts among many in forward propagation. The proposed MoE layer offers efficient inference by activating a fixed number of parameters as the number of experts increases. We evaluate the proposed model on a set of 12 languages, and achieve an average 11.9% relative improvement in WER over the baseline. Compared to an adapter model using ground truth information, our MoE model achieves similar WER and activates similar number of parameters but without any language information. We further show around 3% relative WER improvement by multilingual shallow fusion.
Lossless 4-bit Quantization of Architecture Compressed Conformer ASR Systems on the 300-hr Switchboard Corpus
Authors: Zhaoqing Li; Tianzi Wang; Jiajun Deng; Junhao Xu; Shoukang Hu; Xunying Liu
Venue: Interspeech 2023 (main)

Abstract: State-of-the-art end-to-end automatic speech recognition (ASR) systems are becoming increasingly complex and expensive for practical applications. This paper develops a high-performance and low-footprint 4-bit quantized Conformer ASR system. A key feature of the system design is to account for the fine-grained, varying performance sensitivity at different Conformer components to quantization errors. Neural architectural compression and mixed precision quantization approaches were used to auto-configure the optimal substructures and quantization bit-widths within each Conformer submodule. Experiments conducted on the 300-hr Switchboard data suggest that the obtained auto-configured systems consistently outperform the uniform precision quantized baseline Conformer of comparable bit-widths in terms of word error rate (WER). An overall "lossless" compression ratio of 16.2 times was obtained over the 32-bit full-precision baseline while incurring no statistically significant WER increase.
Compressed MoE ASR Model Based on Knowledge Distillation and Quantization
Authors: Yuping Yuan; Zhao You; Shulin Feng; Dan Su; Yanchun Liang; Xiaohu Shi; Dong Yu
Venue: Interspeech 2023 (main)

Abstract: The mixture of experts (MoE)-based automatic speech recognition (ASR) model can achieve remarkable performance, but pose greater challenges to model deployment for its huge model size. Therefore, it is important to compress the model size and reduce the computational cost. In this paper, we propose a compressed MoE (CMoE) ASR model that simplifies the MoE structure by knowledge distillation and reduces parameter bit-width through quantization, and provide two pipelines (one-stage and two-stage pipelines) to deploy the compression. In quantization, we use binary weight network to quantize the weights to 1-bit for reducing the quantization error and use learned step size quantization to quantize the activations to 4-bit. Experimental results show that the quantized dense network compressed from the MoE based ASR model by our method reduces the size by 150x with very small accuracy loss. The proposed model is expected to be deployed on embedded devices.
Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems
Authors: Jiajun Deng; Guinan Li; Xurong Xie; Zengrui Jin; Mingyu Cui; Tianzi Wang; Shujie Hu; Mengzhe Geng; Xunying Liu
Venue: Interspeech 2023 (main)

Abstract: Rich sources of variability in natural speech present significant challenges to current data intensive speech recognition technologies. To model both speaker and environment level diversity, this paper proposes a novel Bayesian factorised speaker-environment adaptive training and test time adaptation approach for Conformer ASR models. Speaker and environment level characteristics are separately modeled using compact hidden output transforms, which are then linearly or hierarchically combined to represent any speaker-environment combination. Bayesian learning is further utilized to model the adaptation parameter uncertainty. Experiments on the 300-hr WHAM noise corrupted Switchboard data suggest that factorised adaptation consistently outperforms the baseline and speaker label only adapted Conformers by up to 3.1% absolute (10.4% relative) word error rate reductions. Further analysis shows the proposed method offers potential for rapid adaption to unseen speaker-environment conditions.
Text Only Domain Adaptation with Phoneme Guided Data Splicing for End-to-End Speech Recognition
Authors: Wei Wang; Xun Gong; Hang Shao; Dongning Yang; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: Adaptation of end-to-end (E2E) automatic speech recognition (ASR) models to unseen domains remains a challenge due to their monolithic construction, which typically necessitates paired data for customization. While neural text-to-speech (TTS) approaches have shown effectiveness for domain adaptation, they come with the drawback of high computational costs during training and inference. In this paper, we propose a model-free audio synthesis pipeline for domain adaptation, which synthesizes audio with text from the target domain and audio pieces from the source domain, allowing ASR models to be adapted with the on-the-fly synthesized audio. Additionally, we apply layer-wise regularization between speech encodings generated by adapted and unadapted models to prevent overfitting. Our experiments adapt from LIBRI SPEECH to various domains in GIGA SPEECH. The results show a 15-30% relative improvement in target domains compared to shallow fusion, with almost no degradation in the source domain.
Cross-Lingual Cross-Age Adaptation for Low-Resource Elderly Speech Emotion Recognition
Authors: Samuel Cahyawijaya; Holy Lovenia; Willy Chung; Rita Frieske; Zihan Liu; Pascale Fung
Venue: Interspeech 2023 (main)

Abstract: Speech emotion recognition plays a crucial role in human-computer interactions. However, most speech emotion recognition research is biased toward English-speaking adults, which hinders its applicability to other demographic groups in different languages and age groups. In this work, we analyze the transferability of emotion recognition across three different languages--English, Mandarin Chinese, and Cantonese; and 2 different age groups--adults and the elderly. To conduct the experiment, we develop an English-Mandarin speech emotion benchmark for adults and the elderly, BiMotion, and a Cantonese speech emotion dataset, YueMotion. This study concludes that different language and age groups require specific speech features, thus making cross-lingual inference an unsuitable method. However, cross-group data augmentation is still beneficial to regularize the model, with linguistic distance being a significant influence on cross-lingual transferability.
Modular Domain Adaptation for Conformer-Based Streaming ASR
Authors: Qiujia Li; Bo Li; Dongseong Hwang; Tara Sainath; Pedro M. Mengibar
Venue: Interspeech 2023 (main)

Abstract: Speech data from different domains has distinct acoustic and linguistic characteristics. It is common to train a single multidomain model such as a Conformer transducer for speech recognition on a mixture of data from all domains. However, changing data in one domain or adding a new domain would require the multidomain model to be retrained. To this end, we propose a framework called modular domain adaptation (MDA) that enables a single model to process multidomain data while keeping all parameters domain-specific, i.e., each parameter is only trained by data from one domain. On a streaming Conformer transducer trained only on video caption data, experimental results show that an MDA-based model can reach similar performance as the multidomain model on other domains such as voice search and dictation by adding per-domain adapters and per-domain feed-forward networks in the Conformer encoder.
Donât Stop Self-Supervision: Accent Adaptation of Speech Representations via Residual Adapters
Authors: Anshu Bhatia; Sanchit Sinha; Saket Dingliwal; Karthik Gopalakrishnan; Sravan Bodapati; Katrin Kirchhoff
Venue: Interspeech 2023 (main)

Abstract: Speech representations learned in a self-supervised fashion from massive unlabeled speech corpora have been adapted successfully toward several downstream tasks. However, such representations may be skewed toward canonical data characteristics of such corpora and perform poorly on atypical, non-native accented speaker populations. With the state-of-the-art HuBERT model as a baseline, we propose and investigate self-supervised adaptation of speech representations to such populations in a parameter-efficient way via training accent-specific residual adapters. We experiment with 4 accents and choose automatic speech recognition (ASR) as the downstream task of interest. We obtain strong word error rate reductions (WERR) over HuBERT-large for all 4 accents, with a mean WERR of 22.7% with accent-specific adapters and a mean WERR of 25.1% if the entire encoder is accent-adapted. While our experiments utilize HuBERT and ASR as the downstream task, our proposed approach is both model and task-agnostic.