GenAI4ELab/papercli-papers huggingface dataset:
Personalization for BERT-based Discriminative Speech Recognition Rescoring
Authors: Jari Kolehmainen; Yile Gu; Aditya Gourav; Prashanth Gurunath Shivakumar; Ankur Gandhe; Ariya Rastrow; Ivan Bulyko
Venue: Interspeech 2023 (main)

Abstract: Recognition of personalized content remains a challenge in end-to-end speech recognition. We explore three novel approaches that use personalized content in a neural rescoring step to improve recognition: gazetteers, prompting, and a cross-attention based encoder-decoder model. We use internal de-identified en-US data from interactions with a virtual voice assistant supplemented with personalized named entities to compare these approaches. On a test set with personalized named entities, we show that each of these approaches improves word error rate by over 10%, against a neural rescoring baseline. We also show that on this test set, natural language prompts can improve word error rate by 7% without any training and with a marginal loss in generalization. Overall, gazetteers were found to perform the best with a 10% improvement in word error rate (WER), while also improving WER on a general test set by 1%.
On the N-gram Approximation of Pre-trained Language Models
Authors: Aravind Krishnan; Jesujoba O. Alabi; Dietrich Klakow
Venue: Interspeech 2023 (main)

Abstract: Large pre-trained language models (PLMs) have shown remarkable performance across various natural language understanding (NLU) tasks, particularly in low-resource settings. Nevertheless, their potential in Automatic Speech Recognition (ASR) remains largely unexplored. This study investigates the potential usage of PLMs for language modelling in ASR. We compare the application of large-scale text sampling and probability conversion for approximating GPT-2 into an n-gram model. Furthermore, we introduce a vocabulary-restricted decoding method for random sampling, and evaluate the effects of domain difficulty and data size on the usability of generated text. Our findings across eight domain-specific corpora support the use of sampling-based approximation and show that interpolating with a large sampled corpus improves test perplexity over a baseline trigram by 15%. Our vocabulary-restricted decoding method pushes this improvement further by 5% in domain-specific settings.
Record Deduplication for Entity Distribution Modeling in ASR Transcripts
Authors: Tianyu Huang; Chung Hoon Hong; Carl Wivagg; Kanna Shimizu
Venue: Interspeech 2023 (main)

Abstract: Voice digital assistants must keep up with trending search queries. We rely on a speech recognition model using contextual biasing with a rapidly updated set of entities, instead of frequent model retraining, to keep up with trends. There are several challenges with this approach: (1) the entity set must be frequently reconstructed, (2) the entity set is of limited size due to latency and accuracy trade-offs, and (3) finding the true entity distribution for biasing is complicated by ASR misrecognition. We address these challenges and define an entity set by modeling customers' true requested entity distribution from ASR output in production using record deduplication, a technique from the field of entity resolution. Record deduplication resolves or deduplicates coreferences, including misrecognitions, of the same latent entity. Our method successfully retrieves 95% of misrecognized entities and when used for contextual biasing shows an estimated 5% relative word error rate reduction.
Learning When to Trust Which Teacher for Weakly Supervised ASR
Authors: Aakriti Agrawal; Milind Rao; Anit Kumar Sahu; Gopinath Chennupati; Andreas Stolcke
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) training can utilize multiple experts as teacher models, each trained on a specific domain or accent. Teacher models may be opaque in nature since their architecture may be not be known or their training cadence is different from that of the student ASR model. Still, the student models are updated incrementally using the pseudo-labels generated independently by the expert teachers. In this paper, we exploit supervision from multiple domain experts in training student ASR models. This training strategy is especially useful in scenarios where few or no human transcriptions are available. To that end, we propose a Smart-Weighter mechanism that selects an appropriate expert based on the input audio, and then trains the student model in an unsupervised setting. We show the efficacy of our approach using LibriSpeech and LibriLight benchmarks and find an improvement of 4 to 25% over baselines that uniformly weight all the experts, use a single expert model, or combine experts using ROVER.
Text-only Domain Adaptation using Unified Speech-Text Representation in Transducer
Authors: Lu Huang; Boyu Li; Jun Zhang; Lu Lu; Zejun Ma
Venue: Interspeech 2023 (main)

Abstract: Domain adaptation using text-only corpus is challenging in end-to-end(E2E) speech recognition. Adaptation by synthesizing audio from text through TTS is resource-consuming. We present a method to learn Unified Speech-Text Representation in Conformer Transducer(USTR-CT) to enable fast domain adaptation using the text-only corpus. Different from the previous textogram method, an extra text encoder is introduced in our work to learn text representation and is removed during inference, so there is no modification for online deployment. To improve the efficiency of adaptation, single-step and multistep adaptations are also explored. The experiments on adapting LibriSpeech to SPGISpeech show the proposed method reduces the word error rate(WER) by relatively 44% on the target domain, which is better than those of TTS method and textogram method. Also, it is shown the proposed method can be combined with internal language model estimation(ILME) to further improve the performance.
Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model
Authors: Puyuan Peng; Shang-Wen Li; Okko RÃ¤sÃ¤nen; Abdelrahman Mohamed; David Harwath
Venue: Interspeech 2023 (main)

Abstract: In this paper, we show that representations capturing syllabic units emerge when training a self-supervised speech model with a visually-grounded training objective. We demonstrate that a nearly identical model architecture (HuBErT) trained with a masked language modeling loss does not exhibit this same ability, suggesting that the visual grounding objective is responsible for the emergence of this phenomenon. We propose the use of a minimum cut algorithm to automatically predict syllable boundaries in speech, followed by a 2-stage clustering method to group identical syllables together. We show that our model not only outperforms a state-of-the-art syllabic segmentation method on the language it was trained on (English), but also generalizes in a zero-shot fashion to Estonian. Finally, we show that the same model is capable of zero-shot generalization for a word segmentation task on 4 other languages from the Zerospeech Challenge, in some cases beating the previous state-of-the-art.
Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization
Authors: Puyuan Peng; Brian Yan; Shinji Watanabe; David Harwath
Venue: Interspeech 2023 (main)

Abstract: We investigate the emergent abilities of the recently proposed web-scale speech model Whisper, by adapting it to unseen tasks with prompt engineering. We selected three tasks: audio-visual speech recognition (AVSR), code-switched speech recognition (CS-ASR), and speech translation (ST) on unseen language pairs. we design task-specific prompts, by either leveraging another large-scale model, or simply manipulating the special tokens in the default prompts. Experiments show that compared to the default prompts, our proposed prompts improve performance by 10% to 45% on the three zero-shot tasks, and even outperform SotA supervised models on some datasets. In addition, our experiments reveal many interesting properties of Whisper, including its robustness to prompts, bias on accents, and the multilingual understanding in its latent space. Code is available here.
Progress and Prospects for Spoken Language Technology: Results from Five Sexennial Surveys
Authors: Roger K. Moore; Ricard Marxer
Venue: Interspeech 2023 (main)

Abstract: Every six years (since 1997), a survey has been conducted at the IEEE workshop on Automatic Speech Recognition and Understanding (ASRU). The aim has been to gain an insight into the research community's perspective on the 'progress and prospects' for spoken language technology. These surveys have been based on a set of 'statements' describing possible scenarios, and respondents are asked to estimate the year (in the future or in the past) when each given scenario might be realised. A number of the statements have appeared in multiple surveys, hence it has been possible to track changes in opinion over time. This paper presents the combined results from five such surveys, the most recent of which was conducted at ASRU-2021. The latest results reveal a dramatic increase in optimism.
Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Pooling
Authors: Ramon Sanabria; OndÅej Klejch; Hao Tang; Sharon Goldwater
Venue: Interspeech 2023 (main)

Abstract: Acoustic word embeddings are typically created by training a pooling function using pairs of word-like units. For unsupervised systems, these are mined using k-nearest neighbor (KNN) search, which is slow. Recently, mean-pooled representations from a pre-trained self-supervised English model were suggested as a promising alternative, but their performance on target languages was not fully competitive. Here, we explore improvements to both approaches: we use continued pre-training to adapt the self-supervised model to the target language, and we use a multilingual phone recognizer (MPR) to mine phone n-gram pairs for training the pooling function. Evaluating on four languages, we show that both methods outperform a recent approach on word discrimination. Moreover, the MPR method is orders of magnitude faster than KNN, and is highly data efficient. We also show a small improvement from performing learned pooling on top of the continued pre-trained representations.
CASA-ASR: Context-Aware Speaker-Attributed ASR
Authors: Mohan Shi; Zhihao Du; Qian Chen; Fan Yu; Yangze Li; Shiliang Zhang; Jie Zhang; Li-Rong Dai
Venue: Interspeech 2023 (main)

Abstract: Recently, speaker-attributed automatic speech recognition (SAASR) has attracted a wide attention, which aims at answering the question "who spoke what". Different from modular systems, end-to-end (E2E) SA-ASR minimizes the speaker-dependent recognition errors directly and shows a promising applicability. In this paper, we propose a context-aware SAASR (CASA-ASR) model by enhancing the contextual modeling ability of E2E SA-ASR. Specifically, in CASA-ASR, a contextual text encoder is involved to aggregate the semantic information of the whole utterance, and a context-dependent scorer is employed to model the speaker discriminability by contrasting with speakers in the context. In addition, a two-pass decoding strategy is further proposed to fully leverage the contextual modeling ability resulting in a better recognition performance. Experimental results on AliMeeting corpus show that the proposed CASA-ASR model outperforms the original E2E SAASR system with a relative improvement of 11.76% in terms of speaker-dependent character error rate.
Unsupervised Learning of Discrete Latent Representations with Data-Adaptive Dimensionality from Continuous Speech Streams
Authors: Shun Takahashi; Sakriani Sakti
Venue: Interspeech 2023 (main)

Abstract: This work presents a novel deep generative model for unsupervised learning of sparse binary feature representations with data-adaptive dimensionality directly from continuous speech streams. Sharing the critical assumption of unbounded latent dimensionality with previously proposed Bayesian non-parametric approaches, our proposed model can capture the much richer, non-Markovian dependencies between its latent representations. The present work focuses on an investigation of our proposed model's performance in learning linguistically meaningful representations under challenging, realistic scenarios. We train our model with highly speaker-imbalanced datasets and evaluate it on the ABX phone discriminability test. Our model achieves a promising, competitive performance to the state-of-the-art model, despite its huge disadvantage: limited or no access to speaker information during training.
AD-TUNING: An Adaptive CHILD-TUNING Approach to Efficient Hyperparameter Optimization of Child Networks for Speech Processing Tasks in the SUPERB Benchmark
Authors: Gaobin Yang; Jun Du; Maokui He; Shutong Niu; Baoxiang Li; Jiakui Li; Chin-Hui Lee
Venue: Interspeech 2023 (main)

Abstract: In this paper, we propose AD-TUNING, an adaptive CHILD-TUNING approach for hyperparameter tuning of child networks. To address the issue of selecting an optimal hyperparameter set P, which often varies for different tasks in CHILD-TUNING, we first analyze the distribution of parameter importance to ascertain the range of P. Next, we propose a simple yet efficient early-stop algorithm to select the appropriate child network from different sizes for various speech tasks. When evaluated on seven speech processing tasks in the SUPERB benchmark, our proposed framework only requires fine-tuning less than 0.1%~10% of pre-trained model parameters for each task to achieve state-of-the-art results in most of the tasks. For instance, the DER of the speaker diarization task is 9.22% relatively lower than the previously reported best results. Other benchmark results are also very competitive. Our code is available at https://github.com/liyunlongaaa/AD-TUNING.
Distilling knowledge from Gaussian process teacher to neural network student
Authors: Jeremy H. M. Wong; Huayun Zhang; Nancy F. Chen
Venue: Interspeech 2023 (main)

Abstract: Neural Networks (NN) and Gaussian Processes (GP) are different modelling approaches. The former stores characteristics of the training data in its many parameters, and then performs inference by parsing inputs through these parameters. The latter instead performs inference by computing a similarity between the test and training inputs, and then predicts test outputs that are correlated with the reference training outputs of similar inputs. These models may be combined to leverage upon their diversity. However, both combination and the matrix computations for GP inference are expensive. This paper investigates whether a NN student is able to effectively learn from the information distilled from a GP or ensemble teacher. It is computationally cheaper to infer using this student. Experiments on the speechocean762 spoken language assessment dataset suggest that learning is effective.
Segmental SpeechCLIP: Utilizing Pretrained Image-text Models for Audio-Visual Learning
Authors: Saurabhchand Bhati; JesÃºs Villalba; Laureano Moro-Velazquez; Thomas Thebaud; Najim Dehak
Venue: Interspeech 2023 (main)

Abstract: Visually grounded models learn from paired images and their spoken captions. Recently, there have been attempts to utilize the visually grounded models trained from images and their corresponding text captions, such as CLIP, to improve speech-based visually grounded models' performance. However, the majority of these models only utilize the pretrained image encoder. Cascaded SpeechCLIP attempted to generate localized word-level information and utilize both the pretrained image and text encoders. Despite using both, they noticed a substantial drop in retrieval performance. Here, we propose to use a hierarchical segmental audio encoder that can generate a sequence of word-like units from audio. We use the pretrained CLIP text encoder on top of these word-like units representations and show significant improvements over the cascaded variant of SpeechCLIP.
Towards hate speech detection in low-resource languages: Comparing ASR to acoustic word embeddings on Wolof and Swahili
Authors: Christiaan Jacobs; NathanaÃ«l Carraz Rakotonirina; Everlyn Asiko Chimoto; Bruce A. Bassett; Herman Kamper
Venue: Interspeech 2023 (main)

Abstract: We consider hate speech detection through keyword spotting on radio broadcasts. One approach is to build an automatic speech recognition (ASR) system for the target low-resource language. We compare this to using acoustic word embedding (AWE) models that map speech segments to a space where matching words have similar vectors. We specifically use a multilingual AWE model trained on labelled data from well-resourced languages to spot keywords in data in the unseen target language. In contrast to ASR, the AWE approach only requires a few keyword exemplars. In controlled experiments on Wolof and Swahili where training and test data are from the same domain, an ASR model trained on just five minutes of data outperforms the AWE approach. But in an in-the-wild test on Swahili radio broadcasts with actual hate speech keywords, the AWE model (using one minute of template data) is more robust, giving similar performance to an ASR system trained on 30 hours of labelled data.
Mitigating Catastrophic Forgetting for Few-Shot Spoken Word Classification Through Meta-Learning
Authors: Ruan van der Merwe; Herman Kamper
Venue: Interspeech 2023 (main)

Abstract: We consider the problem of few-shot spoken word classification in a setting where a model is incrementally introduced to new word classes. This would occur in a user-defined keyword system where new words can be added as the system is used. In such a continual learning scenario, a model might start to misclassify earlier words as newer classes are added, i.e. catastrophic forgetting. To address this, we propose an extension to model-agnostic meta-learning (MAML). In our new approach, each inner learning loopâwhere a model "learns how to learn" new classesâends with a single gradient update using stored templates from all the classes that the model has already seen (one template per class). We compare this method to OML (another extension of MAML) in few-shot isolated-word classification experiments on Google Commands and FACC. Our method consistently outperforms OML in experiments where the number of shots and the final number of classes are varied.
Online Punctuation Restoration using ELECTRA Model for streaming ASR Systems
Authors: Martin PolÃ¡Äek; Petr Äerva; JindÅich Å½ÄÃ¡nskÃ½; Lenka WeingartovÃ¡
Venue: Interspeech 2023 (main)

Abstract: In this work, we propose a lightweight online approach to automatic punctuation restoration (APR), which can be utilized in speech transcription systems for, e.g., live captioning TV or radio streams. It uses only text input without prosodic features and a fine-tuned ELECTRA-Small model with a two-layer classification head. It allows for restoring question marks, commas, and periods with a very short inference time and a low latency of just three words. Our APR scheme is first tuned and compared to other architectures on a set of manual TV news transcripts. The resulting system is then compared to another real-time APR module utilizing a recurrent network and a combination of text and acoustic features. The test data we use contains automatic transcripts of radio talks and TV debates; we are also publishing this data. The results show that our APR module performs better than the above-mentioned system and yields on the two test sets an average F1 of 71.2% and 69.4%, respectively.
Language Agnostic Data-Driven Inverse Text Normalization
Authors: Szu-Jui Chen; Debjyoti Paul; Yutong Pang; Peng Su; Xuedong Zhang
Venue: Interspeech 2023 (main)

Abstract: The rise of automatic speech recognition (ASR) models has created an urgent need for converting spoken language into written text to provide better user experiences. This has drawn the attention of researchers, particularly for real-time on-device ASR deployment, towards the inverse text normalization (ITN) problem. While data-driven ITN methods have shown great promise in recent studies, the lack of labeled spoken-written datasets is hindering the development for non-English data-driven ITN. To bridge this gap, we propose a language-agnostic data-driven ITN framework that leverages data augmentation and neural machine translation specifically designed for real-time miniature models and low-resource languages. Additionally, we have developed an evaluation method for language-agnostic ITN models when only English data is available. Our empirical evaluation attests to the efficacy of this language-agnostic ITN modeling with data augmentation approach for multiple non-English languages.
How to Estimate Model Transferability of Pre-Trained Speech Models?
Authors: Zih-Ching Chen; Chao-Han Huck Yang; Bo Li; Yu Zhang; Nanxin Chen; Shuo-Yiin Chang; Rohit Prabhavalkar; Hung-yi Lee; Tara Sainath
Venue: Interspeech 2023 (main)

Abstract: In this work, we introduce a ''score-based assessment'' framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks. We leverage upon two representation theories, Bayesian likelihood estimation and optimal transport, to generate rank scores for the PSM candidates using the extracted representations. Our framework efficiently computes transferability scores without actual fine-tuning of candidate models or layers by making a temporal independent hypothesis. We evaluate some popular supervised speech models (and self-supervised speech models in cross-layer and cross-model settings using public data. Experimental results show a high Spearman's rank correlation and low p-value between our estimation framework and fine-tuning ground truth. Our proposed transferability framework requires less computational time and resources, making it a resource-saving and time-efficient approach for tuning speech foundation models.
Transcribing Speech as Spoken and Written Dual Text Using an Autoregressive Model
Authors: Mana Ihori; Hiroshi Sato; Tomohiro Tanaka; Ryo Masumura; Saki Mizuno; Nobukatsu Hojo
Venue: Interspeech 2023 (main)

Abstract: This paper proposes a novel method to jointly generate spoken and written text from input speech for expanding use cases of speech-based applications. The spoken text generated using speech-to-spoken text systems, i.e., speech recognition systems, has disfluencies and no punctuation marks. Thus, spoken text is often converted into written text using a spoken text-to-written text system. However, this cascading is unsuitable for overall optimization and computationally expensive. Although a speech-to-written-text system that directly outputs the written text from the speech is also developed, it cannot output the spoken text. To efficiently produce both spoken and written text from speech, our key advance is to handle a joint text of spoken and written texts in an autoregressive model. This enables us to correctly generate both spoken and written texts by considering their dependencies via a single decoding process. Our experiments demonstrate the effectiveness of the proposed method.