GenAI4ELab/papercli-papers huggingface dataset:
Affective attributes of French caregivers' professional speech
Authors: Jean-Luc Rouas; Yaru Wu; Takaaki Shochi
Venue: Interspeech 2023 (main)

Abstract: In this paper, we detail our approach to studying the vocal characteristics of caregivers in retirement homes. To achieve this goal, we conducted recordings of 20 professional caregivers across two retirement homes. Using headset microphones connected to smartphones, we were able to capture the caregivers' speech while allowing them complete freedom of movement without compromising sound quality. The recordings consisted of three tasks: reading text, informal interviews, and professional role-play scenarios with a fictitious patient. We processed the recordings using an automatic speech recognition system, which provided word or phone sequences and their corresponding timestamps. Our analysis focused on identifying differences in emotional tone, lexical content, speech rate, fundamental frequency, and intensity between spontaneous speech conditions. Ultimately, our aim is to develop automated training tools that capture the unique vocal characteristics of professional caregivers.
ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversion
Authors: Edresson Casanova; Christopher Shulby; Alexander Korolev; Arnaldo Candido Junior; Anderson da Silva Soares; Sandra AluÃ­sio; Moacir Antonelli Ponti
Venue: Interspeech 2023 (main)

Abstract: We explore cross-lingual multi-speaker speech synthesis and cross-lingual voice conversion applied to data augmentation for automatic speech recognition (ASR) systems in low/medium-resource scenarios. Through extensive experiments, we show that our approach permits the application of speech synthesis and voice conversion to improve ASR systems using only one target-language speaker during model training. We also managed to close the gap between ASR models trained with synthesized versus human speech compared to other works that use many speakers. Finally, we show that it is possible to obtain promising ASR training results with our data augmentation method using only a single real speaker in a target language.
Personality-aware Training based Speaker Adaptation for End-to-end Speech Recognition
Authors: Yue Gu; Zhihao Du; Shiliang Zhang; Qian Chen; Jiqing Han
Venue: Interspeech 2023 (main)

Abstract: Speaker adaptation has been widely studied to solve the mismatch between training and test conditions for end-to-end automatic speech recognition (ASR). A key challenge of speaker adaptation is lack of sufficient annotated target-speaker data. Considering the training set is always a large-scale one and contains various speakers, it is likely that utterances in the training set can have similar voice characters with the target speaker, and naturally those similar utterances can be treated as a supplement for target speaker data in the adaptation process. Therefore, we propose personality-aware training (PAT) framework to adapt a pre-trained ASR to the target speaker. In PAT, the small-scale target speaker data is viewed as anchors, and the losses of training samples are re-weighted according to the voice character similarity between the anchors and training samples, where the voice character similarity is derived from the speaker or prosody embedding extractor. Experiments on KeSpeech and MagicData corpora show that, compared with the unadapted system, the proposed method achieves 6.35% and 11.86% relative reduction on character error rate with only 10-minute pseudo-label and true-label adaptation data, respectively.
Target Vocabulary Recognition Based on Multi-Task Learning with Decomposed Teacher Sequences
Authors: Aoi Ito; Tatsuya Komatsu; Yusuke Fujita; Yusuke Kida
Venue: Interspeech 2023 (main)

Abstract: This paper proposes a method for target vocabulary recognition based on multi-task learning with decomposed teacher sequences. The proposed method first decomposes teacher sequences into the target vocabulary and the non-target vocabulary sequences. Then, multi-task learning is performed by calculating losses for both the target vocabulary sequence and the non-target vocabulary sequence. By utilizing information from both target and non-target vocabulary, our proposed method provides more stable training and more accurate recognition of target vocabulary than single-task learning using only the target vocabulary. Experiments conducted on the Corpus of Spontaneous Japanese (CSJ) dataset, using numerals and katakana as target vocabulary, demonstrate the effectiveness of our proposed method. The results show a maximum CER improvement rate of 27% for katakana and 34% for numerals in target vocabulary recognition, as well as an 84% reduction in insertion errors in non-target vocabulary utterances.
Wave to Syntax: Probing spoken language models for syntax
Authors: Gaofei Shen; Afra Alishahi; Arianna Bisazza; Grzegorz ChrupaÅa
Venue: Interspeech 2023 (main)

Abstract: Understanding which information is encoded in deep models of spoken and written language has been the focus of much research in recent years, as it is crucial for debugging and improving these architectures. Most previous work has focused on probing for speaker characteristics, acoustic and phonological information in models of spoken language, and for syntactic information in models of written language. Here we focus on the encoding of syntax in several self-supervised and visually grounded models of spoken language. We employ two complementary probing methods, combined with baselines and reference representations to quantify the degree to which syntactic structure is encoded in the activations of the target models. We show that syntax is captured most prominently in the middle layers of the networks, and more explicitly within models with more parameters.
Effective Training of Attention-based Contextual Biasing Adapters with Synthetic Audio for Personalised ASR
Authors: Burin Naowarat; Philip Harding; Pasquale D'Alterio; Sibo Tong; Bashar Awwad Shiekh Hasan
Venue: Interspeech 2023 (main)

Abstract: Contextual biasing (CB) is an effective approach for contextualising hidden features of neural transducer ASR models to improve rare word recognition. CB relies on relatively large quantities of relevant human annotated natural speech during training, limiting its effectiveness in low-resource scenarios. In this work, we propose a novel approach that reduces the reliance on real speech by using synthesised audios for training CB adapters. We introduce a projection module (PM) that transforms encoder features of synthesised audios prior to CB training to better match real speech. We penalise PM with consistency regularisation to encourage higher similarity between features of real and synthesised speech. The proposed method maintains the same performance on both named-entity and general datasets while using half of the real speech data for CB training. Furthermore, we show a 16% word error rate reduction when the full real-speech training dataset is extended with synthetic utterances.
Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation
Authors: Ziyang Ma; Zhisheng Zheng; Guanrou Yang; Yu Wang; Chao Zhang; Xie Chen
Venue: Interspeech 2023 (main)

Abstract: The excellent generalization ability of self-supervised learning (SSL) for speech foundation models has garnered significant attention. HuBERT is a successful example that utilizes offline clustering to convert speech features into discrete units for a masked language modeling pretext task. However, simply clustering features as targets by k-means does not fully inspire the model's performance. In this work, we present an unsupervised method to improve SSL targets. Two models are proposed, MonoBERT and PolyBERT, which leverage context-independent and context-dependent phoneme-based units for pre-training. Our models outperform other SSL models significantly on the LibriSpeech benchmark without the need for iterative re-clustering and re-training. Furthermore, our models equipped with context-dependent units even outperform target-improvement models that use labeled data during pre-training. How we progressively improve the unit discovery process is demonstrated through experiments.
SlothSpeech: Denial-of-service Attack Against Speech Recognition Models
Authors: Mirazul Haque; Rutvij Shah; Simin Chen; Berrak Sisman; Cong Liu; Wei Yang
Venue: Interspeech 2023 (main)

Abstract: Deep Learning (DL) models have been popular nowadays to execute different speech-related tasks, including automatic speech recognition (ASR). As ASR is being used in different real-time scenarios, it is important that the ASR model remains efficient against minor perturbations to the input. Hence, evaluating efficiency robustness of the ASR model is the need of the hour. We show that popular ASR models like Speech2Text model and Whisper model have dynamic computation based on different inputs, causing dynamic efficiency. In this work, we propose SlothSpeech, a denial-of-service attack against ASR models, which exploits the dynamic behaviour of the model. SlothSpeech uses the probability distribution of the output text tokens to generate perturbations to the audio such that efficiency of the ASR model is decreased. We find that SlothSpeech generated inputs can increase the latency up to 40X times the latency induced by benign input.
CLRL-Tuning: A Novel Continual Learning Approach for Automatic Speech Recognition
Authors: Zhihan Wang; Feng Hou; Ruili Wang
Venue: Interspeech 2023 (main)

Abstract: In this paper, we propose a novel Continual Learning approach, which is Randomly Layer-wise Tuning (CLRL-Tuning) of a pre-trained Automatic Speech Recognition (ASR) model. CLRL-Tuning tackles the randomness of subsequent datasets by updating the parameters of randomly selected encoder layers of the pre-trained model (such as wav2vec 2.0) for every training epoch. CLRL-Tuning is different from the previous approaches in that it neither uses previous datasets, nor expands/runs previous models. Furthermore, we perform experiments to evaluate our approach compared with four strong baselines, including Knowledge Distillation and Gradient Episodic Memory. Our approach achieves significant improvements over the baselines in average word error rate (WER) for the wav2vec 2.0 model. Additionally, we implement ablation studies for our approach by tuning one, three, six and full encoder layers of the model, and experimental results show only tuning one encoder layer of the model at each training epoch is the most effective way to mitigate catastrophic forgetting.
Exploring Sources of Racial Bias in Automatic Speech Recognition through the Lens of Rhythmic Variation
Authors: Li-Fang Lai; Nicole Holliday
Venue: Interspeech 2023 (main)

Abstract: Although studies have shown that one issue of bias in modern automatic speech recognition (ASR) technologies is degraded performance for African American English (AAE) speakers, the mechanism by which systems fail for AAE speakers is still not well-understood. The present study aims to offer insight into this issue by examining whether errors are driven by rhythmic variation in ethnolects. We computed seven quantitative measures of speech rhythm in a reading task as produced by AAE and General American English (GAE) speakers and related these metrics to word error rates. The results confirmed racial bias against AAE speakers with higher error rates when AAE speakers produced more variable durations in vowel sounds. Rhythmic variation, on the other hand, is not a contributing factor for the errors in GAE. The result calls for interdisciplinary collaboration between linguists and ASR builders to add timing components of speech to the system to ensure fairness in artificial intelligence for currently underserved groups.
Can Contextual Biasing Remain Effective with Whisper and GPT-2?
Authors: Guangzhi Sun; Xianrui Zheng; Chao Zhang; Philip C. Woodland
Venue: Interspeech 2023 (main)

Abstract: End-to-end automatic speech recognition (ASR) and large language models, such as Whisper and GPT-2, have recently been scaled to use vast amounts of training data. Despite the large amount of training data, infrequent content words that occur in a particular task may still exhibit poor ASR performance, with contextual biasing a possible remedy. This paper investigates the effectiveness of neural contextual biasing for Whisper combined with GPT-2. Specifically, this paper proposes integrating an adapted tree-constrained pointer generator (TCPGen) component for Whisper and a dedicated training scheme to dynamically adjust the final output without modifying any Whisper model parameters. Experiments across three datasets show a considerable reduction in errors on biasing words with a biasing list of 1000 words. Contextual biasing was more effective when applied to domain-specific data and can boost the performance of Whisper and GPT-2 without losing their generality.
Masked Modeling Duo for Speech: Specializing General-Purpose Audio Representation to Speech using Denoising Distillation
Authors: Daisuke Niizumi; Daiki Takeuchi; Yasunori Ohishi; Noboru Harada; Kunio Kashino
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning general-purpose audio representations have demonstrated high performance in a variety of tasks. Although they can be optimized for application by fine-tuning, even higher performance can be expected if they can be specialized to pre-train for an application. This paper explores the challenges and solutions in specializing general-purpose audio representations for a specific application using speech, a highly demanding field, as an example. We enhance Masked Modeling Duo (M2D), a general-purpose model, to close the performance gap with state-of-the-art (SOTA) speech models. To do so, we propose a new task, denoising distillation, to learn from fine-grained clustered features, and M2D for Speech (M2D-S), which jointly learns the denoising distillation task and M2D masked prediction task. Experimental results show that M2D-S performs comparably to or outperforms SOTA speech models on the SUPERB benchmark, demonstrating that M2D can specialize in a demanding field.
Improving RNN Transducer Acoustic Models for English Conversational Speech Recognition
Authors: Xiaodong Cui; George Saon; Brian Kingsbury
Venue: Interspeech 2023 (main)

Abstract: In this paper we investigate several techniques for improving the performance of RNN transducer (RNNT) acoustic models for conversational speech recognition and report state-of-the-art word error rates (WERs) on the 2000-hour Switchboard dataset. We show that n-best label smoothing and length perturbation which show improved performance on the smaller 300-hour dataset are also very effective on large datasets. We further give a rigorous theoretical interpretation of the n-best label smoothing based on stochastic approximation for training RNNT under the maximum likelihood criterion. Random quantization is also introduced to improve the generalization of RNNT models. On the 2000-hour Switchboard dataset, we report a single model performance of 4.9% and 7.7% WERs on the Switchboard and CallHome portions of NIST Hub5 2000, 7.1% on NIST Hub5 2001 and 6.8% on NIST RT03, without using external LMs.
MixRep: Hidden Representation Mixup for Low-Resource Speech Recognition
Authors: Jiamin Xie; John H. L. Hansen
Venue: Interspeech 2023 (main)

Abstract: In this paper, we present MixRep, a simple and effective data augmentation strategy based on mixup for low-resource ASR. MixRep interpolates the feature dimensions of hidden representations in the neural network that can be applied to both the acoustic feature input and the output of each layer, which generalizes the previous MixSpeech method. Further, we propose to combine the mixup with a regularization along the time axis of the input, which is shown as complementary. We apply MixRep to a Conformer encoder of an E2E LAS architecture trained with a joint CTC loss. We experiment on the WSJ dataset and subsets of the SWB dataset, covering reading and telephony conversational speech. Experimental results show that MixRep consistently outperforms other regularization methods for low-resource ASR. Compared to a strong SpecAugment baseline, MixRep achieves a +6.5% and a +6.7% relative WER reduction on the eval92 set and the Callhome part of the eval'2000 set.
Adapting Multi-Lingual ASR Models for Handling Multiple Talkers
Authors: Chenda Li; Yao Qian; Zhuo Chen; Naoyuki Kanda; Dongmei Wang; Takuya Yoshioka; Yanmin Qian; Michael Zeng
Venue: Interspeech 2023 (main)

Abstract: State-of-the-art large-scale universal speech models (USMs) show a decent automatic speech recognition (ASR) performance across multiple domains and languages. However, it remains a challenge for these models to recognize overlapped speech, which is often seen in meeting conversations. We propose an approach to adapt USMs for multi-talker ASR. We first develop an enhanced version of serialized output training to jointly perform multi-talker ASR and utterance timestamp prediction. That is, we predict the ASR hypotheses for all speakers, count the speakers, and estimate the utterance timestamps at the same time. We further introduce a lightweight adapter module to maintain the multilingual property of the USMs even when we perform the adaptation with only a single language. Experimental results obtained using the AMI and AliMeeting corpora show that our proposed approach effectively transfers the USMs to a strong multilingual multi-talker ASR model with timestamp prediction capability.
Adapter-tuning with Effective Token-dependent Representation Shift for Automatic Speech Recognition
Authors: Dianwen Ng; Chong Zhang; Ruixi Zhang; Yukun Ma; Trung Hieu Nguyen; Chongjia Ni; Shengkui Zhao; Qian Chen; Wen Wang; Eng Siong Chng; Bin Ma
Venue: Interspeech 2023 (main)

Abstract: The use of self-supervised pre-trained speech models has greatly improved speech tasks in low-resource settings. However, fine-tuning the entire model can be computationally expensive and not scalable for multiple tasks (e.g., personalized ASR). While recent approaches have tried to solve this issue by training adapters, they fail to match the performance of full fine-tuning models, possibly due to the challenge of task domain transferability. Our proposed method enhances the performance of vanilla adapter tuning for ASR by using a simple yet effective token-dependent bias. This approach adds a token-specific representation shift (bias) to the intermediate representations of a pre-trained model, which better maps the latent features of the frozen network to the task domain. Our approach yields better recognition results with the adapter tuning strategy and achieves the performance of a full fine-tuning model on clean LibriSpeech while maintaining its lightweight nature.
Model-Internal Slot-triggered Biasing for Domain Expansion in Neural Transducer ASR Models
Authors: Yiting Lu; Philip Harding; Kanthashree Mysore Sathyendra; Sibo Tong; Xuandi Fu; Jing Liu; Feng-Ju Chang; Simon Wiesler; Grant P. Strimel
Venue: Interspeech 2023 (main)

Abstract: Personal rare word recognition is an important yet challenging task for end-to-end speech recognition. Contextual biasing has demonstrated success in tackling this problem. Though effective in improving rare word recognition, these mechanisms can lead to errors due to false-biasing while facing further challenges when attempting to expand them to many domains. To address these limitations, in this work we propose a neural biasing design with a streaming model-internal slot classifier, trained to categorise the domain of each word piece before it is emitted. The neural biasing module can therefore be triggered in a controlled way, permitting natural scaling to many domains while reducing false-biasing and computational cost. Experiments on diverse domain slot types of application names, communications and playlist names demonstrate the proposed architecture results in 26% to 58% relative improvements on personal rare word recognition with minimal impact (0.6% rel.) on general data.
Delay-penalized CTC Implemented Based on Finite State Transducer
Authors: Zengwei Yao; Wei Kang; Fangjun Kuang; Liyong Guo; Xiaoyu Yang; Yifan Yang; Long Lin; Daniel Povey
Venue: Interspeech 2023 (main)

Abstract: Connectionist Temporal Classification (CTC) suffers from the latency problem when applied to streaming models. We argue that in CTC lattice, the alignments that can access more future context are preferred during training, thereby leading to higher symbol delay. In this work we propose the delay-penalized CTC which is augmented with latency penalty regularization. We devise a flexible and efficient implementation based on the differentiable Finite State Transducer (FST). Specifically, by attaching a binary attribute to CTC topology, we can locate the frames that firstly emit non-blank tokens on the resulting CTC lattice, and add the frame offsets to the log-probabilities. Experimental results demonstrate the effectiveness of our proposed delay-penalized CTC, which is able to balance the delay-accuracy trade-off. Furthermore, combining the delay-penalized transducer enables the CTC model to achieve better performance and lower latency. Our work is open-sourced and publicly available.
Text-Only Domain Adaptation for End-to-End Speech Recognition through Down-Sampling Acoustic Representation
Authors: Jiaxu Zhu; Weinan Tong; Yaoxun Xu; Changhe Song; Zhiyong Wu; Zhao You; Dan Su; Dong Yu; Helen Meng
Venue: Interspeech 2023 (main)

Abstract: Mapping two modalities, speech and text, into a shared representation space, is a research topic of using text-only data to improve end-to-end automatic speech recognition (ASR) performance in new domains. However, the length of speech representation and text representation is inconsistent. Although the previous method up-samples the text representation to align with acoustic modality, it may not match the expected actual duration. In this paper, we proposed a novel representations match strategy through down-sampling acoustic representation to align with text modality. By introducing a continuous integrate-and-fire (CIF) module generating acoustic representations consistent with token length, our ASR model can learn unified representations from both modalities better, allowing for domain adaptation using text-only data of the target domain. Experiment results of new domain data demonstrate the effectiveness of the proposed method.
Knowledge Distillation Approach for Efficient Internal Language Model Estimation
Authors: Zhipeng Chen; Haihua Xu; Yerbolat Khassanov; Yi He; Lu Lu; Zejun Ma; Ji Wu
Venue: Interspeech 2023 (main)

Abstract: Internal language model estimation (ILME) has demonstrated its efficacy in domain adaptation for end-to-end (E2E) ASR. However, the performance improvement is achieved at the expense of computational cost, compared with conventional shallow fusion. To estimate the internal language model prior, one should run an extra forward operation on either ASR decoder or a separate density ratio (DR) language model (LM) for each decoding utterance. In this paper, we propose to employ knowledge distillation (KD) approach to realize efficient ILME for the Listen-Attend-Spell (LAS) E2E ASR model. First, we extensively explore diverse ILME and DR methods. We find that the ILM can be approximated with a DR-LM much smaller than the original ASR decoder. Furthermore, to reach the performance of ILME, we propose to employ the estimated ILM as teacher to teach a small DR-LM by KD. In this way, we achieve the best of both worlds: comparable performance to ILME and high efficiency of DR with a small DR-LM.