GenAI4ELab/papercli-papers huggingface dataset:
Lifelong Learning MOS Prediction for Synthetic Speech Quality Evaluation
Authors: FÃ©lix Saget; Meysam Shamsi; Marie Tahon
Venue: Interspeech 2024 (main)

Abstract: Mean Opinion Score (MOS) has been a long-standing standard for perceptive evaluation of quality of speech synthesis models; however, this criterion is hardly reproducible, and costly. Automatic, neural MOS predictors have emerged as a solution to the objective assessment of synthetic speech. These predictors are trained once on data collected from past listening tests, and thus may suffer from adaptation to new technology breakthrough in speech synthesis. In this study, we investigate the applicability of lifelong learning for MOS predictors, where the training samples would be fed to the model in the chronological order. A sequential lifelong mode and a cumulative lifelong mode have been compared with traditional batch training using the BVCC and Blizzard Challenge datasets. The experiments show the advantages of lifelong learning in cross-corpus evaluation as well as in a constrained data availability scenario.
Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
Authors: Chin Yuen Kwok; Jia Qi Yip; Eng Siong Chng
Venue: Interspeech 2024 (main)

Abstract: Continual Learning (CL) involves fine-tuning pre-trained models with new data while maintaining the performance on the pre-trained data. This is particularly relevant for expanding multilingual ASR (MASR) capabilities. However, existing CL methods, mainly designed for computer vision and reinforcement learning tasks, often yield sub-optimal results when directly applied to MASR. We hypothesise that this is because CL of the auto-regressive decoder in the MASR model is difficult. To verify this, we propose four optimizations on the decoder. They include decoder-layer gradient surgery, freezing unused token embeddings, suppressing output of newly added tokens, and learning rate re-scaling. Our experiments on adapting Whisper to 10 unseen languages from the Common Voice dataset demonstrate that these optimizations reduce the Average Word Error Rate (AWER) of pretrained languages from 14.2% to 12.4% compared with Experience Replay, without compromising the AWER of new languages.
ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets
Authors: Jiatong Shi; Shih-Heng Wang; William Chen; Martijn Bartelds; Vanya Bannihatti Kumar; Jinchuan Tian; Xuankai Chang; Dan Jurafsky; Karen Livescu; Hung-yi Lee; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: ML-SUPERB evaluates self-supervised learning (SSL) models on the tasks of language identification and automatic speech recognition (ASR). This benchmark treats the models as feature extractors and uses a single shallow downstream model, which can be fine-tuned for a downstream task. However, real-world use cases may require different configurations. This paper presents ML-SUPERB 2.0, which is a new benchmark for evaluating pre-trained SSL and supervised speech models across downstream models, fine-tuning setups, and efficient model adaptation approaches. We find performance improvements over the setup of ML-SUPERB. However, performance depends on the downstream model design. Also, we find large performance differences between languages and datasets, suggesting the need for more targeted approaches to improve multilingual ASR performance.
Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognition
Authors: AndrÃ©s PiÃ±eiro-MartÃ­n; Carmen GarcÃ­a-Mateo; Laura Docio-Fernandez; MarÃ­a del Carmen LÃ³pez-PÃ©rez; Georg Rehm
Venue: Interspeech 2024 (main)

Abstract: This paper addresses the challenge of integrating low-resource languages into multilingual automatic speech recognition (ASR) systems. We introduce a novel application of weighted cross-entropy, typically used for unbalanced datasets, to facilitate the integration of low-resource languages into pre-trained multilingual ASR models within the context of continual multilingual learning. We fine-tune the Whisper multilingual ASR model on five high-resource languages and one low-resource language, employing language-weighted dynamic cross-entropy and data augmentation. The results show a remarkable 6.69% word error rate (WER) reduction for the low-resource language compared to the fine-tuned model without applying our approach, and a 48.86% WER reduction compared to the original Whisper model. In addition, our approach yields an average WER reduction of 3.29% across the six languages, showing no degradation for the high-resource languages.
M2ASR: Multilingual Multi-task Automatic Speech Recognition via Multi-objective Optimization
Authors: A F M Saif; Lisha Chen; Xiaodong Cui; Songtao Lu; Brian Kingsbury; Tianyi Chen
Venue: Interspeech 2024 (main)

Abstract: To enable the capability of speech models across multiple languages, training multilingual, multi-task automatic speech recognition (ASR) models has gained growing interest. However, different languages and tasks result in distinct training objectives, potentially leading to conflicts during training and degrading the model's performance. To overcome this issue, we introduce M2ASR, a multilingual, multi-task ASR framework, which formulates the problem as a constrained multi-objective optimization (MOO), where multilingual multi-task supervised training augmented by speech-to-text translation (S2TT) serve as supervised objectives and are subject to the desired performance of multilingual unsupervised training. We employ MOO techniques to avoid conflicts among multiple linguistic representations and tasks during training. Extensive experiments demonstrate that M2ASR outperforms conventional multilingual ASR models by 28.3% to 38.6% across diverse ASR tasks.
MSR-86K: An Evolving, Multilingual Corpus with 86,300 Hours of Transcribed Audio for Speech Recognition Research
Authors: Song Li; Yongbin You; Xuezhi Wang; Zhengkun Tian; Ke Ding; Guanglu Wan
Venue: Interspeech 2024 (main)

Abstract: Recently, multilingual artificial intelligence assistants, exemplified by ChatGPT, have gained immense popularity. As a crucial gateway to human-computer interaction, multilingual automatic speech recognition (ASR) has also garnered significant attention, as evidenced by systems like Whisper. However, the proprietary nature of the training data has impeded researchersâ efforts to study multilingual ASR. This paper introduces MSR-86K, an evolving, large-scale multilingual corpus for speech recognition research. The corpus is derived from publicly accessible videos on YouTube, comprising 15 languages and a total of 86,300 hours of transcribed ASR data. We also introduce how to use the MSR-86K corpus and other open-source corpora to train a robust multilingual ASR model that is competitive with Whisper. MSR-86K will be publicly released on HuggingFace, and we believe that such a large corpus will pave new avenues for research in multilingual ASR.
Improving Multilingual ASR Robustness to Errors in Language Input
Authors: Brady Houston; Omid Sadjadi; Zejiang Hou; Srikanth Vishnubhotla; Kyu J. Han
Venue: Interspeech 2024 (main)

Abstract: Explicitly adding language information to multilingual ASR models during training has been shown to improve their performance. However, this also requires using language information during inference. In cascaded systems, this language label may come from external language identification models, which are susceptible to errors. In this work, we characterize the sensitivity to errors in language inputs of several common language-incorporation strategies used in multilingual ASR. We show that some of these strategies are highly sensitive to the correctness of language information being used during inference, and also demonstrate that introducing a small amount of language label noise during training can greatly improve the modelâs robustness to incorrect language information. As multilingual ASR continues to become more common, this work demonstrates the importance of understanding the sensitivity of these models to language inputs and ensuring models are robust to errors.
Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions
Authors: Jiwon Suh; Injae Na; Woohwan Jung
Venue: Interspeech 2024 (main)

Abstract: End-to-end automatic speech recognition (E2E ASR) systems have significantly improved speech recognition through training on extensive datasets. Despite these advancements, they still struggle to accurately recognize domain specific words, such as proper nouns and technical terminologies. To address this problem, we propose a method to utilize the state-of-the-art Whisper without modifying its architecture, preserving its generalization performance while enabling it to leverage descriptions effectively. Moreover, we propose two additional training techniques to improve the domain specific ASR: decoder fine-tuning, and context perturbation. We also propose a method to use a Large Language Model (LLM) to generate descriptions with simple metadata, when descriptions are unavailable. Our experiments demonstrate that proposed methods notably enhance domain-specific ASR accuracy on real-life datasets, with LLM-generated descriptions outperforming human-crafted ones in effectiveness.
A Multitask Training Approach to Enhance Whisper with Open-Vocabulary Keyword Spotting
Authors: Yuang Li; Min Zhang; Chang Su; Yinglu Li; Xiaosong Qiao; Mengxin Ren; Miaomiao Ma; Daimeng Wei; Shimin Tao; Hao Yang
Venue: Interspeech 2024 (main)

Abstract: The recognition of rare named entities, such as personal names and terminologies, is challenging for automatic speech recognition (ASR) systems, especially when they are not frequently observed in the training data. In this paper, we introduce keyword spotting enhanced Whisper (KWS-Whisper), a novel ASR system that leverages the Whisper model and performs open-vocabulary keyword spotting (OV-KWS) on the hidden states of the Whisper encoder to recognize user-defined named entities. These entities serve as prompts for the Whisper decoder. To optimize the model, we propose a multitask training approach that learns OV-KWS and contextual-ASR tasks. We evaluate our approach on Chinese Aishell hot word subsets and two internal code-switching test sets and show that it significantly improves the entity recall compared to the original Whisper model. Moreover, we demonstrate that the OV-KWS can be a plug-and-play module to enhance the ASR error correction methods and frozen Whisper models.
CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions
Authors: Mario Zusag; Laurin Wagner; Bernhad Thallinger
Venue: Interspeech 2024 (main)

Abstract: We demonstrate that carefully adjusting the tokenizer of the Whisper speech recognition model significantly improves the precision of word-level timestamps when applying dynamic time warping to the decoderâs cross-attention scores. We fine- tune the model to produce more verbatim speech transcriptions and employ several techniques to increase robustness against multiple speakers and background noise. These adjustments achieve state-of-the-art performance on benchmarks for verba- tim speech transcription, word segmentation, and the timed de- tection of filler events, and can further mitigate transcription hallucinations. The code is available open source.
On Disfluency and Non-lexical Sound Labeling for End-to-end Automatic Speech Recognition
Authors: Peter Mihajlik; Yan Meng; Mate S Kadar; Julian Linke; Barbara Schuppler; Katalin MÃ¡dy
Venue: Interspeech 2024 (main)

Abstract: Spontaneous speech contains a significant amount of disfluencies and non-lexical sounds (e.g., backchannels, filled pauses), which are often difficult to transcribe. Disfluency labeling for automatic speech recognition (ASR) aims at editing these phenomena in the transcription to improve overall recognition accuracy. Such labeling techniques typically delete non-lexical/disfluent labels from the prediction, where classical ASR techniques either ignore or treat them as lexical items. Our results, obtained by systematic comparison and detailed evaluation of various disfluency labeling methods on two different language conversational corpora, suggest that neither of the previous approaches are optimal. We propose to distinguish between filled pauses and meaningful conversational grunts and show that keeping the non-lexical labels is not only possible but as low as 7% label error rates can be achieved for highly important categories (including 'mhm') while preserving a decent WER.
Inclusive ASR for Disfluent Speech: Cascaded Large-Scale Self-Supervised Learning with Targeted Fine-Tuning and Data Augmentation
Authors: Dena Mujtaba; Nihar R. Mahapatra; Megan Arney; J. Scott Yaruss; Caryn Herring; Jia Bin
Venue: Interspeech 2024 (main)

Abstract: Automatic speech recognition (ASR) systems often falter while processing stuttering-related disfluencies---such as involuntary blocks and word repetitions---yielding inaccurate transcripts.  A critical barrier to progress is the scarcity of large, annotated disfluent speech datasets. Therefore, we present an inclusive ASR design approach, leveraging large-scale self-supervised learning on standard speech followed by targeted fine-tuning and data augmentation on a smaller, curated dataset of disfluent speech. Our data augmentation technique enriches training datasets with various disfluencies, enhancing ASR processing of these speech patterns. Results show that fine-tuning wav2vec 2.0 with even a relatively small, labeled dataset, alongside data augmentation, can significantly reduce word error rates for disfluent speech. Our approach not only advances ASR inclusivity for people who stutter, but also paves the way for ASRs that can accommodate wider speech variations.
DualPure: An Efficient Adversarial Purification Method for Speech Command Recognition
Authors: Hao Tan; Xiaochen Liu; Huan Zhang; Junjian Zhang; Yaguan Qian; Zhaoquan Gu
Venue: Interspeech 2024 (main)

Abstract: Adversarial examples pose a security threat to Autopilot's speech command recognition module, which attracted widespread attention from researchers. Previous works purify the malicious adversarial perturbations through pre-processing data from the time and frequency domain information. However, these methods either have a weak purification capacity or require a significant purification cost. To tackle this problem, we propose a real-time and efficient purification-based defense method DualPure, which combines the two defense aspects in the time and frequency domain for copurification. Specifically, we first disrupt the potential malicious perturbation in the sample at the waveform level and then apply an unconditional diffusion model to purify the feature at the frequency level. Numerous experiments show that the proposed method can effectively purify and achieve good adversarial robustness in white-box attacks (+ 6.3%) and black-box attacks (+  1.08%).
A Comparative Analysis of Bilingual and Trilingual Wav2Vec Models for Automatic Speech Recognition in Multilingual Oral History Archives
Authors: Jan LeheÄka; Josef V. Psutka; Lubos Smidl; Pavel Ircing; Josef Psutka
Venue: Interspeech 2024 (main)

Abstract: In this paper, we are comparing monolingual Wav2Vec 2.0 models with various multilingual models to see whether we could improve speech recognition performance on a unique oral history archive containing a lot of mixed-language sentences. Our main goal is to push forward research on this unique dataset, which is an extremely valuable part of our cultural heritage. Our results suggest that monolingual speech recognition models are, in most cases, superior to multilingual models, even when processing the oral history archive full of mixed-language sentences from non-native speakers. We also performed the same experiments on the public CommonVoice dataset to verify our results. We are contributing to the research community by releasing our pre-trained models to the public.
A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models
Authors: Anton de la Fuente; Dan Jurafsky
Venue: Interspeech 2024 (main)

Abstract: This study asks how self-supervised speech models represent suprasegmental categories like Mandarin lexical tone, English lexical stress, and English phrasal accents. Through a series of probing tasks, we make layer-wise comparisons of English and Mandarin 12 layer monolingual models. Our findings suggest that 1) English and Mandarin wav2vec 2.0 models learn contextual representations of abstract suprasegmental categories which are strongest in the middle third of the network. 2) Models are better at representing features that exist in the language of their training data, and this difference is driven by enriched context in transformer blocks, not local acoustic representation. 3) Fine-tuned wav2vec 2.0 improves performance in later layers compared to pre-trained models mainly for lexically contrastive features like tone and stress, 4) HuBERT and WavLM learn similar representations to wav2vec 2.0, differing mainly in later layer performance. Our results extend previous understanding of how models represent suprasegmentals and offer new insights into the language-specificity and contextual nature of these representations.
Fine-Tuning Strategies for Dutch Dysarthric Speech Recognition: Evaluating the Impact of Healthy, Disease-Specific, and Speaker-Specific Data
Authors: Spyretta Leivaditi; Tatsunari Matsushima; Matt Coler; Shekhar Nayak; Vass Verkhodanova
Venue: Interspeech 2024 (main)

Abstract: Despite significant advancements in automatic speech recognition technology  (ASR) the performance of such systems on dysarthric speech is still inadequate for widespread use. One key reason is the lack of sufficiently rich and diverse dysarthric speech datasets to train machine learning models that could handle all types and varieties of such speech. Motivated by the data scarcity problem, as well as by successful applications of self-supervised learning (SSL) in ASR for low-resource languages, this paper investigates and evaluates the effectiveness of three different data-centric SSL training strategies in improving Dutch dysarthric speech recognition. The first strategy involves fine-tuning with both dysarthric and healthy speech data, the second with disease-specific data and the third with speaker-specific data. The first and third strategies are proven effective, while the second one, though ineffective, provides valuable insights for further research.
Dysarthric Speech Recognition Using Curriculum Learning and Articulatory Feature Embedding
Authors: I-Ting Hsieh; Chung-Hsien Wu
Venue: Interspeech 2024 (main)

Abstract: Recognizing speech in individuals with articulation disorders is a challenging task due to limited resources and diverse speaker characteristics. Domain adaptation is commonly employed to address these issues, and in this paper, we apply curriculum learning, a method within this approach, to Automatic Speech Recognition (ASR). To enhance the efficiency of curriculum learning, we reorganize the dataset. Additionally, we incorporate speaker and articulatory features to capture the pronunciation characteristics of patients. Experimental results demonstrate that our proposed method achieves an 11.37% improvement compared to the baseline.
Enhancing Dysarthric Speech Recognition for Unseen Speakers via Prototype-Based Adaptation
Authors: Shiyao Wang; Shiwan Zhao; Jiaming Zhou; Aobo Kong; Yong Qin
Venue: Interspeech 2024 (main)

Abstract: Dysarthric speech recognition (DSR) presents a formidable challenge due to inherent inter-speaker variability, leading to severe performance degradation when applying DSR models to new dysarthric speakers. Traditional speaker adaptation methodologies typically involve fine-tuning models for each speaker, but this strategy is cost-prohibitive and inconvenient for disabled users, requiring substantial data collection. To address this issue, we introduce a prototype-based approach that markedly improves DSR performance for unseen dysarthric speakers without additional fine-tuning. Our method employs a feature extractor trained with HuBERT to produce per-word prototypes that encapsulate the characteristics of previously unseen speakers. These prototypes serve as the basis for classification. Additionally, we incorporate supervised contrastive learning to refine feature extraction. By enhancing representation quality, we further improve DSR performance, enabling effective personalized DSR.
An efficient text augmentation approach for contextualized Mandarin speech recognition
Authors: Naijun Zheng; Xucheng Wan; Kai Liu; Ziqing Du; Zhou Huan
Venue: Interspeech 2024 (main)

Abstract: Although contextualized automatic speech recognition (ASR) systems are commonly used to improve the recognition of uncommon words, their effectiveness is hindered by the inherent limitations of speech-text data availability. To address this challenge, our study proposes to leverage extensive text-only datasets and contextualize pre-trained ASR models using a straightforward text-augmentation (TA) technique, all while keeping computational costs minimal. In particular, to contextualize a pre-trained CIF-based ASR, we construct a codebook using limited speech-text data. By utilizing a simple codebook lookup process, we convert available text-only data into latent text embeddings. These embeddings then enhance the inputs for the contextualized ASR. Our experiments on diverse Mandarin test sets demonstrate that our TA approach significantly boosts recognition performance. The top-performing system shows relative CER improvements of up to 30% on rare words and 15% across all words in general.
Investigating ASR Error Correction with Large Language Model and Multilingual 1-best Hypotheses
Authors: Sheng Li; Chen Chen; Chin Yuen Kwok; Chenhui Chu; Eng Siong Chng; Hisashi Kawai
Venue: Interspeech 2024 (main)

Abstract: This paper investigates using pre-trained large language models (LLMs) to improve multilingual automatic speech recognition (ASR) outputs. Current popular methods involve feeding the N-best ASR output into LLMs. Although this approach demonstrates improved results, obtaining N-best hypotheses is time-consuming and unavailable sometimes. To develop a more general method, this paper investigates LLM-based ASR error correction with 1-best hypotheses. We fine-tuned a multilingual LLM covering more than 100 languages and let it correct 1-best hypotheses errors from different speech foundation models. The experiment shows that the proposed method effectively enhances the ASR result only using 1-best hypotheses. Moreover, we also noticed that knowledge-transferring between the languages using the same writing system in the LLM can effectively correct low-resourced languages' hypotheses.
Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clipping
Authors: Lun Wang; Om Thakkar; Zhong Meng; Nicole Rafidi; Rohit Prabhavalkar; Arun Narayanan
Venue: Interspeech 2024 (main)

Abstract: Gradient clipping plays a vital role in training large-scale automatic speech recognition (ASR) models. It is typically applied to minibatch gradients to prevent gradient explosion, and to the individual sample gradients to mitigate unintended memorization. This work systematically investigates the impact of a specific granularity of gradient clipping, namely per-core clipping (PCC), across training a wide range of ASR models. We empirically demonstrate that PCC can effectively mitigate unintended memorization in ASR models. Surprisingly, we find that PCC positively influences ASR performance metrics, leading to improved convergence rates and reduced word error rates. To avoid tuning the additional hyperparameter introduced by PCC, we further propose a novel variant, adaptive per-core clipping (APCC), for streamlined optimization. Our findings highlight the multifaceted benefits of PCC as a strategy for robust, privacy-forward ASR model training.