GenAI4ELab/papercli-papers huggingface dataset:
LASER: Learning by Aligning Self-supervised Representations of Speech for Improving Content-related Tasks
Authors: Amit Meghanani; Thomas Hain
Venue: Interspeech 2024 (main)

Abstract: Self-supervised learning (SSL)-based speech models are extensively used for full-stack speech processing. However, it has been observed that improving SSL-based speech representations using unlabeled speech for content-related tasks is challenging and computationally expensive. Recent attempts have been made to address this issue with cost-effective self-supervised fine-tuning (SSFT) approaches. Continuing in this direction, a cost-effective SSFT method named âLASER: Learning by Aligning Self-supervised Representationsâ is presented. LASER is based on the soft-DTW alignment loss with temporal regularisation term. Experiments are conducted with HuBERT and WavLM models and evaluated on the SUPERB benchmark for two content-related tasks: automatic speech recognition (ASR) and phoneme recognition (PR). A relative improvement of 3.7% and 8.2% for HuBERT, and 4.1% and 11.7% for WavLM are observed, for the ASR and PR tasks respectively, with only < 3 hours of fine-tuning on a single GPU.
Self-Train Before You Transcribe
Authors: Robert Flynn; Anton Ragni
Venue: Interspeech 2024 (main)

Abstract: When there is a mismatch between the training and test domains, current speech recognition systems show significant performance degradation. Self-training methods, such as noisy student teacher training, can help address this and enable the adaptation of models under such domain shifts. However, self-training typically requires a collection of unlabelled target domain data. For settings where this is not practical, we investigate the benefit of performing noisy student teacher training on recordings in the test set as a test-time adaptation approach. Similarly to the dynamic evaluation approach in language modelling, this enables the transfer of information across utterance boundaries and functions as a method of domain adaptation. A range of in-domain and out-of-domain datasets are used for experiments demonstrating large relative gains of up to 32.2%. Interestingly, our method showed larger gains than the typical self-training setup that utilises separate adaptation data.
Unsupervised Online Continual Learning for Automatic Speech Recognition
Authors: Steven Vander Eeckt; Hugo Van hamme
Venue: Interspeech 2024 (main)

Abstract: Adapting Automatic Speech Recognition (ASR) models to new domains leads to Catastrophic Forgetting (CF) of previously learned information. This paper addresses CF in the challenging context of Online Continual Learning (OCL), with tasks presented as a continuous data stream with unknown boundaries. We extend OCL for ASR into the unsupervised realm, by leveraging self-training (ST) to facilitate unsupervised adaptation, enabling models to adapt continually without label dependency and without forgetting previous knowledge. Through comparative analysis of various OCL and ST methods across two domain adaptation experiments, we show that UOCL suffers from significantly less forgetting compared to supervised OCL, allowing UOCL methods to approach the performance levels of supervised OCL. Our proposed UOCL extensions further boosts UOCL's efficacy. Our findings represent a significant step towards continually adaptable ASR systems, capable of leveraging unlabeled data across diverse domains.
Dual-path Adaptation of Pretrained Feature Extraction Module for Robust Automatic Speech Recognition
Authors: Hao Shi; Tatsuya Kawahara
Venue: Interspeech 2024 (main)

Abstract: Self-supervised learning (SSL)-based pretrained models have significantly improved automatic speech recognition (ASR) performance. As the feature extraction (FE) module has also been well-trained with a large amount of training data, freezing the FE during finetuning for downstream ASR tasks is common. When there is a severe mismatch between the simulated noisy data for pretraining and real noisy data, however, finetuning the FE with the real noisy data should be done without losing the effective information of the pretrained FE. In this paper, we propose a dual-path adaptation of the FE to address this problem. It combines the frozen pretrained FE path and the finetuned-adapted FE path with convolutional fusion layers. Moreover, adapters are inserted into the Transformer encoder. The experimental results using the CHiMEâ4 dataset show that using adapters for the FE or the Transformer encoder is effective, but achieving synergy of these two is challenging. Finetuning of the FE combined with adapters in the encoder realizes effective model adaptation. Moreover, the proposed method utilizes the complementarity between the pretrained and the finetuned FE paths, achieving significant improvements even with noise-robust WavLM models.
Hierarchical Multi-Task Learning with CTC and Recursive Operation
Authors: Nahomi Kusunoki; Yosuke Higuchi; Tetsuji Ogawa; Tetsunori Kobayashi
Venue: Interspeech 2024 (main)

Abstract: We present hierarchical recursive CTC (HR-CTC), an effective hierarchical multi-task learning (HMTL) model for end-to-end automatic speech recognition (ASR). HMTL enables a model to learn suitable intermediate representations for predicting high-level and sparse targets (e.g., words). This is achieved by applying auxiliary CTC losses to intermediate layers of the model, which are calculated using lower-level targets (e.g., phonemes or smaller subwords). In this work, we propose to enhance the hierarchical generation capability in HMTL by designing a recursive structure that iteratively uses the same model layers to refine intermediate predictions. These improved predictions are used to explicitly condition the deeper model layers, thereby facilitating more accurate predictions at the higher level. Experimental results show that HR-CTC outperforms conventional HMTL models across various ASR tasks, providing an additional benefit of balancing accuracy and inference speed.
Boosting CTC-based ASR using inter-layer attention-based CTC loss
Authors: Keigo Hojo; Yukoh Wakabayashi; Kengo Ohta; Atsunori Ogawa; Norihide Kitaoka
Venue: Interspeech 2024 (main)

Abstract: This paper addresses improving the performance of CTC-based models, which leverage the intermediate outputs of all encoder layers with an attention mechanism. Several previous studies have used the intermediate outputs of the encoder layer to modify CTC-based models. Here, we focus on the role of the Transformer encoder layer, and each encoder layer is computed for two CTC losses by weighting the intermediate outputs of its lower and upper layers using an attention mechanism. By dividing the layer into two groups, it is expected to be possible to calculate the loss, taking into account both acoustic and linguistic features. Experimental results showed that the proposed method improved the baseline recognition performance of TEDLIUM2 speech data, achieving a WER of 9.9% on the dev set and 11.8% on the test set. Our method outperformed the conventional methods for WER with only slightly increased inference speed measured by RTF.
Self-training ASR Guided by Unsupervised ASR Teacher
Authors: Hyung Yong Kim; Byeong-Yeol Kim; Yunkyu Lim; Jihwan Park; Shukjae Choi; Yooncheol Ju; Jinseok Park; Youshin Lim; Seung Woo Yu; Hanbin Lee; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: Self-training has gained increasing attention due to its notable performance improvement in speech recognition. However, conventional self-training techniques have two key limitations: (1) labeled dataset is required for training a teacher to produce a pseudo-target, and (2) the first teacher trained with the small labeled dataset faces an over-fitting issue, generating noisy pseudo-targets for unseen datasets. Our approach adopts an unsupervised automatic speech recognition model as the teacher, thus solely utilizing the unlabeled dataset. As the proposed model also learns phonetic information from the UASR teacher at the intermediate layer, the pseudo-target at the higher layer contains more ASR-related information than that of Data2vec2. Experimental results on the LibriSpeech show that our model outperforms Data2vec2, the state-of-the-art self-supervised learning model, achieving 8.9% and 4.3% relative word error rate reduction on test-clean and test-other.
Personality-memory Gated Adaptation: An Efficient Speaker Adaptation for Personalized End-to-end Automatic Speech Recognition
Authors: Yue Gu; Zhihao Du; Shiliang Zhang; jiqing Han; Yongjun He
Venue: Interspeech 2024 (main)

Abstract: In real-world applications, a preferred personalized automatic speech recognition (ASR) model should exhibit robust recognition capabilities that include both personalization and generalization. Speaker adaptation is a commonly used approach towards personalized ASR. However, most speaker adaptation methods only focus on improving the performance of the target speaker but neglect or even sacrifice generalization. In this paper, we propose the Personality-memory Gated Adaptation (PGA) approach to adapt models on target speakers while preserving generalized recognition performance. Specifically, we incorporate parallel adapters into the encoder to capture the target speakerâs vocal characteristics. The hidden outputs of the adapters and original encoder layers are fused via a scalar gate, which is derived from the similarity between input samples and personality memory units, i.e., personality embeddings from the target speaker. In this manner, the vocal characteristics of the target speaker are memorized by adapters while the ASR backbone preserves generalization. Experimental results show that, compared to the unadapted ASR backbone, the proposed PGA achieves 21.62% relative character error rate (CER) reduction on the target domain but a negligible CER increase on the source domain with limited adaptation data and training steps.
Speaker Personalization for Automatic Speech Recognition using Weight-Decomposed Low-Rank Adaptation
Authors: George Joseph; Arun Baby
Venue: Interspeech 2024 (main)

Abstract: Personalizing automated speech recognition (ASR) for voice assistant systems is often considered the holy grail, requiring meticulous attention to detail in model optimization. When dealing with limited speaker data, the selection of hyper-parameters becomes paramount in fine-tuning large ASR models. One effective method for this optimization is low-rank adaptation (LoRA), which proves instrumental in enhancing the performance of large language models (LLMs). A variation of LoRA, Weight-Decomposed Low-Rank Adaptation (DoRA) also promises enhanced performance. In our study, we employed LoRA and DoRA, to refine the state-of-the-art cascaded conformer transducer model for speaker personalization. This involved adding a small number of speaker-specific weights to the existing model and fine-tuning them accordingly. Experimental assessments show an average relative improvement of 20% in word error rate across speakers with limited data, showcasing its efficacy in addressing the challenge of personalizing ASR systems in real-world applications.
Online Subloop Search via Uncertainty Quantization for Efficient Test-Time Adaptation
Authors: Jae-Hong Lee; Sang-Eon Lee; Dong-Hyun Kim; DoHee Kim; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Online test-time adaptation (OTTA) methods have demonstrated their effectiveness in real-time adapting to the target domain for speech recognition tasks. However, a common thread among these existing methods is their reliance on repetitive learning for each test utterance through a subloop, imposing prohibitive computational costs. This paper highlights the inefficiency inherent in applying a uniform number of subloop iterations to every test sample. To address this issue, we propose the online subloop search (OSS) method, which implicitly adjusts the number of iterations based on the test sample and domain characteristics. The proposed method operates within a framework comprising a chaser model updated via stochastic gradient descent and a leader model updated through the exponential moving average. The OSS method quantifies and quantizes the uncertainty in the chaser model relative to the leader model, using the quantized value to predict the number of iterations for the subloop.
ROAR: Reinforcing Original to Augmented Data Ratio Dynamics for Wav2vec2.0 Based ASR
Authors: Vishwanath Pratap Singh; Federico Malato; Ville HautamÃ¤ki; Md. Sahidullah; Tomi Kinnunen
Venue: Interspeech 2024 (main)

Abstract: While automatic speech recognition (ASR) greatly benefits 
from data augmentation, the augmentation recipes themselves
 tend to be heuristic. In this paper, we address one of
the heuristic approach associated with balancing the right
 amount of augmented data in ASR training by introducing
a reinforcement learning (RL) based dynamic adjustment
of original-to-augmented data ratio (OAR). Unlike
the fixed OAR approach in conventional data augmentation,
 our proposed method employs a deep Q-network (DQN) as
the RL mechanism to learn the optimal dynamics of OAR 
throughout the wav2vec2.0 based ASR training. We conduct
 experiments using the LibriSpeech dataset with varying 
amounts of training data, specifically, the 10Min, 1H,
10H, and 100H splits to evaluate the efficacy of the proposed
 method under different data conditions. Our proposed
 method, on average, achieves a relative improvement
of 4.96% over the open-source wav2vec2.0 base model on
standard LibriSpeech test sets.
Online Knowledge Distillation of Decoder-Only Large Language Models for Efficient Speech Recognition
Authors: Jeehye Lee; Hyeji Seo
Venue: Interspeech 2024 (main)

Abstract: Large language models (LLMs), which show promising performance in generation tasks, have proven their capabilities to be applied in a wide range of tasks. Although there are several approaches to adapt LLMs as decoder in speech recognition tasks, these can slow down inference speed, which is an important issue for the product-level systems. To address this problem, we introduce online knowledge distillation methods to transfer information from the decoder-only LLMs to a more compact Transformer decoder during the training phase. Implementing our proposed methods on a multilingual low-resource dataset, we achieved a 8.2% relative character error rate (CER) reduction compared to the LLM decoder model with much lower inference cost and a 34.7% relative CER reduction compared to the attention-based encoder-decoder (AED) model. Furthermore, we obtained a 14.9% relative CER reduction along with the same inference cost on a general Korean dataset.
Efficient and Robust Long-Form Speech Recognition with Hybrid H3-Conformer
Authors: Tomoki Honda; Shinsuke Sakai; Tatsuya Kawahara
Venue: Interspeech 2024 (main)

Abstract: Recently, Conformer has achieved state-of-the-art performance in many speech recognition tasks. However, the Transformer-based models show significant deterioration for long-form speech, such as lectures, because the self-attention mechanism becomes unreliable with the computation of the square order of the input length. To solve the problem, we incorporate a kind of state-space model, Hungry Hungry Hippos (H3), to replace or complement the multi-head self-attention (MHSA). H3 allows for efficient modeling of long-form sequences with a linear-order computation. In experiments using two datasets of CSJ and LibriSpeech, our proposed H3-Conformer model performs efficient and robust recognition of long-form speech. Moreover, we propose a hybrid of H3 and MHSA and show that using H3 in higher layers and MHSA in lower layers provides significant improvement in online recognition. We also investigate a parallel use of H3 and MHSA in all layers, resulting in the best performance.
Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Prompting
Authors: Yosuke Kashiwagi; Hayato Futami; Emiru Tsunoo; Siddhant Arora; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: End-to-end multilingual speech recognition models handle multiple languages through a single model, often incorporating language identification to automatically detect the language of incoming speech. Since the common scenario is where the language is already known, these models can perform as language-specific by using language information as prompts, which is particularly beneficial for attention-based encoder-decoder architectures. However, the Connectionist Temporal Classification (CTC) approach, which enhances recognition via joint decoding and multi-task training, does not normally incorporate language prompts due to its conditionally independent output tokens. To overcome this, we introduce an encoder prompting technique within the self-conditioned CTC framework, enabling language-specific adaptation of the CTC model in a zero-shot manner. Our method has shown to significantly reduce errors by 28% on average and by 41% on low-resource languages.
Quantifying Unintended Memorization in BEST-RQ ASR Encoders
Authors: Virat Shejwalkar; Om Thakkar; Arun Narayanan
Venue: Interspeech 2024 (main)

Abstract: Self-supervised ASR encoders are increasingly being adopted in real-world applications as they enable downstream ASR tasks with impressive performances. This raises concerns around privacy of the data used to train such encoders, especially since neural networks are known to unintentionally memorize rare/unique samples from their training data. To this end, we perform the first systematic auditing of unintended memorization in ASR encoders. Specifically, we focus on a state-of-the-art Conformer-based ASR encoder pre-trained using the BEST-RQ technique, which forms the foundation of many real-world ASR applications. We propose a novel auditing method that can successfully demonstrate such memorization in ASR encoders, even for  samples occurring just once in their training data. Finally, we show the promise of pre-training with per-sample gradient clipping towards mitigating such memorization in ASR encoders without significantly impacting downstream model quality.
SWAN: SubWord Alignment Network for HMM-free word timing estimation in end-to-end automatic speech recognition
Authors: Woo Hyun Kang; Srikanth Vishnubhotla; Rudolf Braun; Yogesh Virkar; Raghuveer Peri; Kyu J. Han
Venue: Interspeech 2024 (main)

Abstract: End-to-end (E2E) automatic speech recognition (ASR) systems often exploited pre-trained hidden Markov model (HMM) systems for word timing estimation (WTE), due to their inability to predict word boundaries. However, training an HMM is difficult for low-resource languages due to the lack of phonetic transcriptions, leading to a high demand for HMM-free WTE methods, particularly for multilingual ASR systems. In this paper, we propose a novel framework for performing WTE without the need for any HMM or phonetic labels. Specifically, the proposed method trains an alignment network using the outputs of the E2E ASR encoder and a voice activity detection module to generate the frame-level subword labels. In our experiments, the proposed method outperforms previous HMM-free WTE methods in a multilingual scenario. Notably, in the Fleurs dataset, we obtain a relative improvement of 57% over previous work in terms of accumulated averaging shift across 5 languages.
Spontaneous Speech-Based Suicide Risk Detection Using Whisper and Large Language Models
Authors: Ziyun Cui; Chang Lei; Wen Wu; Yinan Duan; Diyang Qu; Ji Wu; Runsen Chen; Chao Zhang
Venue: Interspeech 2024 (main)

Abstract: The early detection of suicide risk is important since it enables the intervention to prevent potential suicide attempts. This paper studies the automatic detection of suicide risk based on spontaneous speech from adolescents, and collects a Mandarin dataset with 15 hours of suicide speech from more than a thousand adolescents aged from ten to eighteen for our experiments. To leverage the diverse acoustic and linguistic features embedded in spontaneous speech, both the Whisper speech model and textual large language models (LLMs) are used for suicide risk detection. Both all-parameter finetuning and parameter-efficient finetuning approaches are used to adapt the pre-trained models for suicide risk detection, and multiple audio-text fusion approaches are evaluated to combine the representations of Whisper and the LLM. The proposed system achieves a detection accuracy of 0.807 and an F1-score of 0.846 on the test set with 119 subjects, indicating promising potential for real suicide risk detection applications.
Spoken Word2Vec: Learning Skipgram Embeddings from Speech
Authors: Mohammad Amaan Sayeed; Hanan Aldarmaki
Venue: Interspeech 2024 (main)

Abstract: Text word embeddings that encode distributional semantics work by modeling contextual similarities of frequently occurring words. Acoustic word embeddings, on the other hand, typically encode low-level phonetic similarities. Semantic embeddings for spoken words have been previously explored using analogous algorithms to Word2Vec, but the resulting vectors still mainly encoded phonetic rather than semantic features. In this paper, we examine the assumptions and architectures used in previous works and show experimentally how shallow skipgram-like algorithms fail to encode distributional semantics when the input units are acoustically correlated. We illustrate the potential of an alternative deep end-to-end variant of the model and examine the effects on the resulting  embeddings, showing positive results of semantic relatedness in the embedding space.
SAMSEMO: New dataset for multilingual and multimodal emotion recognition
Authors: Pawel Bujnowski; Bartlomiej Kuzma; Bartlomiej Paziewski; Jacek Rutkowski; Joanna Marhula; Zuzanna Bordzicka; Piotr Andruszkiewicz
Venue: Interspeech 2024 (main)

Abstract: The task of emotion recognition using image, audio and text modalities has recently attained popularity due to its various potential applications. However, the list of large-scale multimodal datasets is very short and all available datasets have significant limitations. We present SAMSEMO, a novel dataset for multimodal and multilingual emotion recognition. Our collection of over 23k video scenes is multilingual as it includes video scenes in 5 languages (EN, DE, ES, PL and KO). Video scenes are heterogeneous, they come from diverse sources and are accompanied with rich manually collected metadata and emotion annotations. In the paper, we also study the valence and arousal of audio features of our data for the most important emotion classes and compare them with the features of CMU-MOSEI data. Moreover, we perform multimodal experiments for emotion recognition with SAMSEMO and show how to use a multilingual model to improve the detection of imbalanced classes.
LLM-Driven Multimodal Opinion Expression Identification
Authors: Bonian Jia; Huiyao Chen; Yueheng Sun; Meishan Zhang; Min Zhang
Venue: Interspeech 2024 (main)

Abstract: Opinion Expression Identification (OEI) is essential in NLP for applications ranging from voice assistants to depression diagnosis. This study extends OEI to encompass multimodal inputs, underlining the significance of auditory cues in delivering emotional subtleties beyond the capabilities of text. We introduce a novel multimodal OEI (MOEI) task, integrating text and speech to mirror real-world scenarios. Utilizing CMU MOSEI and IEMOCAP datasets, we construct the CI-MOEI dataset. Additionally, Text-to-Speech (TTS) technology is applied to the MPQA dataset to obtain the CIM-OEI dataset. We design a template for the OEI task to take full advantage of the generative power of large language models (LLMs). Advancing further, we propose an LLM-driven method STOEI, which combines speech and text modal to identify opinion expressions. Our experiments demonstrate that MOEI significantly improves the performance while our method outperforms existing methods by 9.20% and obtains SOTA results.