GenAI4ELab/papercli-papers huggingface dataset:
LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems
Authors: Tahir Javed; Janki Nawale; Sakshi Joshi; Eldho George; Kaushal Bhogale; Deovrat Mehendale; Mitesh M. Khapra
Venue: Interspeech 2024 (main)

Abstract: Hindi, one of the most spoken language of India, exhibits a diverse array of accents due to its usage among individuals from diverse linguistic origins. To enable a robust evaluation of Hindi ASR systems on multiple accents, we create a benchmark, LAHAJA, which contains read and extempore speech on a diverse set of topics and use cases, with a total of 12.5 hours of Hindi audio, sourced from 132 speakers spanning 83 districts of India. We evaluate existing open-source and commercial models on LAHAJA and find their performance to be poor. We then train models using different datasets and find that our model trained on multilingual data with good speaker diversity outperforms existing models by a significant margin. We also present a fine-grained analysis which shows that the performance declines for speakers from North-East and South India, especially with content heavy in named entities and specialized terminology.
LearnerVoice: A Dataset of Non-Native English Learnersâ Spontaneous Speech
Authors: Haechan Kim; Junho Myung; Seoyoung Kim; Sungpah Lee; Dongyeop Kang; Juho Kim
Venue: Interspeech 2024 (main)

Abstract: Prevalent ungrammatical expressions and disfluencies in spontaneous speech from second language (L2) learners pose unique challenges to Automatic Speech Recognition (ASR) systems. However, few datasets are tailored to L2 learner speech. We publicly release LearnerVoice, a dataset consisting of 50.04 hours of audio and transcriptions of L2 learnersâ spontaneous speech. Our linguistic analysis reveals that transcriptions in our dataset contain L2S (L2 learnerâs Spontaneous speech) features, consisting of ungrammatical expressions and disfluencies (e.g., filler words, word repetitions, self-repairs, false starts), significantly more than native speech datasets. Fine-tuning whisper-small.en with LearnerVoice achieves a WER of 10.26%, 44.2% lower than vanilla whisper-small.en. Furthermore, our qualitative analysis indicates that 54.2% of errors from the vanilla model on LearnerVoice are attributable to L2S features, with 48.1% of them being reduced in the fine-tuned model.
MinSpeech: A Corpus of Southern Min Dialect for Automatic Speech Recognition
Authors: Jiayan Lin; Shenghui Lu; Hukai Huang; Wenhao Guan; Binbin Xu; Hui Bu; Qingyang Hong; Lin Li
Venue: Interspeech 2024 (main)

Abstract: This paper presents MinSpeech, a speech corpus of Southern Min (also known as Hokkien), to propel research in dialect speech recognition. Despite the linguistic and cultural importance of Southern Min, there is still a notable scarcity of publicly accessible speech corpus for this dialect. MinSpeech provides 2237 hours of unlabeled audio and 1778 hours of labeled audio, sourced diversely and encompassing various contexts. Mandarin text is employed as labels to enable cross-linguistic alignment and transformation. Using this corpus, we have developed baseline systems, including supervised models (Kaldi Chain and Conformer) and two self-supervised models (Wav2vec 2.0 and HuBERT). These systems were assessed on an automatic speech recognition (ASR) task to the Southern Min dialect. Experiments illustrate that the corpus offers practical assistance and resources for speech processing of this dialect. MinSpeech dataset is available at https://minspeech.github.io/.
Cross-modal Features Interaction-and-Aggregation Network with Self-consistency Training for Speech Emotion Recognition
Authors: Ying Hu; Huamin Yang; Hao Huang; Liang He
Venue: Interspeech 2024 (main)

Abstract: In recent years, much research has been into speech emotion recognition (SER) using multimodal data. Selective fusion of the features from different modalities is critical for  multimodal SER. In this paper, we propose a cross-modal features interaction-and-aggregation network (CFIA-Net) with self-consistency training for SER. Specifically, we design a cross-modal features interaction-and-aggregation (CFIA) module to adaptively interact and integrate the features of audio and text modalities. Moreover, we introduce a self-consistency training strategy, which exploits the features from deeper layers to supervise those from shallower ones to obtain the SER task-related information. The experimental results show that compared with other bimodal SER methods, the CFIA-Net achieves the state-of-the-art performance on the weighted accuracy (WA) of 83.37%  and unweighted accuracy (UA) of 83.67% on the IEMOCAP dataset.
Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning
Authors: Arnav Goel; Medha Hira; Anubha Gupta
Venue: Interspeech 2024 (main)

Abstract: Advent of modern deep learning techniques has given rise to advancements in the field of Speech Emotion Recognition (SER). However, most systems prevalent in the field fail to generalize to speakers not seen during training. This study focuses on handling challenges of multilingual SER, specifically on unseen speakers. We introduce CAMuLeNet, a novel architecture leveraging co-attention based fusion and multitask learning to address this problem. Additionally, we benchmark pretrained encoders of Whisper, HuBERT, Wav2Vec2.0, and WavLM using 10-fold leave-speaker-out cross-validation on five existing multilingual benchmark datasets: IEMOCAP, RAVDESS, CREMA-D, EmoDB and CaFE and, release a novel dataset for SER on the Hindi language (BhavVani). CAMuLeNet shows an average improvement of approximately 8% over all benchmarks on unseen speakers determined by our cross-validation strategy.
SELM: Enhancing Speech Emotion Recognition for Out-of-Domain Scenarios
Authors: Hazim Bukhari; Soham Deshmukh; Hira Dhamyal; Bhiksha Raj; Rita Singh
Venue: Interspeech 2024 (main)

Abstract: Speech Emotion Recognition (SER) has been traditionally formulated as a classification task. However, emotions are generally a spectrum whose distribution varies from situation to situation leading to poor Out-of-Domain (OOD) performance. We take inspiration from statistical formulation of Automatic Speech Recognition (ASR) and formulate the SER task as generating the most likely sequence of text tokens to infer emotion. The formulation breaks SER into predicting acoustic model features weighted by language model prediction. As an instance of this approach, we present SELM, an audio-conditioned language model for SER that predicts different emotion views. We train SELM on curated speech emotion corpus and test it on three OOD datasets (RAVDESS, CREMAD, IEMOCAP) not used in training. SELM achieves significant improvements over the state-of-the-art baselines, with 17\% and 7\% relative accuracy gains for RAVDESS and CREMA-D, respectively. Moreover, SELM can further boost its performance by FewShot Learning using a few annotated examples. The results highlight the effectiveness of our SER formulation, especially to improve performance in OOD scenarios.
The Processing of Stress in End-to-End Automatic Speech Recognition Models
Authors: Martijn Bentum; Louis ten Bosch; Tom Lentz
Venue: Interspeech 2024 (main)

Abstract: Listeners use stress to facilitate word recognition and speech segmentation. Classical ASR systems did not incorporate stress in their recognition process. In contrast, end-to-end ASR systems may use the information carried by stress. The present study shows that Wav2vec 2.0 is indeed sensitive to stress, and that this sensitivity is not a mere reflection of acoustic correlates of stress. Diagnostic classifiers of the CNN output reveal vowel-specific stress representations, that perform on par with acoustic features.  Stress classifiers trained on transformer layers outperform classifiers based on acoustic correlates, but degrade when context is removed, showing that higher layers take the relative nature of stress into account. Results obtained by testing a stress classifier on a vowel it is not trained on, show that stress processing is to some extent abstract, i.e., the classifier does not simply detect a set of stressed vowel representations but rather, their common denominator.
LingWav2Vec2: Linguistic-augmented wav2vec 2.0 for Vietnamese Mispronunciation Detection
Authors: Tuan Nguyen; Huy Dat Tran
Venue: Interspeech 2024 (main)

Abstract: Pronunciation error detection algorithms rely on both acoustic and linguistic information to identify errors. However, these algorithms face challenges due to limited training data, often just a few hours, insufficient for building robust phoneme recognition models. This has led to the adoption of self-supervised learning models like wav2vec 2.0. We propose an innovative approach that combine canonical text and audio inputs to balance the trade-off between accurate phoneme recognition performance and pronunciation scoring. This is done by feeding audio-encoded and normalized canonical phoneme embedding into a linguistic encoder including multi-head attention (MHA) layer and specifically designed feed forward module (FFN). Our system, with only 4.3 million parameters on top of pretrained wav2vec 2.0, achieved top-1 performance at the VLSP Vietnamese Mispronunciation Detection 2023 challenge with 9.72% relative improvement of F1 score over the previous state-of-the-art.
Learning from memory-based models
Authors: Rhiannon Mogridge; Anton Ragni
Venue: Interspeech 2024 (main)

Abstract: Recent work on the CPC2 speech intelligibility task shows promising results using an architecture inspired by memory models from the field of human psychology. This is surprising, given that previous work has shown memory models of this type to be inferior to parametric models, such as transformers, in most modern applications. This paper shows that the difference in performance is reduced or eliminated by using high quality features. Furthermore, we show for the first time that, despite being widely used in the field of human psychology and also for speech and language tasks, this model is a special case of a neural network. Experimental results from different tasks and datasets (CPC2/TIMIT/GoEmotions) confirm that this type of memory model is competitive with equivalently complex parametric models given sufficiently good feature representation, suggesting that high quality features may allow the use of simple, interpretable models without sacrificing performance.
Towards End-to-End Unified Recognition for Mandarin and Cantonese
Authors: Meiling Chen; Pengjie Liu; Heng Yang; Haofeng Wang
Venue: Interspeech 2024 (main)

Abstract: Constructing an automatic speech recognition (ASR) system that supports Mandarin and Cantonese is demanding and challenging. The method of pre-training two speech recognition models and then selecting a specific model for recognition through extra means is resource-consuming and complex. This paper presents an end-to-end system for unified Mandarin-Cantonese recognition and a complete model training method in scenarios where high-resource and low-resource languages coexist, while reducing complexity. The impact of different modeling units on character error rate (CER) and training efficiency was also studied. Besides, this system incorporates a language identification (LID) module to reduce context confusion during recognition. Experiments show that compared to Mandarin-only and Cantonese-only models, our system achieves 12.71% and 21.23% relative CER reduction for Mandarin and Cantonese respectively and training efficiency can be doubled.
Shared-Adapters: A Novel Transformer-based Parameter Efficient Transfer Learning Approach For Childrenâs Automatic Speech Recognition
Authors: Thomas Rolland; Alberto Abad
Venue: Interspeech 2024 (main)

Abstract: Automatic Speech Recognition (ASR) often faces challenges in processing children's speech due to data scarcity. Training large ASR models becomes particularly challenging in such scenarios. To mitigate this issue, fine-tuning is commonly employed, leveraging pre-trained adult models. However, fine-tuning large pre-trained models with limited data poses its own challenges. In response, this study investigates Parameter-Efficient Finetuning (PEFT) for childrenâs ASR. Various PEFT approaches are explored, with a specific emphasis on good ASR performance while minimising the number of parameters during training. Our investigation identifies residual Adapters as the most efficient technique.  Moreover, motivated by Transformer-based model redundancies, we propose the Shared-Adapter and its highly parameter-efficient variant, the Light Shared-Adapter. Our findings demonstrate that Shared-Adapters strike an exceptional balance between recognition performance and parameter efficiency.
AdaRA: Adaptive Rank Allocation of Residual Adapters for Speech Foundation Model
Authors: Zhouyuan Huo; Dongseong Hwang; Gan Song; Khe Chai Sim; Weiran Wang
Venue: Interspeech 2024 (main)

Abstract: A recent paradigm shift in artificial intelligence has witnessed the emergence of foundation models. These foundation models, possessing billions of parameters and trained on extensive datasets, are anticipated to demonstrate superior generalization across diverse downstream tasks. Residual Adapters represent a broadly employed methodology for efficient adaptation, achieved by updating a limited set of additive parameters while maintaining a fixed bottleneck dimension. However, when the parameter budget is constrained, allocating additive parameters uniformly across layers proves sub-optimal. In this paper, we propose a novel adaptive efficient adaptation method that automatically determines the optimal number of bottleneck dimensions for Residual Adapters at different layers. Experimental results confirm that the proposed method effectively learns an optimal additive parameter allocation, surpassing the performance of comparable methods in speech recognition domain adaptation.
Leveraging Adapter for Parameter-Efficient ASR Encoder
Authors: Kyuhong Shim; Jinkyu Lee; Hyunjae Kim
Venue: Interspeech 2024 (main)

Abstract: The expansion of speech models emphasizes the importance of parameter efficiency in practical automatic speech recognition (ASR) systems. Parameter sharing, which reuses the same parameter multiple times, has emerged as a promising solution to reduce storage requirements. However, previous studies have often faced challenges in balancing the number of parameters with performance. In this paper, we propose a novel architecture that effectively reduces the number of parameters while minimizing performance degradation. The key idea is to insert a lightweight adapter module that adjusts the features generated by shared parameters, thereby enhancing the diversity of representations. We introduce a unique adapter module and parameter-sharing configuration tailored for Conformer-based ASR encoders. Experimental results demonstrate that the proposed architecture reduces approximately 50% of parameters and 20% of computations without compromising speech recognition performance.
Whisper Multilingual Downstream Task Tuning Using Task Vectors
Authors: Ji-Hun Kang; Jae-Hong Lee; Mun-Hak Lee; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Recently, the size of automatic speech recognition (ASR) models has been increasing, similar to large language models (LLMs), and efficient tuning to enhance the performance of downstream tasks with limited resources remains a challenge. In this paper, we propose a simple and effective downstream task tuning method using task vectors. We utilize task vectors to orient the pre-trained Whisper model in the weight space, moving in that direction to achieve downstream task adaptation. We demonstrate that the model can be adjusted through arithmetic operations of the task vector, and this adjustment is reflected in the Whisper. Furthermore, we can efficiently construct a generalized model by summing vectors. We set the direction of the model weight space for each multilingual language as the task vector to evaluate its effectiveness. We confirm that the task vector serves as a simple and effective approach for tuning downstream tasks in ASR using the Common Voice multilingual dataset.
Speaker-Smoothed kNN Speaker Adaptation for End-to-End ASR
Authors: Shaojun Li; Daimeng Wei; Hengchao Shang; Jiaxin Guo; ZongYao Li; Zhanglin Wu; Zhiqiang Rao; Yuanchang Luo; Xianghui He; Hao Yang
Venue: Interspeech 2024 (main)

Abstract: Despite recent improvements in End-to-End Automatic Speech Recognition (E2E ASR) systems, the performance can degrade due to vocal characteristic mismatches between training and testing data, particularly with limited target speaker adaptation data. We propose a novel speaker adaptation approach Speaker-Smoothed kNN that leverages k-Nearest Neighbors (kNN) retrieval techniques to improve model output by finding correctly pronounced tokens from its pre-built datastore during the decoding phase. Moreover, we utilize x-vector to dynamically adjust kNN interpolation parameters for data sparsity issue. This approach was validated using KeSpeech and MagicData corpora under in-domain and all-domain settings. Our method consistently performs comparably to fine-tuning without the associated performance degradation during speaker changes. Furthermore, in the all-domain setting, our method achieves state-of-the-art results, reducing the CER in both single speaker and multi-speaker test scenarios.
Qifusion-Net: Layer-adapted Stream/Non-stream Model for End-to-End Multi-Accent Speech Recognition
Authors: Jinming Chen; Jingyi Fang; Yuanzhong Zheng; Yaoxuan Wang; Haojun Fei
Venue: Interspeech 2024 (main)

Abstract: Currently, end-to-end (E2E) speech recognition methods have achieved promising performance. However, auto speech recognition (ASR) models still face challenges in recognizing multi-accent speech accurately. We propose a layer-adapted fusion (LAF) model, called Qifusion-Net, which does not require any prior knowledge about the target accent. Based on dynamic chunk strategy, our approach enables streaming decoding and can extract frame-level acoustic feature, facilitating fine-grained information fusion. Experiment results demonstrate that our proposed methods outperform the baseline with relative reductions of 22.1% and  17.2% in character error rate (CER) across multi accent test datasets on KeSpeech and MagicData-RMAC.
HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition
Authors: Ji Won Yoon; Beom Jun Woo; Nam Soo Kim
Venue: Interspeech 2024 (main)

Abstract: Pre-training with self-supervised models, such as Hidden-unit BERT (HuBERT) and wav2vec 2.0, has brought significant improvements in automatic speech recognition (ASR). However, these models usually require an expensive computational cost to achieve outstanding performance, slowing down the inference speed. To improve the model efficiency, we introduce an early exit scheme for ASR, namely HuBERT-EE, that allows the model to stop the inference dynamically. In HuBERT-EE, multiple early exit branches are added at the intermediate layers. When the intermediate prediction of the early exit branch is confident, the model stops the inference, and the corresponding result can be returned early. We investigate the proper early exiting criterion and fine-tuning strategy to effectively perform early exiting. Experimental results on the LibriSpeech show that HuBERT-EE can accelerate the inference of the HuBERT while simultaneously balancing the trade-off between the performance and the latency.
MaLa-ASR: Multimedia-Assisted LLM-Based ASR
Authors: Guanrou Yang; Ziyang Ma; Fan Yu; Zhifu Gao; Shiliang Zhang; Xie Chen
Venue: Interspeech 2024 (main)

Abstract: As more and more information-rich data like video become available, utilizing multi-modal auxiliary information to enhance audio tasks has sparked widespread research interest. The recent surge in research on LLM-based audio models provides fresh perspectives for tackling audio tasks. Given that LLM can flexibly ingest multiple inputs, we propose MaLa-ASR, an LLM-based ASR model that can integrate textual keywords extracted from presentation slides to improve recognition of conference content. MaLa-ASR yields average WERs of 9.4% and 11.7% on the L95 and S95 subsets of the SlideSpeech corpus, representing a significant relative WER drop of 27.9% and 44.7% over the baseline model reported in SlideSpeech. MaLa-ASR underscores LLM's strong performance in speech tasks and the capability to integrate auxiliary information conveniently. By adding keywords to the input prompt, the biased word error rate (B-WER) reduces relatively by 46.0% and 44.2%, establishing a new SOTA on this dataset.
Spoken-to-written text conversion with Large Language Model
Authors: HyunJung Choi; Muyeol Choi; Yohan Lim; Minkyu Lee; Seonhui Kim; Seung Yun; Donghyun Kim; SangHun Kim
Venue: Interspeech 2024 (main)

Abstract: The improvement in end-to-end speech recognition systems has enhanced the readability of results, making it easier for users to understand texts and reducing translation errors. Korean uses both written and spoken forms, making it crucial to standardize pronunciation notation for high readability. Inverse Text Normalization (ITN) technology, which converts pronunciation into readable written form, can be applied in preprocessing training corpora or post-processing speech recognition outcomes. Recent Korean ITN research utilizes transformer models based on training data with both notations, facing performance degradation due to data scarcity. This paper proposes using Large Language Models for ITN to address this issue, overcoming the performance decline from limited data. The proposed method showed an 12.6% reduction in Error Reduction Rate (ERR).
MM-KWS: Multi-modal Prompts for Multilingual User-defined Keyword Spotting
Authors: Zhiqi Ai; Zhiyong Chen; Shugong Xu
Venue: Interspeech 2024 (main)

Abstract: In this paper, we propose MM-KWS, a novel approach to user-defined keyword spotting leveraging multi-modal enrollments of text and speech templates. Unlike previous methods that focus solely on either text or speech features, MM-KWS extracts phoneme, text, and speech embeddings from both modalities. These embeddings are then compared with the query speech embedding to detect the target keywords. To ensure the applicability of MM-KWS across diverse languages, we utilize a feature extractor incorporating several multilingual pre-trained models. Subsequently, we validate its effectiveness on Mandarin and English tasks. In addition, we have integrated advanced data augmentation tools for hard case mining to enhance MM-KWS in distinguishing confusable words. Experimental results on the LibriPhrase and WenetPhrase datasets demonstrate that MM-KWS outperforms prior methods significantly.
Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation
Authors: Andrew Rouditchenko; Yuan Gong; Samuel Thomas; Leonid Karlinsky; Hilde Kuehne; Rogerio Feris; James Glass
Venue: Interspeech 2024 (main)

Abstract: Audio-Visual Speech Recognition (AVSR) uses lip-based video to improve performance in noise. Since videos are harder to obtain than audio, the video training data of AVSR models is usually limited to a few thousand hours. In contrast, speech models such as Whisper are trained with hundreds of thousands of hours of data, and thus learn a better speech-to-text decoder. The huge training data difference motivates us to adapt Whisper to handle video inputs. Inspired by Flamingo which injects visual features into language models, we propose Whisper-Flamingo which integrates visual features into the Whisper speech recognition and translation model with gated cross attention. Our audio-visual Whisper-Flamingo outperforms audio-only Whisper on English speech recognition and En-X translation for 6 languages in noisy conditions. Moreover, Whisper-Flamingo is a versatile model and conducts all of these tasks using one set of parameters, while prior methods are trained separately on each language.