GenAI4ELab/papercli-papers huggingface dataset:
Less is More: Accurate Speech Recognition & Translation without Web-Scale Data
Authors: Krishna C. Puvvada; Piotr Å»elasko; He Huang; Oleksii Hrinchuk; Nithin Rao Koluguri; Kunal Dhawan; Somshubra Majumdar; Elena Rastorgueva; Zhehuai Chen; Vitaly Lavrukhin; Jagadeesh Balam; Boris Ginsburg
Venue: Interspeech 2024 (main)

Abstract: Recent advances in speech recognition and translation rely on hundreds of thousands of hours of Internet speech data. We argue that state-of-the art accuracy can be reached without relying on web-scale data. Canary - multilingual ASR and speech translation model, outperforms current state-of-the-art models â Whisper, OWSM, and Seamless-M4T on English, French, Spanish, and German languages, while being trained on an order of magnitude less data than these models. Three key factors enables such dataefficient model: (1) a FastConformer-based attention encoderdecoder architecture (2) training on synthetic data generated with machine translation and (3) advanced training techniques: data-balancing, dynamic data blending, dynamic bucketing and noise-robust fine-tuning. The model, weights, and training code will be open-sourced.
The Greek podcast corpus: Competitive speech models for low-resourced languages with weakly supervised data
Authors: Georgios Paraskevopoulos; Chara Tsoukala; Athanasios Katsamanis; Vassilis Katsouros
Venue: Interspeech 2024 (main)

Abstract: The development of speech technologies for languages with limited digital representation poses significant challenges, pri- marily due to the scarcity of available data. This issue is exacerbated in the era of large, data-intensive models. Recent research has underscored the potential of leveraging weak supervision to augment the pool of available data. In this study, we compile an 800-hour corpus of Modern Greek from podcasts and employ Whisper large-v3 to generate silver transcriptions. This corpus is utilized to fine-tune our models, aiming to assess the efficacy of this approach in enhancing ASR performance. Our analysis spans 16 distinct podcast domains, alongside evaluations on established datasets for Modern Greek. The findings indicate consistent WER improvements, correlating with increases in both data volume and model size. Our study confirms that assembling large, weakly supervised corpora serves as a cost-effective strategy for advancing speech technologies in under-resourced languages.
Speech Recognition for Greek Dialects: A Challenging Benchmark
Authors: Socrates Vakirtzian; Chara Tsoukala; Stavros Bompolas; Katerina Mouzou; Vivian Stamou; Georgios Paraskevopoulos; Antonios Dimakis; Stella Markantonatou; Angela Ralli; Antonios Anastasopoulos
Venue: Interspeech 2024 (main)

Abstract: Language technologies should be judged on their usefulness in real-world use cases. Despite recent impressive progress in automatic speech recognition (ASR), an often overlooked aspect in ASR research and evaluation is language variation in the form of non-standard dialects or language varieties.  To this end, this work introduces a challenging benchmark that focuses on four varieties of Greek (Aivaliot, Cretan, Griko, Messenian) encompassing challenges related to data availability, orthographic conventions, and complexities arising from language contact. Initial experiments with state-of-the-art models and established cross-lingual transfer techniques highlight the difficulty of adapting to such low-resource varieties.
LUPET: Incorporating Hierarchical Information Path into Multilingual ASR
Authors: Wei Liu; Jingyong Hou; Dong Yang; Muyong Cao; Tan Lee
Venue: Interspeech 2024 (main)

Abstract: Toward high-performance multilingual automatic speech recognition (ASR), various types of linguistic information and model design have demonstrated their effectiveness independently. They include language identity (LID), phoneme information, language-specific processing modules, and cross-lingual self-supervised speech representation. It is expected that leveraging their benefits synergistically in a unified solution would further improve the overall system performance. This paper presents a novel design of a hierarchical information path, named LUPET, which sequentially encodes, from the shallow layers to deep layers, multiple aspects of linguistic and acoustic information at diverse granularity scales. The path starts from LID prediction, followed by acoustic unit discovery, phoneme sharing, and finally token recognition routed by a mixture-of-expert.  ASR experiments are carried out on 10 languages in the Common Voice corpus. The results demonstrate the superior performance of LUPET as compared to the baseline systems. Most importantly, LUPET effectively mitigates the issue of performance compromise of high-resource languages with low-resource ones in the multilingual setting.
Whispering in Norwegian: Navigating Orthographic and Dialectic Challenges
Authors: Per E Kummervold; Javier de la Rosa; Freddy Wetjen; Rolv-Arild Braaten; Per Erik Solberg
Venue: Interspeech 2024 (main)

Abstract: This paper presents NB-Whisper, a tailored adaptation of OpenAIâs Whisper model, specifically fine-tuned to address the unique challenges of Norwegian language Automatic Speech Recognition (ASR). We highlight its key contributions and summarise the results achieved in converting spoken Norwegian into written forms and translating other languages into Norwegian. By training on a 22,000 hour weakly aligned dataset, we show that we are able to improve the Norwegian BokmÃ¥l transcription by OpenAI Whisper Large-v3 from a WER of 10.4 to 6.6 on the Fleurs Dataset and from 6.8 to 2.2 on the NST dataset.
EFFUSE: Efficient Self-Supervised Feature Fusion for E2E ASR in Low Resource and Multilingual Scenarios
Authors: Tejes Srivastava; Jiatong Shi; William Chen; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: Self-Supervised Learning (SSL) models have demonstrated exceptional performance in various speech tasks, particularly in low-resource and multilingual domains. Recent works show that fusing diverse SSL models could achieve superior performance compared to using one SSL model. However, fusing models increases the overall parameter size, leading to higher computational costs. We propose EFFUSE, a novel approach that uses a single SSL model to mimic the features of multiple SSL models via prediction, resulting in a lightweight framework with competitive performance. Our experiments show that EFFUSE outperforms individual SSL models in multilingual speech recognition tasks. Our best performing model achieves an average SUPERB score increase of 63.5 (6.3%) from the SSL baselines in Multilingual Speech Universal PERformance Benchmark (ML-SUPERB), while decreasing parameter size on average by 317M parameters (49%) from the fusion models.
Enhancing Neural Transducer for Multilingual ASR with Synchronized Language Diarization
Authors: Amir Hussein; Desh Raj; Matthew Wiesner; Daniel Povey; Paola Garcia; Sanjeev Khudanpur
Venue: Interspeech 2024 (main)

Abstract: In multilingual environments, seamless language switching, including code-switching (CS) within utterances, is essential for real-time applications. Conventional Automatic Speech Recognition (ASR) combined with language diarization requires post-processing to synchronize language labels with recognized words accurately, presenting a considerable challenge. In this study, we introduce a multitask learning framework that synchronizes Language Identification (LID) with ASR, utilizing a neural transducer architecture. This auxiliary task integrates both acoustic and lexical features to perform LID. Furthermore, we use resulting language representation as an auxiliary input to improve ASR. We demonstrate the efficacy of our proposed approach on conversational multilingual (Arabic, Spanish, Mandarin) and CS (Spanish-English, Mandarin-English) test sets.
SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR
Authors: Shuaishuai Ye; Shunfei Chen; Xinhui Hu; Xinkang Xu
Venue: Interspeech 2024 (main)

Abstract: In this work, we propose a Switch-Conformer-based MoE system named SC-MoE for unified streaming and non-streaming code-switching (CS) automatic speech recognition (ASR), where we design a streaming MoE layer consisting of three language experts, which correspond to Mandarin, English, and blank, respectively, and equipped with a language identification (LID) network with a Connectionist Temporal Classification (CTC) loss as a router in the encoder of SC-MoE to achieve a real-time streaming CS ASR system. To further utilize the language information embedded in text, we also incorporate MoE layers into the decoder of SC-MoE. In addition, we introduce routers into every MoE layer of the encoder and the decoder and achieve better recognition performance. Experimental results show that the SC-MoE significantly improves CS ASR performances over baseline with comparable computational efficiency.
Optimizing Automatic Speech Assessment: W-RankSim Regularization and Hybrid Feature Fusion Strategies
Authors: Chung-Wen Wu; Berlin Chen
Venue: Interspeech 2024 (main)

Abstract: Automatic Speech Assessment (ASA) has seen notable advancements with the utilization of self-supervised features (SSL) in recent research. However, a key challenge in ASA lies in the imbalanced distribution of data, particularly evident in English test datasets. To address this challenge, we approach ASA as an ordinal classification task, introducing Weighted Vectors Ranking Similarity (W-RankSim) as a novel regularization technique. W-RankSim encourages closer proximity of weighted vectors in the output layer for similar classes, implying that feature vectors with similar labels would be gradually nudged closer to each other as they converge towards corresponding weighted vectors. Extensive experimental evaluations confirm the effectiveness of our approach in improving ordinal classification performance for ASA. Furthermore, we propose a hybrid model that combines SSL and handcrafted features, showcasing how the inclusion of handcrafted features enhances performance in an ASA system.
Context-Aware Speech Recognition Using Prompts for Language Learners
Authors: Jian Cheng
Venue: Interspeech 2024 (main)

Abstract: We aim to enhance automatic speech recognition (ASR) systems with context-aware prompts, improving accuracy without needing complex domain-specific language models or fine-tuning. This is particularly valuable for spoken language learning, where instruction/assessment apps often present short spoken texts to elicit spoken responses. These elicitors reduce the range of expected, sensible spoken responses. Prompting ASR engines (Whisper and Gemini Audio) with an utterance's elicitor yields context-awareness and significantly improves performance. In two L2 English datasets, using elicitor texts as prompts improved Whisper and Gemini accuracy by up to 24.0% (relative WER). For one activity type, the elicitor text reduces errors in target words by half. Out-of-domain, prompt-enhanced Gemini bettered a conventional ASR system trained on in-domain data by 35.3% (relative WER); enhanced Whisper bettered it by 21.3%.
A Dataset and Two-pass System for Reading Miscue Detection
Authors: Raj Gothi; Rahul Kumar; Mildred Pereira; Nagesh Nayak; Preeti Rao
Venue: Interspeech 2024 (main)

Abstract: Automatic speech recognition (ASR) has long been viewed as a promising solution to the resource-intensive task of oral reading fluency assessment. The demands on ASR accuracy, however, tend to be high, especially when applied to obtaining reliable reading diagnostics. The prior knowledge of reading prompts is typically used to limit the system WER. The accurate detection of mispronounced words, which can be relatively few in number, while limiting false positives, remains challenging. In this work, we present a new manually transcribed dataset of 1,110 elementary school children reading connected text in L2 English with wide-ranging proficiencies. Apart from local features derived from alternate decodings under different linguistic context constraints, we use an additional deep acoustic model. We discuss the performance gains achieved in a second pass over initial hybrid ASR hypotheses.
Oversampling, Augmentation and Curriculum Learning for Speaking Assessment with Limited Training Data
Authors: Tin Mei Lun; Ekaterina Voskoboinik; Ragheb Al-Ghezi; Tamas Grosz; Mikko Kurimo
Venue: Interspeech 2024 (main)

Abstract: Automated assessment systems for spontaneous speech are an increasingly important component in language proficiency tests and learning platforms. These systems have seen remarkable development in recent years, driven by advances in self-supervised learning. Nevertheless, in languages such as Finnish and Finland Swedish, their performance is still limited by the low-resource and imbalance nature of their data. To alleviate these issues, this work evaluates two data-level methods: oversampling and curriculum learning. Our results reveal that combining these methods results in the greatest boost to model performance, achieved without additional data or modification to the model structure.
Analysis and Visualization of Directional Diversity in Listening Fluency of World Englishes Speakers in the Framework of Mutual Shadowing
Authors: Yu Tomita; Yingxiang Gao; Nobuaki Minematsu; Noriko Nakanishi; Daisuke Saito
Venue: Interspeech 2024 (main)

Abstract: English is spoken as a lingua franca with a diversity of pronunciations, called accents, and they have been well studied so far. In this study, a diversity of listening behaviors are focused on, and listening disfluencies are measured objectively while listening to World Englishes (WE). When speaker X listens to Y fluently, it does not always mean that Y listens to X fluently. After collecting different passages read aloud by different WE speakers, the collected oral passages are shadowed by the speakers themselves to quantify their listening disfluencies. Results show that, when X listens to Y, X's listening disfluency becomes larger when Y's pronunciation deviates from X's to a larger degree. Further, a method is proposed to visualize simultaneously a) how fluently a speaker listens to WE speakers and b) how fluently the WE speakers listen to that specific speaker. With this visualization, WE speakers are grouped based on their communicability in global contexts.
Quantifying the Role of Textual Predictability in Automatic Speech Recognition
Authors: Sean Robertson; Gerald Penn; Ewan Dunbar
Venue: Interspeech 2024 (main)

Abstract: A long-standing question in automatic speech recognition research is how to attribute errors to the ability of a model to model the acoustics, versus its ability to leverage higher-order context (lexicon, morphology, syntax, semantics). We validate a novel approach which models error rates as a function of relative textual predictability, and yields a single number, k, which measures the effect of textual predictability on the recognizer. We use this method to demonstrate that a Wav2Vec 2.0-based model makes greater stronger use of textual context than a hybrid ASR model, in spite of not using an explicit language model, and also use it to shed light on recent results demonstrating poor performance of standard ASR systems on African-American English. We demonstrate that these mostly represent failures of  acoustic-phonetic modelling. We show how this approach can be used straightforwardly in diagnosing and improving ASR.
TM-PATHVQA: 90000+ Textless Multilingual Questions for Medical Visual Question Answering
Authors: Tonmoy Rajkhowa; Amartya Roy Chowdhury; Sankalp Nagaonkar; Achyut Mani Tripathi; Mahadeva Prasanna
Venue: Interspeech 2024 (main)

Abstract: In healthcare and medical diagnostics, Visual Question Answering (VQA) may emerge as a pivotal tool in scenarios where analysis of intricate medical images becomes critical for accurate diagnoses. Current text-based VQA systems limit their utility in scenarios where hands-free interaction and accessibility are crucial while performing tasks. A speech-based VQA system may provide a better means of interaction where information can be accessed while performing tasks simultaneously. To this end, this work implements a speech-based VQA system by introducing a Textless Multilingual Pathological VQA (TM-PathVQA) dataset, an expansion of the PathVQA dataset, containing spoken questions in English, German and French. This dataset comprises 98,397 multilingual spoken questions and answers based on 5,004 pathological images along with 70 hours of audio. Finally, this work benchmarks and compares TM-PathVQA systems implemented using various combinations of acoustic and visual features.
Towards Multilingual Audio-Visual Question Answering
Authors: Orchid Chetia Phukan; Priyabrata Mallick; Swarup Ranjan Behera; Aalekhya Satya Narayani; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2024 (main)

Abstract: In this paper, we work towards extending Audio-Visual Question Answering (AVQA) to multilingual settings. Existing AVQA research has predominantly revolved around English and replicating it for addressing AVQA in other languages requires a substantial allocation of resources. As a scalable solution, we leverage machine translation and present two multilingual AVQA datasets for eight languages created from existing benchmark AVQA datasets. This prevents extra human annotation efforts of collecting questions and answers manually. To this end, we propose, MERA framework, by leveraging state-of-the-art (SOTA) video, audio, and textual foundation models for AVQA in multiple languages. We introduce a suite of models namely MERA-L, MERA-C, MERA-T with varied model architectures to benchmark the proposed datasets. We believe our work will open new research directions and act as a reference benchmark for future works in multilingual AVQA.
Reinforcement Learning from Answer Reranking Feedback for Retrieval-Augmented Answer Generation
Authors: Minh Nguyen; Toan Quoc Nguyen; Kishan KC; Zeyu Zhang; Thuy Vu
Venue: Interspeech 2024 (main)

Abstract: Retrieval-augmented generation (RAG) is a method to improve accuracy and reliability of large language models (LLMs) for open-domain question answering (ODQA). Traditional approaches rely on supervised learning, which can result in misaligned user intent and system output. Reinforcement learning from human feedback (RLHF) addresses this issue by training a reward model using human preference feedback. In this work, we introduce a novel RLHF framework for ODQA, leveraging existing large-scale answer reranking datasets for training a reward model. In particular, our reward model for ODQA plays two complementary roles: (i) providing ranking scores as rewards for PPO, and (ii) retrieving relevant facts that enable the ODQA system to formulate a factual answer. Experimental results indicate that our proposed framework is effective for RLHF, leading to near-expert performance for ODQA.
Instruction Data Generation and Unsupervised Adaptation for Speech Language Models
Authors: Vahid Noroozi; Zhehuai Chen; Somshubra Majumdar; Steve Huang; Jagadeesh Balam; Boris Ginsburg
Venue: Interspeech 2024 (main)

Abstract: In this paper, we propose three methods for generating synthetic samples to train and evaluate multimodal large language models capable of processing both text and speech inputs. Addressing the scarcity of samples containing both modalities, synthetic data generation emerges as a crucial strategy to enhance the performance of such systems and facilitate the modeling of cross-modal relationships between the speech and text domains. Our process employs large language models to generate textual components and text-to-speech systems to generate speech components. The proposed methods offer a practical and effective means to expand the training dataset for these models. Experimental results show progress in achieving an integrated understanding of text and speech. We also highlight the potential of using unlabeled speech data to generate synthetic samples comparable in quality to those with available transcriptions, enabling the expansion of these models to more languages.
On the Use of Plausible Arguments in Explainable Conversational AI
Authors: Martina Di Bratto; Maria Di Maro; Antonio Origlia
Venue: Interspeech 2024 (main)

Abstract: Conversational Artificial Intelligence has evolved to facilitate more efficient communication of user preferences through dialogues. This paper delves into Argumentative Conversational AI systems, and, more specifically in the definition of a methodology of selecting and using plausible arguments to support recommendations. We propose a cross-disciplinary model grounded in cognitive pragmatics to enhance recommendation quality. We evaluate this linguistically motivated strategy in isolation using simulated dialogues and collect human judgements to verify that the expected interaction is believable. Next, we test the full interaction model with human users to evaluate its usability. Results indicate high scores for naturalness and argument selection, validating the system's plausibility and effectiveness. Concerning usability, the system is perceived as attractive and reliable although technical issues concerning the system's reactiveness are present.
Rapport-Driven Virtual Agent: Rapport Building Dialogue Strategy for Improving User Experience at First Meeting
Authors: Muhammad Yeza Baihaqi; Angel Garcia Contreras; Seiya Kawano; Koichiro Yoshino
Venue: Interspeech 2024 (main)

Abstract: Rapport is known as a conversational aspect focusing on relationship building, which influences outcomes in collaborative tasks. This study aims to establish human-agent rapport through small talk by using a rapport-building strategy. We implemented this strategy for the virtual agents based on dialogue strategies by prompting a large language model (LLM). In particular, we utilized two dialogue strategiesâpredefined sequence and free-formâto guide the dialogue generation framework. We conducted analyses based on human evaluations, examining correlations between total turn, utterance characters, rapport score, and user experience variables: naturalness, satisfaction, interest, engagement, and usability. We investigated correlations between rapport score and naturalness, satisfaction, engagement, and conversation flow. Our experimental results also indicated that using free-form to prompt the rapport-building strategy performed the best in subjective scores.
Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrieval
Authors: Lifeng Zhou; Yuke Li; Rui Deng; Yuting Yang; Haoqi Zhu
Venue: Interspeech 2024 (main)

Abstract: The success of speech-image retrieval relies on establishing an effective alignment between speech and image. Existing methods often model cross-modal interaction through simple cosine similarity of the global feature of each modality, which fall short in capturing fine-grained details within modalities. To address this issue, we introduce an effective framework and a novel learning task named cross-modal denoising(CMD) to enhance cross-modal interaction to achieve finerlevel cross-modal alignment. Specifically, CMD is a denoising task designed to reconstruct semantic features from noisy features within one modality by interacting features from another modality. Notably, CMD operates exclusively during model training and can be removed during inference without adding extra inference time. The experimental results demonstrate that our framework outperforms the state-of-the-art method by 2.0% in mean R@1 on the Flickr8k dataset and by 1.7% in mean R@1 on the SpokenCOCO dataset for the speech-image retrieval tasks, respectively. These experimental results validate the efficiency and effectiveness of our framework.