GenAI4ELab/papercli-papers huggingface dataset:
AR-NLU: A Framework for Enhancing Natural Language Understanding Model Robustness against ASR Errors
Authors: Emmy Phung; Harsh Deshpande; Ahmad Emami; Kanishk Singh
Venue: Interspeech 2024 (main)

Abstract: A major challenge with pipeline spoken language understanding systems is that errors in the upstream automatic speech recognition (ASR) engine adversely impact downstream natural language understanding (NLU) models. To address this challenge, we propose an ASR-Robust NLU (AR-NLU) framework that extends a pre-existing NLU model by training it simultaneously on two input streams: human generated or gold transcripts and noisy ASR transcripts. We apply contrastive learning to make the model learn the same representations and predictions for both gold and ASR inputs, thereby enhancing its robustness against ASR noises. To demonstrate the effectiveness of this framework, we present two AR-NLU models: a Robust Intent DEtection (RIDE) and ASR-Robust BI-encoder for NameD Entity Recognition (AR-BINDER). Experimental results show that our proposed AR-NLU framework is applicable to various NLU models and significantly outperforms the original models in both sequence and token classification tasks.
Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language Understanding
Authors: Mohan Li; Simon Keizer; Rama Doddipatla
Venue: Interspeech 2024 (main)

Abstract: Zero-shot spoken language understanding (SLU) enables systems to comprehend user utterances in new domains without prior exposure to training data. Recent studies often rely on large language models (LLMs), leading to excessive footprints and complexity. This paper proposes the use of Whisper, a standalone speech processing model, for zero-shot end-to-end (E2E) SLU. To handle unseen semantic labels, SLU tasks are integrated into a question-answering (QA) framework, which prompts the Whisper decoder for semantics deduction. The system is efficiently trained with prefix-tuning, optimising a minimal set of parameters rather than the entire Whisper model. We show that the proposed system achieves a 40.7% absolute gain for slot filling (SLU-F1) on SLURP compared to a recently introduced zero-shot benchmark. Furthermore, it performs comparably to a Whisper-GPT-2 modular system under both in-corpus and cross-corpus evaluation settings, but with a relative 34.8% reduction in model parameters.
VN-SLU: A Vietnamese Spoken Language Understanding Dataset
Authors: Tuyen Tran; Khanh Le; Ngoc Dang Nguyen; Minh Vu; Huyen Ngo; Woomyoung Park; Thi Thu Trang Nguyen
Venue: Interspeech 2024 (main)

Abstract: Spoken Language Understanding (SLU) is a crucial task in spoken language processing. Despite the availability of numerous English datasets for research, there is a scarcity of resources for low-resource languages like Vietnamese. This paper introduces VN-SLU, the first dataset explicitly designed for Vietnamese SLU. VN-SLU includes 17,321 utterances from 240 Vietnamese speakers, obtained through novel crowd-sourcing methods. We propose a web tool for scenario generation and label validation, ensuring dataset quality and diversity. This tool prompts participants to confirm intents and slot values in smart home and virtual assistant dialogues, ensuring precise alignment. Experimental results highlight the challenging nature of the chosen test set sampling strategy in intent accuracy, SLU-F1, and utterance accuracy. Additionally, we explore the integration of pitch information into the Vietnamese SLU system. Results show improved performance compared to the baseline model.
Textless Dependency Parsing by Labeled Sequence Prediction
Authors: Shunsuke Kando; Yusuke Miyao; Jason Naradowsky; Shinnosuke Takamichi
Venue: Interspeech 2024 (main)

Abstract: Traditional spoken language processing involves cascading an automatic speech recognition (ASR) system into text processing models. In contrast, ``textless'' methods process speech representations without ASR systems, enabling the direct use of acoustic speech features. Although their effectiveness is shown in capturing acoustic features, it is unclear in capturing lexical knowledge. This paper proposes a textless method for dependency parsing, examining its effectiveness and limitations. Our proposed method predicts a dependency tree from a speech signal  without transcribing, representing the tree as a labeled sequence. scading method outperforms the textless method in overall parsing accuracy, the latter excels in instances with important acoustic features. Our findings highlight the importance of fusing word-level representations and sentence-level prosody for enhanced parsing performance. The code and models are made publicly available.
Towards Speech Classification from Acoustic and Vocal Tract data in Real-time MRI
Authors: Yaoyao Yue; Michael Proctor; Luping Zhou; Rijul Gupta; Tharinda Piyadasa; Amelia Gully; Kirrie Ballard; Craig Jin
Venue: Interspeech 2024 (main)

Abstract: Real-time magnetic resonance image (rtMRI) data of the upper airway provides a rich source of information about vocal tract shaping that can inform phonemic analysis and classification. We describe a multimodal phonemic classifier that combines articulatory data with speech audio features to improve performance. A deep network model processes rtMRI video data using ResNet18 and speech audio using a custom CNN and then combines the two data streams using a Transformer layer to fully explore the correlation of the two streams towards better vowel-consonant-vowel classification via the Transformer's multi-head self-attention mechanism. The classification accuracy of both the unimodal and multimodal models show substantial improvement on previous work (> 38%). The addition of audio features improves classification accuracy in the multimodal model by 7% compared with the unimodal model using articulatory data.  We analyze the model and discuss the phonetic implications.
Efficient SQA from Long Audio Contexts: A Policy-driven Approach
Authors: Alexander Johnson; Peter Plantinga; Pheobe Sun; Swaroop Gadiyaram; Abenezer Girma; Ahmad Emami
Venue: Interspeech 2024 (main)

Abstract: Spoken question answering (SQA), or the process of retrieving relevant information from speech recordings, has become a topic of great interest due in part to the recent growth of large language models. However, many state-of-the-art SQA have memory and compute costs that scale linearly with the length of the input audio context, making them infeasible to apply on long speech recordings (meetings, podcasts, etc.). This paper proposes a method which uses deep Q-learning to learn a policy for skipping over irrelevant audio segments in a longer audio file without analyzing them for more efficient SQA. In this framework, an agent model is trained on BERT sentence embeddings extracted from lightweight ASR transcripts to make decisions on how far it can safely skip through an audio file in order to move closer to the target answer span.  Applied on the CORAAL QA database, this work approaches the performance of SOTA SQA systems while using less than half of the compute to analyze audio.
BESST Dataset: A Multimodal Resource for Speech-based Stress Detection and Analysis
Authors: Jan PeÅ¡Ã¡n; VojtÄch JuÅÃ­k; Martin KarafiÃ¡t; Jan ÄernockÃ½
Venue: Interspeech 2024 (main)

Abstract: The Brno Extended Stress and Speech Test (BESST) dataset is a new resource for the speech research community, offering multimodal audiovisual, physiological and psychological data that enable investigations into the interplay between stress and speech. In this paper, we introduce the BESST dataset and provide a details of its design, collection protocols, and technical aspects. The dataset comprises speech samples, physiological signals (including electrocardiogram, electrodermal activity, skin temperature, and acceleration data), and video recordings from 90 subjects performing stress-inducing tasks. It comprises 16.9 hours of clean Czech speech data, averaging 15 minutes of clean speech per participant. The data collection procedure involves the induction of cognitive and physical stress induced by Reading Span task (RSPAN) and Hand Immersion (HIT) task respectively. The BESST dataset was collected under stringent ethical standards and is accessible for research and development.
HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing
Authors: Arnon Turetzky; Or Tal; Yael Segal; Yehoshua Dissen; Ella Zeldes; Amit Roth; Eyal Cohen; Yosi Shrem; Bronya R. Chernyak; Olga Seleznova; Joseph Keshet; Yossi Adi
Venue: Interspeech 2024 (main)

Abstract: We present HebDB, a weakly supervised dataset for spoken language processing in the Hebrew language. HebDB offers roughly 2500 hours of natural and spontaneous speech recordings in the Hebrew language, consisting of a large variety of speakers and topics. We provide raw recordings together with a pre-processed, weakly supervised, and filtered version. The goal of HebDB is to further enhance research and development of spoken language processing tools for the Hebrew language. Hence, we additionally provide two baseline systems for Automatic Speech Recognition (ASR): (i) a self-supervised model; and (ii) a fully supervised model. We present the performance of these two methods optimized on HebDB and compare them to current multi-lingual ASR alternatives. Results suggest the proposed method reaches better results than the evaluated baselines considering similar model sizes. Dataset, code, and models are publicly available under https://pages.cs.huji.ac.il/adiyoss-lab/HebDB/.
GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech
Authors: Wenbin Wang; Yang Song; Sanjay Jha
Venue: Interspeech 2024 (main)

Abstract: This paper introduces GLOBE, a high-quality English corpus with worldwide accents, specifically designed to address the limitations of current zero-shot speaker adaptive Text-to-Speech (TTS) systems that exhibit poor generalizability in adapting to speakers with accents. Compared to commonly used English corpora, such as LibriTTS and VCTK, GLOBE is unique in its inclusion of utterances from 23,519 speakers and covers 164 accents worldwide, along with detailed metadata for these speakers. Compared to its original corpus, i.e., Common Voice, GLOBE significantly improves the quality of the speech data through rigorous filtering and enhancement processes, while also populating all missing speaker metadata. The final curated GLOBE corpus includes 535 hours of speech data at a 24 kHz sampling rate. Our benchmark results indicate that the speaker adaptive TTS model trained on the GLOBE corpus can synthesize speech with better speaker similarity and comparable naturalness than that trained on other popular corpora. We will release GLOBE publicly after acceptance. The GLOBE dataset is available at https://globecorpus.github.io/.
STraDa: A Singer Traits Dataset
Authors: Yuexuan Kong; Viet-Anh Tran; Romain Hennequin
Venue: Interspeech 2024 (main)

Abstract: There is a limited amount of large-scale public datasets that contain downloadable music audio files and rich lead singer metadata. To provide such a dataset to benefit research in singing voices, we created Singer Traits Dataset (STraDa) with two subsets: automatic-strada and annotated-strada. The automatic-strada contains twenty-five thousand tracks across numerous genres and languages of more than five thousand unique lead singers, which includes cross-validated lead singer metadata as well as other track metadata. The annotated-strada consists of two hundred tracks that are balanced in terms of 2 genders, 5 languages, and 4 age groups. To show its use for model training and bias analysis thanks to its metadata's richness and downloadable audio files, we benchmarked singer sex classification (SSC) and conducted bias analysis.
MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features
Authors: Katharina Anderer; Andreas Reich; Matthias WÃ¶lfel
Venue: Interspeech 2024 (main)

Abstract: This paper presents a benchmark dataset for aligning lecture videos with corresponding slides and introduces a novel multimodal algorithm leveraging features from speech, text, and images. It achieves an average accuracy of 0.82 in comparison to SIFT (0.56) while being approximately 11 times faster. Using dynamic programming the algorithm tries to determine the optimal slide sequence. The results show that penalizing slide transitions increases accuracy. Features obtained via optical character recognition (OCR) contribute the most to a high matching accuracy, followed by image features. The findings highlight that audio transcripts alone provide valuable information for alignment and are beneficial if OCR data is lacking. Variations in matching accuracy across different lectures highlight the challenges associated with video quality and lecture style. The novel multimodal algorithm demonstrates robustness to some of these challenges, underscoring the potential of the approach.
MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Dataset
Authors: Kim Sung-Bin; Lee Chae-Yeon; Gihun Son; Oh Hyun-Bin; Janghoon Ju; Suekyeong Nam; Tae-Hyun Oh
Venue: Interspeech 2024 (main)

Abstract: Recent studies in speech-driven 3D talking head generation have achieved convincing results in verbal articulations. However, generating accurate lip-syncs degrades when applied to input speech in other languages, possibly due to the lack of datasets covering a broad spectrum of facial movements across languages. In this work, we introduce a novel task to generate 3D talking heads from speeches of diverse languages. We collect a new multilingual 2D video dataset comprising over 420 hours of talking videos in 20 languages. With our proposed dataset, we present a multilingual enhanced model that incorporates language-specific style embeddings, enabling it to capture the unique mouth movements associated with each language. Additionally, we present a metric for assessing lip-sync accuracy in multilingual settings. We demonstrate that training a 3D talking head model with our proposed dataset significantly enhances its multilingual performance. Codes and datasets are available at https://multitalk.github.io/.
Towards measuring fairness in speech recognition: Fair-Speech dataset
Authors: Irina-Elena Veliche; Zhuangqun Huang; Vineeth Ayyat Kochaniyan; Fuchun Peng; Ozlem Kalinli; Michael L. Seltzer
Venue: Interspeech 2024 (main)

Abstract: The current public datasets for speech recognition (ASR) tend not to focus specifically on the fairness aspect, such as performance across different demographic groups. This paper introduces a novel dataset, Fair-Speech, a publicly released corpus to help researchers evaluate their ASR models for accuracy across a diverse set of self-reported demographic information, such as age, gender, ethnicity, geographic variation and whether the participants consider themselves native English speakers. Our dataset includes approximately 26.5K utterances in recorded speech by 593 people in the United States, who were paid to record and submit audios of themselves saying voice commands. We also provide ASR baselines, including on models trained on transcribed and untranscribed social media videos and open source models.
Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio
Authors: Yi Lu; Yuankun Xie; Ruibo Fu; Zhengqi Wen; Jianhua Tao; Zhiyong Wang; Xin Qi; Xuefei Liu; Yongwei Li; Yukun Liu; Xiaopeng Wang; Shuchen Shi
Venue: Interspeech 2024 (main)

Abstract: With the proliferation of Large Language Model (LLM) based deepfake audio, there is an urgent need for effective detection methods. Previous deepfake audio generation methods typically involve a multi-step generation process, with the final step using a vocoder to predict the waveform from hand-crafted features. However, LLM-based audio is directly generated from discrete neural codecs in an end-to-end generation process, skipping the final step of vocoder processing. This poses a significant challenge for current audio deepfake detection (ADD) models based on vocoder artifacts. To effectively detect LLM-based deepfake audio, we focus on the core of the generation process, the conversion from neural codec to waveform. We propose Codecfake dataset, which is generated by seven representative neural codec methods.  Experiment results show that codec-trained ADD models exhibit a 41.406% reduction in average equal error rate compared to vocoder-trained ADD models on the Codecfake test set.
SER Evals: In-domain and Out-of-domain benchmarking for speech emotion recognition
Authors: Mohamed Osman; Daniel Z. Kaplan; Tamer Nadeem
Venue: Interspeech 2024 (main)

Abstract: Speech emotion recognition (SER) has made significant strides with the advent of powerful self-supervised learning (SSL) models. However, the generalization of these models to diverse languages and emotional expressions remains a challenge. We propose a large-scale  benchmark to evaluate the robustness and adaptability of state-of-the-art SER models in both in-domain and out-of-domain settings. Our benchmark includes a diverse set of multilingual datasets, focusing on less commonly used corpora to assess generalization to new data. We employ logit adjustment to account for varying class distributions and establish a single dataset cluster for systematic evaluation. Surprisingly, we find that the Whisper model, primarily designed for automatic speech recognition, outperforms dedicated SSL models in cross-lingual SER. Our results highlight the need for more robust and generalizable SER models, and our benchmark serves as a valuable resource to drive future research in this direction.
The MARRYS helmet: A new device for researching and training âjaw dancingâ
Authors: Vidar Freyr Gudmundsson; Keve MÃ¡rton GÃ¶nczi; Malin Svensson Lundmark; Donna Erickson; Oliver Niebuhr
Venue: Interspeech 2024 (main)

Abstract: The paper introduces a new device for analyzing, teaching, and training jaw movements: the MARRYS helmet. We outline the motivation for the development of the helmet, describe its key advantages and features relative to those of the Electromagnetic Articulograph (EMA) and illustrate by means of selected study portraits the possible uses of the MARRYS helmet in the various fields of the empirical and applied speech sciences.
Exploiting Foundation Models and Speech Enhancement for Parkinson's Disease Detection from Speech in Real-World Operative Conditions
Authors: Moreno La Quatra; Maria Francesca Turco; TorbjÃ¸rn Svendsen; Giampiero Salvi; Juan Rafael Orozco-Arroyave; Sabato Marco Siniscalchi
Venue: Interspeech 2024 (main)

Abstract: This work is concerned with devising a robust Parkinson's (PD) disease detector from speech in real-world operating conditions using (i) foundational models, and (ii) speech enhancement (SE) methods. To this end, we first fine-tune several foundational-based models on the standard PC-GITA (s-PC-GITA) clean data. Our results demonstrate superior performance to previously proposed models. Second, we assess the generalization capability of the PD models on the extended PC-GITA (e-PC-GITA) recordings, collected in real-world operative conditions, and observe a severe drop in performance moving from ideal to real-world conditions. Third, we align training and testing conditions applaying off-the-shelf SE techniques on e-PC-GITA, and a significant boost in performance is observed only  for the foundational-based models. Finally, combining the two best foundational-based models trained on  s-PC-GITA, namely WavLM Base and Hubert Base, yielded top performance on the enhanced e-PC-GITA.
Sustained Vowels for Pre- vs Post-Treatment COPD Classification
Authors: Andreas Triantafyllopoulos; Anton Batliner; Wolfgang Mayr; Markus Fendler; Florian Pokorny; Maurice Gerczuk; Shahin Amiriparian; Thomas Berghaus; BjÃ¶rn Schuller
Venue: Interspeech 2024 (main)

Abstract: Chronic obstructive pulmonary disease (COPD) is a serious inflammatory lung disease affecting millions of people around the world. Due to an obstructed airflow from the lungs, it also becomes manifest in patients' vocal behaviour. Of particular importance is the detection of an exacerbation episode, which marks an acute phase  and often requires hospitalisation and treatment. Previous work has shown that it is possible to distinguish between a pre- and a post-treatment state using automatic analysis of read speech. In this contribution, we examine whether sustained vowels can provide a complementary lens for telling apart these two states. Using a cohort of 50 patients, we show that the inclusion of sustained vowels can improve performance to up to 79% unweighted average recall, from a 71% baseline using read speech. We further identify and interpret the most important acoustic features that characterise the manifestation of COPD in sustained vowels.
Adversarial Robustness Analysis in Automatic Pathological Speech Detection Approaches
Authors: Mahdi Amiri; Ina Kodrasi
Venue: Interspeech 2024 (main)

Abstract: Automatic pathological speech detection relies on deep learning (DL), showing promising performance for various pathologies.  Despite the critical importance of robustness in healthcare applications like pathological speech detection, the sensitivity of DL-based pathological speech detection approaches to adversarial attacks remains unexplored. This paper explores the impact of acoustically imperceptible adversarial perturbations on DL-based pathological speech detection. Imperceptibility of perturbations, generated using the projected gradient descent algorithm, is evaluated using speech enhancement metrics. Results reveal a high vulnerability of DL-based pathological speech detection to adversarial perturbations, with adversarial training ineffective in enhancing robustness. Analysis of the perturbations provide insights into the speech components that the approaches attend to. These findings highlight the need for research in robust pathological speech detection.
Automatic Children Speech Sound Disorder Detection with Age and Speaker Bias Mitigation
Authors: Gahye Kim; Yunjung Eom; Selina S. Sung; Seunghee Ha; Tae-Jin Yoon; Jungmin So
Venue: Interspeech 2024 (main)

Abstract: Addressing speech sound disorders (SSD) in early childhood is pivotal for mitigating cognitive and communicative impediments. Previous works on automatic SSD detection rely on audio features without considering the age and speaker bias which results in degraded performance. In this paper, we propose an SSD detection system in which debiasing techniques are applied to mitigate the biases. For the age bias, we use a multi-head model where the feature extractor is shared across different age groups but the final decision is made using the age-dependent classifier. For the speaker bias, we augment the dataset by mixing the audios of the multiple speakers in the same age group. When evaluated with our Korean SSD dataset, the proposed method showed significant improvements over previous approaches.
Reference-Free Estimation of the Quality of Clinical Notes Generated from Doctor-Patient Conversations
Authors: Mojtaba Kadkhodaie Elyaderani; John Glover; Thomas Schaaf
Venue: Interspeech 2024 (main)

Abstract: This paper describes a simple yet robust approach to performing reference-free estimation of the quality of automatically-generated clinical notes derived from doctor-patient conversations. In the absence of human-written reference notes, this approach works by generating a diverse collection of "pseudo-reference notes" and comparing the generated note against those pseudo-references. This method has been applied to estimate the quality of clinical note sections generated by three different note generation models, using a collection of evaluation metrics that are based on natural language inference and clinical concept extraction. Our experiments show the proposed approach is robust to the choice of note generation models, and consistently produces higher correlations with reference-based counterparts when compared against a strong baseline method.