GenAI4ELab/papercli-papers huggingface dataset:
MM-NodeFormer: Node Transformer Multimodal Fusion for Emotion Recognition in Conversation
Authors: Zilong Huang; Man-Wai Mak; Kong Aik Lee
Venue: Interspeech 2024 (main)

Abstract: Emotion Recognition in Conversation (ERC) has great prospects in human-computer interaction and medical consultation. Existing ERC approaches mainly focus on information in the text and speech modalities and often concatenate multimodal features without considering the richness of emotional information in individual modalities. We propose a multimodal network called MM-NodeFormer for ERC to address this issue. The network leverages the characteristics of different Transformer encoding stages to fuse the emotional features from the text, audio, and visualÂ modalitiesÂ according to their emotional richness. The module considers text as the main modality and audio and visual as auxiliary modalities, leveraging the complementarity between the main and auxiliary modalities. We conducted extensive experiments on two public benchmark datasets, IEMOCAP and MELD, achieving an accuracy of 74.24% and 67.86%, respectively, significantly higher than many state-of-the-art approaches.
Emotional Cues Extraction and Fusion for Multi-modal Emotion Prediction and Recognition in Conversation
Authors: Haoxiang Shi; Ziqi Liang; Jun Yu
Venue: Interspeech 2024 (main)

Abstract: Emotion Prediction in Conversation (EPC) aims to forecast the emotions of forthcoming utterances by utilizing preceding dialogues. Previous EPC approaches relied on simple context modeling for emotion extraction, overlooking fine-grained emotion cues at the word level. Additionally, prior works failed to account for the intrinsic differences between modalities, resulting in redundant information. To overcome these limitations, we propose an emotional cues extraction and fusion network, which consists of two stages: a modality-specific learning stage that utilizes word-level labels and prosody learning to construct emotion embedding spaces for each modality, and a two-step fusion stage for integrating multi-modal features. Moreover, the emotion features extracted by our model are also applicable to the Emotion Recognition in Conversation (ERC) task. Experimental results validate the efficacy of the proposed method, demonstrating superior performance on both IEMOCAP and MELD datasets.
Participant-Pair-Wise Bottleneck Transformer for Engagement Estimation from Video Conversation
Authors: Keita Suzuki; Nobukatsu Hojo; Kazutoshi Shinoda; Saki Mizuno; Ryo Masumura
Venue: Interspeech 2024 (main)

Abstract: This study investigates the task of estimating the engagement of a target participant from video and audio during a multi-person conversation. For this task, interaction should be modeled effectively, considering the redundancy of video and audio across frames among multiple participants. Conventional Transformer-based methods in multimodal sentiment analysis succeeded in such efficient modeling by constraining the at- tention across multimodal data streams to go through only a small set of latent fusion units (âglobal tokensâ) that form an attention bottleneck. However, performance can be limited in the multi-person model because it needs to model interaction among a larger number of data streams based on only a single global token sequence. To address this problem, we propose a participant-pair-wise bottleneck transformer (PPBT) that involves multiple global token sequences, each of which is dedicated to a particular pair of participants and demonstrates its effect.
Well, what can you do with messy data? Exploring the prosody and pragmatic function of the discourse marker "well" with found data and speech synthesis
Authors: Johannah O'Mahony; Catherine Lai; Ãva SzÃ©kely
Venue: Interspeech 2024 (main)

Abstract: Recently, there has been growing interest in the synthesis of conversational speech prosody. Conversational prosody is variable and carries many pragmatic functions. As speech synthesis research moves to using large amounts of untranscribed data, it is crucial that we understand the subtle pragmatic differences prosody can make. This study focuses on discourse markers, which are linguistic elements that perform various communicative functions, with their specific roles often linked to their prosodic realisation. In this paper, we explore the prosodic realisation of well using an unlabelled corpus of conversational speech. We use clustering to explore the variation in its prosodic realisation and identify common patterns in a data-driven manner. We synthesise the cluster centroids using controllable speech synthesis. Finally, we evaluate how the prosodic realisation of well affects the meaning of an utterance.
Learning from Multiple Annotator Biased Labels in Multimodal Conversation
Authors: Kazutoshi Shinoda; Nobukatsu Hojo; Saki Mizuno; Keita Suzuki; Satoshi Kobashikawa; Ryo Masumura
Venue: Interspeech 2024 (main)

Abstract: In multimodal conversation analysis, annotating social signals such as speakers' communication skills is inherently subjective and prone to individual annotator bias, which is annotator's tendency to assign labels based on their values. These biases can contribute to label distributions biased towards specific speakers and classes that match annotators' values, leading to degraded classification performance for minority classes and speakers. Existing methods for addressing class imbalance and dataset bias often overlook the variable biases introduced by multiple annotators, which can lead to overfitting to the majority. Thus, we propose a novel two-stage debiasing method, MAD-LM, that first learns the typical label distribution for each annotator and then promotes the learning of untypical labels. MAD-LM effectively mitigates performance degradation for the minority in a multimodal conversation dataset with multiple annotator labels, while maintaining the performance for the majority.
Non-Linear Inference Time Intervention: Improving LLM Truthfulness
Authors: Jakub Hoscilowicz; Adam Wiacek; Jan Chojnacki; Adam Cieslak; Leszek Michon; Artur Janicki
Venue: Interspeech 2024 (main)

Abstract: In this work, we explore LLM's internal representation space to identify attention heads that contain the most truthful and accurate information. We further developed the Inference Time Intervention (ITI) framework, which lets bias LLM without the need for fine-tuning. The improvement manifests in introducing a non-linear multi-token probing and multi-token intervention: Non-Linear ITI (NL-ITI), which significantly enhances performance on evaluation benchmarks. NL-ITI is tested on diverse multiple-choice datasets, including TruthfulQA, on which we report over 14% relative MC1 (accuracy of model pointing to the correct answer) improvement with respect to the baseline ITI results. Moreover, we achieved a 10% relative improvement over the recently released Truth Forest (TrFf) method that also focused on ITI improvement.
Evaluating Speech Recognition Performance Towards Large Language Model Based Voice Assistants
Authors: Zhe Liu; Suyoun Kim; Ozlem Kalinli
Venue: Interspeech 2024 (main)

Abstract: In recent years, there has been a rise in the popularity of large language model (LLM) based voice assistants. A practical question being raised in the evaluation of cascaded automatic speech recognition (ASR) systems in LLM-powered voice assistants is how to determine whether any errors in ASR transcriptions will result in task failures for the downstream assistants. Thus, measuring ASR systems that can reflect voice assistants' perception and judgement becomes increasingly important. In this paper, we propose novel evaluation metrics by leveraging the same assistant LLM to project ASR hypotheses into a vector space and compute their semantic distances with respect to the references. We perform experiments on a curated OpenAssistant test set and demonstrate that our presented methods with semantic embeddings calculated from LLMs are superior to conventional metrics on evaluating ASR performance towards LLM based voice assistants.
Automatic Assessment of Dysarthria using Speech and synthetically generated Electroglottograph signal
Authors: Fathima Zaheera; Supritha Shetty; Gayadhar Pradhan; Deepak K T
Venue: Interspeech 2024 (main)

Abstract: The formants are flat and dispersed in the short-term magnitude spectra (STMS) of dysarthric speech. This paper investigates the possibility of enhancing the performance of an automated dysarthric assessment by exploiting the complementary perceptual cues present in the STMS of speech and synthetically generated Electroglottograph (EGG) signal. To capture the complementary information through a single acoustic feature representation, the log Mel filterbank energy (LMFE) computed from both kinds of signal is averaged. The resulting LMFE is then used for the computation of Mel frequency cepstral coefficients (MFCCs). The analytical and experimental results presented on the UA-Speech corpus validate the efficacy of the proposed approach. For the x-vector-based automated dysarthric assessment system the accuracy and F1 score improved from 73% and 64% to 78% and 71%, respectively, in a speaker and text-independent mode when the MFCCs are computed from the averaged LMFE.
CDSD: Chinese Dysarthria Speech Database
Authors: Yan Wan; Mengyi Sun; Xinchen Kang; Jingting Li; Pengfei Guo; Ming Gao; Su-Jing Wang
Venue: Interspeech 2024 (main)

Abstract: Dysarthric speech poses significant challenges for individuals with dysarthria, impacting their ability to communicate socially. Despite the widespread use of Automatic Speech Recognition (ASR), accurately recognizing dysarthric speech remains a formidable task, largely due to the limited availability of dysarthric speech data. To address this gap, we developed the Chinese Dysarthria Speech Database (CDSD), the most extensive collection of Chinese dysarthria data to date, featuring 133 hours of recordings from 44 speakers. Our benchmarks reveal a best Character Error Rate (CER) of 16.4%. Compared to the CER of 20.45% from our additional human experiments, Dysarthric Speech Recognition (DSR) demonstrates its potential in significant improvement of communication for individuals with dysarthria. The CDSD database will be made  publicly available at http://melab.psych.ac.cn/CDSD.html.
Exploring Syllable Discriminability during Diadochokinetic Task with Increasing Dysarthria Severity for Patients with Amyotrophic Lateral Sclerosis
Authors: Neelesh Samptur; Tanuka Bhattacharjee; Anirudh Chakravarty K; Seena Vengalil; Yamini Belur; Atchayaram Nalini; Prasanta Kumar Ghosh
Venue: Interspeech 2024 (main)

Abstract: We explore the discriminability among /pa/, /ta/, and /ka/ syllables, spoken during diadochokinetic (DDK) task, at varied severity levels of amyotrophic lateral sclerosis (ALS) induced dysarthria. Though DDK rate is known to decline with increasing severity, the extent to which the discriminability among the syllables gets impacted at each severity level is not well understood. We perform manual and automatic classification of these three syllables on 100 ALS and 35 healthy subjects. Manual classification is done through listening tests. Spectral and self-supervised speech cues with deep neural classifiers are used for automatic classification. Manual classification accuracies decline from 84.07% on healthy utterances to 27.41% on utterances of the most severe patients. Automatic methods are found to outperform humans achieving 15.93% and 50.37% higher accuracies (absolute), respectively. Thus, discriminative acoustic cues seem to persist among the syllables, which automatic methods capture.
Beyond Binary: Multiclass Paraphasia Detection with Generative Pretrained Transformers and End-to-End Models
Authors: Matthew Perez; Aneesha Sampath; Minxue Niu; Emily Mower Provost
Venue: Interspeech 2024 (main)

Abstract: Aphasia is a language disorder that can lead to speech errors known as paraphasias, which involve the misuse, substitution, or invention of words. Automatic paraphasia detection can help those with Aphasia by facilitating clinical assessment and treatment planning options. However, most automatic paraphasia detection works have focused solely on binary detection, which involves recognizing only the presence or absence of a paraphasia. Multiclass paraphasia detection represents an unexplored area of research that focuses on identifying multiple types of paraphasias and where they occur in a given speech segment. We present novel approaches that use a generative pretrained transformer (GPT) to identify paraphasias from transcripts as well as two end-to-end approaches that focus on modeling both automatic speech recognition (ASR) and paraphasia classification as multiple sequences vs. a single sequence. We demonstrate that a single sequence model outperforms GPT baselines for multiclass paraphasia detection.
Electroglottography for the assessment of dysphonia in Parkinson's disease and multiple system atrophy
Authors: Khalid Daoudi; Solange MilhÃ© de Saint Victor; Alexandra Foubert-Samier; Margherita Fabbri; Anne Pavy-Le Traon; Olivier Rascol; Virginie Woisard; Wassilios G. Meissner
Venue: Interspeech 2024 (main)

Abstract: Electroglottography (EGG) is a noninvasive technique which allows accurate measurements of vocal folds dynamics and perturbations during speech. It has been widely used in medical diagnosis and monitoring of several laryngeal pathologies, but its use in neurological disorders has been very limited. This paper presents the first study on EGG in early stages of Parkinson's disease (PD) and an atypical parkinsonian disorder, multiple system atrophy (MSA). Our first objective was to investigate whether EGG can reveal distinctive dysphonia features that could help in the challenging early differential diagnosis between PD and MSA-P, the parkinsonian variant of MSA. The second objective was to verify some hypothesis on early markers of PD drawn from speech-alone acoustic analysis.  For MSA-P patients, our analysis revealed a reduced open quotient and confirmed excessive pitch variation. The analysis also mitigated some hypothesis on dysphonia in early stages of PD.
CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction
Authors: Xueyuan Chen; Dongchao Yang; Dingdong Wang; Xixin Wu; Zhiyong Wu; Helen Meng
Venue: Interspeech 2024 (main)

Abstract: Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech. It still suffers from low speaker similarity and poor prosody naturalness. In this paper, we propose a multi-modal DSR model by leveraging neural codec language modeling to improve the reconstruction results, especially for the speaker similarity and prosody naturalness. Our proposed model consists of: (i) a multi-modal content encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual inputs; (ii) a speaker codec encoder to extract and normalize the speaker-aware codecs from the dysarthric speech, in order to provide original timbre and normal prosody; (iii) a codec language model based speech decoder to reconstruct the speech based on the extracted phoneme embeddings and normalized codecs. Evaluations on the commonly used UASpeech corpus show that our proposed model can achieve significant improvements in terms of speaker similarity and prosody naturalness.
On the Effectiveness of Acoustic BPE in Decoder-Only TTS
Authors: Bohan Li; Feiyu Shen; Yiwei Guo; Shuai Wang; Xie Chen; Kai Yu
Venue: Interspeech 2024 (main)

Abstract: Discretizing speech into tokens and generating them by a decoder-only model have been a promising direction for text-to-speech (TTS) and spoken language modeling (SLM). To shorten the sequence length of speech tokens, acoustic byte-pair encoding (BPE) has emerged in SLM that treats speech tokens from self-supervised semantic representations as characters to further compress the token sequence. But the gain in TTS has not been fully investigated, and the proper choice of acoustic BPE remains unclear. In this work, we conduct a comprehensive study on various settings of acoustic BPE to explore its effectiveness in decoder-only TTS models with semantic speech tokens. Experiments on LibriTTS verify that acoustic BPE uniformly increases the intelligibility and diversity of synthesized speech, while showing different features across BPE settings. Hence, acoustic BPE is a favorable tool for decoder-only TTS.
Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks
Authors: Kai-Wei Chang; Ming-Hao Hsu; Shan-Wen Li; Hung-yi Lee
Venue: Interspeech 2024 (main)

Abstract: Ever since the development of GPT-3 in the natural language processing (NLP) field, in-context learning (ICL) has played an essential role in utilizing large language models (LLMs). By presenting the LM utterance-label demonstrations at the input, the LM can accomplish few-shot learning without relying on gradient descent or requiring explicit modification of its parameters. This enables the LM to perform various downstream tasks in a black-box manner. Despite the success of ICL in NLP, little work is exploring the possibility of ICL in speech processing. This study is the first work exploring ICL for speech classification tasks with textless speech LM. We first show that the current speech LM lacks the ICL capability. We then perform warmup training on the speech LM, equipping the LM with demonstration learning capability. This paper explores and proposes the first speech LM capable of performing unseen classification tasks in an ICL manner.
Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models
Authors: Chun-Yi Kuan; Wei-Ping Huang; Hung-yi Lee
Venue: Interspeech 2024 (main)

Abstract: Large audio-language models (LALMs) enhance traditional large language models by integrating audio perception capabilities, allowing them to tackle audio-related tasks. Previous research has primarily focused on assessing the performance of LALMs across various tasks, yet overlooking their reliability, particularly concerning issues like object hallucination. In our study, we introduce methods to assess the extent of object hallucination of publicly available LALMs. Our findings reveal that LALMs are comparable to specialized audio captioning models in their understanding of audio content, but struggle to answer discriminative questions, specifically those requiring the identification of the presence of particular object sounds within an audio clip. This limitation highlights a critical weakness in current LALMs: their inadequate understanding of discriminative queries. Moreover, we explore the potential of prompt engineering to enhance LALMs' performance on discriminative questions.
Can Large Language Models Understand Spatial Audio?
Authors: Changli Tang; Wenyi Yu; Guangzhi Sun; Xianzhao Chen; Tian Tan; Wei Li; Jun Zhang; Lu Lu; Zejun Ma; Yuxuan Wang; Chao Zhang
Venue: Interspeech 2024 (main)

Abstract: This paper explores enabling large language models (LLMs) to understand spatial information from multichannel audio, a skill currently lacking in auditory LLMs. By leveraging LLMsâ advanced cognitive and inferential abilities, the aim is to enhance understanding of 3D environments via audio. We study 3 spatial audio tasks: sound source localization (SSL), far-field speech recognition (FSR), and localisation-informed speech extraction (LSE), achieving notable progress in each task. For SSL, our approach achieves an MAE of 2.70â¦ on the Spatial LibriSpeech dataset, substantially surpassing the prior benchmark of about 6.60â¦. Moreover, our model can employ spatial cues to improve FSR accuracy and execute LSE by selectively attending to sounds originating from a specified direction via text prompts, even amidst overlapping speech. These findings highlight the potential of adapting LLMs to grasp physical audio concepts, paving the way for LLM-based agents in 3D environments.
DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding
Authors: Suwon Shon; Kwangyoun Kim; Yi-Te Hsu; Prashant Sridhar; Shinji Watanabe; Karen Livescu
Venue: Interspeech 2024 (main)

Abstract: The integration of pre-trained text-based large language models (LLM) with speech input has enabled instruction-following capabilities for diverse speech tasks. This integration requires the use of a speech encoder, a speech adapter, and an LLM, trained on diverse tasks. We propose the use of discrete speech units (DSU), rather than continuous-valued speech encoder outputs, that are converted to the LLM token embedding space using the speech adapter. We generate DSU using a self-supervised speech encoder followed by k-means clustering. The proposed model shows robust performance on speech inputs from seen/unseen domains and instruction-following capability in spoken question answering. We also explore various types of DSU extracted from different layers of the self-supervised speech encoder, as well as Mel frequency Cepstral Coefficients (MFCC). Our findings suggest that the ASR task and datasets are not crucial in instruction-tuning for spoken question answering tasks.
DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
Authors: Ke-Han Lu; Zhehuai Chen; Szu-Wei Fu; He Huang; Boris Ginsburg; Yu-Chiang Frank Wang; Hung-yi Lee
Venue: Interspeech 2024 (main)

Abstract: Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language models (LLMs). In this paper, we propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities, enabling SLMs to interpret and generate comprehensive natural language descriptions, thereby facilitating the capability to understand both linguistic and non-linguistic features in speech. Enhanced with the proposed approach, our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. Moreover, we discover that the aligned model exhibits a zero-shot instruction-following capability without explicit speech instruction tuning. These findings highlight the potential to reshape instruction-following SLMs by incorporating rich, descriptive speech captions.
COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning
Authors: Jing Pan; Jian Wu; Yashesh Gaur; Sunit Sivasankaran; Zhuo Chen; Shujie Liu; Jinyu Li
Venue: Interspeech 2024 (main)

Abstract: We present a cost-effective method to integrate speech into a large language model (LLM), resulting in a Contextual Speech Model with Instruction-following/in-context-learning Capabilities (COSMIC) multi-modal LLM. Using GPT-3.5, we generate Speech Comprehension Test Question-Answer (SQA) pairs from speech transcriptions for supervised instruction tuning. With under 30 million trainable parameters and only 450 hours of English speech data, COSMIC demonstrates emerging capabilities in instruction-following and in-context learning. Equipped with such capabilities, COSMIC achieves a maximum 33.18 BLEU score in 0-shot EN-to-X speech to text translation (S2TT) and a significant boost in the 1-shot setting. Additionally, there is an average 25.8% relative Word Error Rate (WER) reduction for 1-shot cross-domain adaptation. COSMIC exhibits a significant automatic speech recognition (ASR) accuracy gain in contextual biasing tasks due to its instruction-following capability.