GenAI4ELab/papercli-papers huggingface dataset:
Zero-Shot Fake Video Detection by Audio-Visual Consistency
Authors: Xiaolou Li; Zehua Liu; Chen Chen; Lantian Li; Li Guo; Dong Wang
Venue: Interspeech 2024 (main)

Abstract: Recent studies have advocated the detection of fake videos as a one-class detection task, predicated on the hypothesis that the consistency between audio and visual modalities of genuine data is more significant than that of fake data. This methodology, which solely relies on genuine audio-visual data while negating the need for forged counterparts, is thus delineated as a âzero-shotâ detection paradigm. This paper introduces a novel zero-shot detection approach anchored in content consistency across audio and video. By employing pre-trained ASR and VSR models, we recognize the audio and video content sequences, respectively. Then, the edit distance between the two sequences is computed to assess whether the claimed video is genuine. Experimental results indicate that, compared to two mainstream approaches based on semantic consistency and temporal consistency, our approach achieves superior generalizability across various deepfake techniques and demonstrates strong robustness against audio-visual perturbations. Finally, state-of the-art performance gains can be achieved by simply integratingthe decision scores of these three systems.
Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert
Authors: Han EunGi; Oh Hyun-Bin; Kim Sung-Bin; Corentin Nivelet Etcheberry; Suekyeong Nam; Janghoon Ju; Tae-Hyun Oh
Venue: Interspeech 2024 (main)

Abstract: Speech-driven 3D facial animation has recently garnered attention due to its cost-effective usability in multimedia production. However, most current advances overlook the intelligibility of lip movements, limiting the realism of facial expressions. In this paper, we introduce a method for speech-driven 3D facial animation to generate accurate lip movements, proposing an audio-visual multimodal perceptual loss. This loss provides guidance to train the speech-driven 3D facial animators to generate plausible lip motions aligned with the spoken transcripts. Furthermore, to incorporate the proposed audio-visual perceptual loss, we devise an audio-visual lip reading expert leveraging its prior knowledge about correlations between speech and lip motions. We validate the effectiveness of our approach through broad experiments, showing noticeable improvements in lip synchronization and lip readability performance. Codes are available at https://3d-talking-head-avguide.github.io/.
Autoregressive cross-interlocutor attention scores meaningfully capture conversational dynamics
Authors: Matthew McNeill; Rivka Levitan
Venue: Interspeech 2024 (main)

Abstract: This paper analyzes attention scores over a conversational partner's historical turns trained with an autoregressive prosodic objective. Following a qualitative observation that these attention scores seem to organize dialogue history into topic segments, we demonstrate that they capture meaningful dialogue structure based on several quantitative measures. This finding has implications for spoken dialogue system design and analysis of entrainment and conversational dynamics in human-human and human-machine communication.
ConvoCache: Smart Re-Use of Chatbot Responses
Authors: Conor Atkins; Ian Wood; Mohamed Ali Kaafar; Hassan Asghar; Nardine Basta; Michal Kepkowski
Venue: Interspeech 2024 (main)

Abstract: We present ConvoCache, a conversational caching system that solves the problem of slow and expensive generative AI models in spoken chatbots. ConvoCache finds a semantically similar prompt in the past and reuses the response. In this paper we evaluate ConvoCache on the DailyDialog dataset. We find that ConvoCache can apply a UniEval coherence threshold of 90% and respond to 89% of prompts using the cache with an average latency of 214ms, replacing LLM and voice synthesis that can take over 1s. To further reduce latency we test prefetching and find limited usefulness. Prefetching with 80% of a request leads to a 63% hit rate, and a drop in overall coherence. ConvoCache can be used with any chatbot to reduce costs by reducing usage of generative AI by up to 89%.
Joint Learning of Context and Feedback Embeddings in Spoken Dialogue
Authors: Livia Qian; Gabriel Skantze
Venue: Interspeech 2024 (main)

Abstract: Short feedback responses, such as backchannels, play an important role in spoken dialogue. So far, most of the modeling of feedback responses has focused on their timing, often neglecting how their lexical and prosodic form influence their contextual appropriateness and conversational function. In this paper, we investigate the possibility of embedding short dialogue contexts and feedback responses in the same representation space using a contrastive learning objective. In our evaluation, we primarily focus on how such embeddings can be used as a context-feedback appropriateness metric and thus for feedback response ranking in U.S. English dialogues. Our results show that the model outperforms humans given the same ranking task and that the learned embeddings carry information about the conversational function of feedback responses.
DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing
Authors: Neha Sahipjohn; Ashishkumar Gudmalwar; Nirmesh Shah; Pankaj Wasnik; Rajiv Ratn Shah
Venue: Interspeech 2024 (main)

Abstract: Audio-visual alignment after dubbing is a challenging research problem. To this end, we propose a novel method, DubWise: Multi-modal Large Language Model (LLM)-based Text-to-Speech (TTS), which can control the speech duration of synthesized speech in such a way that it aligns well with the speakerâs lip movements given in the reference video even when the spoken text is different or in a different language. To accomplish this, we propose to utilize cross-modal attention techniques in a pre-trained GPT-based TTS. We combine linguistic tokens from text, speaker identity tokens via a voice cloning network, and video tokens via a proposed duration controller network. We demonstrate the effectiveness of our system on the Lip2Wav-Chemistry and LRS2 datasets. Also, the proposed method achieves improved lip sync and naturalness compared to the SOTAs for the same language but different text (i.e., non-parallel) and the different language, different text (i.e., cross-lingual) scenarios.
Contextual Interactive Evaluation of TTS Models in Dialogue Systems
Authors: Siyang Wang; Ãva SzÃ©kely; Joakim Gustafson
Venue: Interspeech 2024 (main)

Abstract: Evaluation of text-to-speech (TTS) models is currently dominated by Mean-Opinion-Score (MOS) listening test, but MOS has been increasingly questioned for its validity. MOS tests place listeners in a passive setup, in which they do not actively interact with the TTS and usually evaluate isolated utterances without context. Thus it gives no indication for how well a TTS model suits an interactive application like spoken dialogue system, in which the capability of generating appropriate speech in the dialogue context is paramount. We aim to take a first step towards addressing this shortcoming by evaluating several state-of-the-art neural TTS models, including one that adapts to dialogue context, in a custom-built spoken dialogue system. We present system design, experiment setup, and results. Our work is the first to evaluate TTS in contextual dialogue system interactions. We also discuss the shortcomings and future opportunities of the proposed evaluation paradigm.
GSQA: An End-to-End Model for Generative Spoken Question Answering
Authors: Min-Han Shih; Ho-Lam Chung; Yu-Chi Pai; Ming-Hao Hsu; Guan-Ting Lin; Shang-Wen Li; Hung-yi Lee
Venue: Interspeech 2024 (main)

Abstract: In recent advancements in spoken question answering (SQA), endto-end models have made significant strides. However, previous research has primarily focused on extractive span selection. While this extractive-based approach is effective when answers are present directly within the input, it falls short in addressing abstractive questions, where answers are not directly extracted but inferred from the given information. To bridge this gap, we introduce the first end-to-end Generative Spoken Question Answering (GSQA) model that empowers the system to engage in abstractive reasoning. The challenge in training our GSQA model lies in the absence of a spoken abstractive QA dataset. We propose using text models for initialization and leveraging the extractive QA dataset to transfer knowledge from the text generative model to the spoken generative model. Experimental results indicate that our model surpasses the previous extractive model by 3% on extractive QA datasets. Furthermore, the GSQA model has only been fine-tuned on the spoken extractive QA dataset. Despite not having seen any spoken abstractive QA data, it can still closely match the performance of the cascade model. In conclusion, our GSQA model shows the potential to generalize to a broad spectrum of questions, thus further expanding the SQA capabilities of abstractive QA.
Resource-Efficient Speech Quality Prediction through Quantization Aware Training and Binary Activation Maps
Authors: Mattias Nilsson; Riccardo Miccini; Clement Laroche; Tobias Piechowiak; Friedemann Zenke
Venue: Interspeech 2024 (main)

Abstract: As speech processing systems in mobile and edge devices become more commonplace, the demand for unintrusive speech quality monitoring increases. Deep learning methods provide high-quality estimates of objective and subjective speech quality metrics. However, their significant computational requirements are often prohibitive on resource-constrained devices. To address this issue, we investigated binary activation maps (BAMs) for speech quality prediction on a convolutional architecture based on DNSMOS. We show that the binary activation model with quantization aware training matches the predictive performance of the baseline model. It further allows using other compression techniques. Combined with 8-bit weight quantization, our approach results in a 25-fold memory reduction during inference, while replacing almost all dot products with summations. Our findings show a path toward substantial resource savings by supporting mixed-precision binary multiplication in hard- and software.
Towards interfacing large language models with ASR systems using confidence measures and prompting
Authors: Maryam Naderi; Enno Hermann; Alexandre Nanchen; Sevada Hovsepyan; Mathew Magimai.-Doss
Venue: Interspeech 2024 (main)

Abstract: As large language models (LLMs) grow in parameter size and capabilities, such as interaction through prompting, they open up new ways of interfacing with automatic speech recognition (ASR) systems beyond rescoring n-best lists. This work investigates post-hoc correction of ASR transcripts with LLMs. To avoid introducing errors into likely accurate transcripts, we propose a range of confidence-based filtering methods. Our results indicate that this can improve the performance of less competitive ASR systems.
Text Injection for Neural Contextual Biasing
Authors: Zhong Meng; Zelin Wu; Rohit Prabhavalkar; Cal Peyser; Weiran Wang; Nanxin Chen; Tara N. Sainath; Bhuvana Ramabhadran
Venue: Interspeech 2024 (main)

Abstract: Neural contextual biasing effectively improves automatic
speech recognition (ASR) for crucial phrases within a speakerâs
context, particularly those that are infrequent in the training
data. This work proposes contextual text injection (CTI) to 
enhance contextual ASR. CTI leverages not only the paired 
speech-text data, but also a much larger corpus of unpaired 
text to optimize the ASR model and its neural biasing component.
Unpaired text is converted into speech-like representations 
and used to guide the modelâs attention towards relevant 
bias phrases. Moreover, we introduce a contextual text-injected
(CTI) minimum word error rate (MWER) training, which minimizes
the expected WER caused by contextual biasing when
 unpaired text is injected into the model. Experiments show that
CTI with 100 billion text sentences can achieve up to 43.3% relative
 WER reduction from a strong neural biasing model. CTIMWER
provides a further relative improvement of 23.5%.
Prompting Large Language Models with Mispronunciation Detection and Diagnosis Abilities
Authors: Minglin Wu; Jing Xu; Xixin Wu; Helen Meng
Venue: Interspeech 2024 (main)

Abstract: Large Language Models (LLMs) have demonstrated significant achievements across diverse modalities. In this paper, we propose ATP-LLM, a framework that utilizes Audio and Text to Prompt LLMs to perform mispronunciation detection and diagnosis (MDD) tasks in second language (L2) English. ATP-LLM consists of an audio encoder and an LLM decoder. The audio encoder converts L2 English speech into speech representations digestible for LLMs. These speech representations, along with the corresponding canonical pronunciation, serve as audio and text prompts that enable the LLM decoder to generate the phones articulated by L2 English learners. Experiments show that our proposed ATP-LLM achieves a new state-of-the-art (SOTA) performance on the CU-CHLOE corpus with a Phone Error Rate (PER) of 8.56% and an F1 of 82.02%, outperforming the existing wav2vec2-CTC method whose PER and F1 are 8.98% and 80.93%, respectively.
Acceleration of Posteriorgram-based DTW by Distilling the Class-to-class Distances Encoded in the Classifier Used to Calculate Posteriors
Authors: Haitong Sun; Jaehyun Choi; Nobuaki Minematsu; Daisuke Saito
Venue: Interspeech 2024 (main)

Abstract: In media technology, comparison of a sequential data with another is a fundamental technique for many applications, and dynamic time warping is often conducted. Conventionally, two sequences of raw features were compared, and recently, more abstract representations are used, which are obtained with deep neural networks. One of these representations is posteriorgram, where each frame is composed of n-dimensional class posteriors, and frame-to-frame distance is often calculated using a divergence metric such as Bhattacharyya distance. In this study, a novel method is proposed to distill the class-to-class distances encoded in any classifier used to calculate posteriors, and the distances are effectively used to accelerate posteriorgram DTW by approximating it as DTW between two sequences of most likely classes. Our proposal shows a much faster and even stabler performance and guarantees no requirement of calculating frame-to-frame distances on the fly during DTW processing.
VECL-TTS: Voice identity and Emotional style controllable Cross-Lingual Text-to-Speech
Authors: Ashishkumar Gudmalwar; Nirmesh Shah; Sai Akarsh; Pankaj Wasnik; Rajiv Ratn Shah
Venue: Interspeech 2024 (main)

Abstract: Despite the significant advancements in Text-to-Speech (TTS) systems, their full utilization in automatic dubbing remains limited. This task necessitates the extraction of voice identity and emotional style from a reference speech in a source language and subsequently transferring them to a target language using cross-lingual TTS techniques. While previous approaches have mainly concentrated on controlling voice identity within the cross-lingual TTS framework, there has been limited work on incorporating emotion and voice identity together. To this end, we introduce an end-to-end Voice Identity and Emotional Style Controllable Cross-Lingual (VECL) TTS system using multilingual speakers and an emotion embedding network. Moreover, we introduce content and style consistency losses to enhance the quality of synthesized speech further. The proposed system achieved an average relative improvement of 8.83% compared to the state-of-the-art (SOTA) methods on a database comprising English and three Indian languages (Hindi, Telugu, and Marathi).
Sparse Binarization for Fast Keyword Spotting
Authors: Jonathan Svirsky; Uri Shaham; Ofir Lindenbaum
Venue: Interspeech 2024 (main)

Abstract: With the increasing prevalence of voice-activated devices and applications, keyword spotting (KWS) models enable users to interact with technology hands-free, enhancing convenience and accessibility in various contexts. Deploying KWS models on edge devices, such as smartphones and embedded systems, offers significant benefits for real-time applications, privacy, and bandwidth efficiency. However, these devices often possess limited computational power and memory. This necessitates optimizing neural network models for efficiency without significantly compromising their accuracy. To address these challenges, we propose a novel keyword-spotting model based on sparse input representation followed by a linear classifier. The model is four times faster than the previous state-of-the-art edge device-compatible model with better accuracy. We show that our method is also more robust in noisy environments while being fast. Our code is available at: https://github.com/jsvir/sparknet.
Quantifying the effect of speech pathology on automatic and human speaker verification
Authors: Bence Mark Halpern; Thomas Tienkamp; Wen-Chin Huang; Lester Phillip Violeta; Teja Rebernik; Sebastiaan de Visscher; Max Witjes; Martijn Wieling; Defne Abur; Tomoki Toda
Venue: Interspeech 2024 (main)

Abstract: This study investigates how surgical intervention for speech pathology (specifically, as a result of oral cancer surgery) impacts the performance of an automatic speaker verification (ASV) system. Using two recently collected Dutch datasets with parallel pre and post-surgery audio from the same speaker, NKI-OC-VC and SPOKE, we assess the extent to which speech pathology influences ASV performance, and whether objective/subjective measures of speech severity are correlated with the performance. Finally, we carry out a perceptual study to compare judgements of ASV and human listeners. Our findings reveal that pathological speech negatively affects ASV performance, and the severity of the speech is negatively correlated with the performance. There is a moderate agreement in perceptual and objective scores of speaker similarity and severity, however, we could not clearly establish in the perceptual study, whether the same phenomenon also exists in human perception.
Investigation of Layer-Wise Speech Representations in Self-Supervised Learning Models: A Cross-Lingual Study in Detecting Depression
Authors: Bubai Maji; Rajlakshmi Guha; Aurobinda Routray; Shazia Nasreen; Debabrata Majumdar
Venue: Interspeech 2024 (main)

Abstract: Automated depression detection (ADD) from speech signals allows early identification and intervention, reducing costs to medical healthcare. However, most of the existing ADD studies are trained and evaluated on a single language corpus with a lack of sufficient training data. These limits the generalizability of models in other demographic groups in distinct languages. In this study, Semi-Supervised Learning (SSL) was applied to depression detection on two different language datasets. We evaluate the HuBERT and WavLM models in single-language, mixed-language, and cross-language scenarios to investigate the generalization to diverse populations at different recording environments. Moreover, we thoroughly analyzed layer-wise performance in the upstream model and pooling methods (i.e. max and mean pooling) in the downstream task. The results show that the WavLM features generalize better than the HuBERT features. Our best model surpasses previous works in the frozen upstream conditions.
Detection of Cognitive Impairment And Alzheimer's Disease Using a Speech- and Language-Based Protocol
Authors: Tanya Talkar; Sherman Charles; Chelsea Krantsevich; Kan Kawabata
Venue: Interspeech 2024 (main)

Abstract: Alzheimerâs Disease (AD) is a neurodegenerative disease that primarily affects cognitive functions and memory loss. The maximum effectiveness of medications rely on early detection of the presence of AD, which requires invasive extensive blood tests and/or brain imaging. In this study, we present a speech-based tool to detect the presence of mild cognitive impairment (MCI) and probable AD (pAD). Through the Speech Vitals application, in conjunction with the Bio-Hermes study, we administer a set of speech and language-based tasks targeting motor and cognitive impairments. We extract metrics from speech and language tasks to generate models that discriminate between healthy controls and individuals with MCI or pAD with an AUC of 0.83, and between controls and individuals with pAD with an AUC of 0.90. These models highlight the promise of a speech-based tool for screening cognitive impairment and AD, which could help inform individuals when they should seek additional diagnostic tools.
Analyzing Multimodal Features of Spontaneous Voice Assistant Commands for Mild Cognitive Impairment Detection
Authors: Nana Lin; Youxiang Zhu; Xiaohui Liang; John A. Batsis; Caroline Summerour
Venue: Interspeech 2024 (main)

Abstract: Mild cognitive impairment (MCI) is a major public health concern due to its high risk of progressing to dementia. This study investigates the potential of detecting MCI with spontaneous voice assistant (VA) commands from 35 older adults in a controlled setting. Specifically, a command-generation task is designed with pre-defined intents for participants to freely generate commands that are more associated with cognitive ability than read commands. We develop MCI classification and regression models with audio, textual, intent, and multimodal fusion features. We find the command-generation task outperforms the command-reading task with an average classification accuracy of 82%, achieved by leveraging multimodal fusion features. In addition, generated commands correlate more strongly with memory and attention subdomains than read commands. Our results confirm the effectiveness of the command-generation task and imply the promise of using longitudinal in-home commands for MCI detection.
Prosody-Driven Privacy-Preserving Dementia Detection
Authors: Dominika Woszczyk; Ranya Aloufi; Soteris Demetriou
Venue: Interspeech 2024 (main)

Abstract: Speaker embeddings extracted from voice recordings have been proven valuable for dementia detection. However, by their nature, these embeddings contain identifiable information which raises privacy concerns. In this work, we aim to anonymize embeddings while preserving the diagnostic utility for dementia detection. Previous studies rely on adversarial learning and models trained on the target attribute and struggle in limited-resource settings. We propose a novel approach that leverages domain knowledge to disentangle prosody features relevant to dementia from speaker embeddings without relying on a dementia classifier. Our experiments show the effectiveness of our approach in preserving speaker privacy (speaker recognition F1-score .01%) while maintaining high dementia detection score F1-score of 74% on the ADReSS dataset. Our results are also on par with a more constrained classifier-dependent system on ADReSSo (.01% and .66%), and have no impact on synthesized speech naturalness.
Voice Disorder Analysis: a Transformer-based Approach
Authors: Alkis Koudounas; Gabriele Ciravegna; Marco Fantini; Erika Crosetti; Giovanni Succo; Tania Cerquitelli; Elena Baralis
Venue: Interspeech 2024 (main)

Abstract: Voice disorders are pathologies significantly affecting patient quality of life. However, non-invasive automated diagnosis of these pathologies is still under-explored, due to both a shortage of pathological voice data, and diversity of the recording types used for the diagnosis. This paper proposes a novel solution that adopts transformers directly working on raw voice signals and addresses data shortage through synthetic data generation and data augmentation. Further, we consider many recording types at the same time, such as sentence reading and sustained vowel emission, by employing a Mixture of Expert ensemble to align the predictions on different data types. The experimental results, obtained on both public and private datasets, show the effectiveness of our solution in the disorder detection and classification tasks and largely improve over existing approaches.