GenAI4ELab/papercli-papers huggingface dataset:
Optimizing the role of human evaluation in LLM-based spoken document summarization systems
Authors: Margaret Kroll; Kelsey Kraus
Venue: Interspeech 2024 (main)

Abstract: The emergence of powerful LLMs has led to a paradigm shift in abstractive summarization of spoken documents. The properties that make LLMs so valuable for this task -- creativity, ability to produce fluent speech, and ability to abstract information from large corpora -- also present new challenges to evaluating their content. Quick, cost-effective automatic evaluations such as ROUGE and BERTScore offer promise, but do not yet show competitive performance when compared to human evaluations. We draw on methodologies from the social sciences to propose an evaluation paradigm for spoken document summarization explicitly tailored for generative AI content. We provide detailed evaluation criteria and best practices guidelines to ensure robustness in the experimental design, replicability, and trustworthiness of human evaluation studies. We additionally include two case studies that show how these human-in-the-loop evaluation methods have been implemented at a major U.S. technology company.
Key-Element-Informed sLLM Tuning for Document Summarization
Authors: Sangwon Ryu; Heejin Do; Yunsu Kim; Gary Geunbae Lee; Jungseul Ok
Venue: Interspeech 2024 (main)

Abstract: Remarkable advances in large language models (LLMs) have enabled high-quality text summarization. However, this capability is currently accessible only through LLMs of substantial size or proprietary LLMs with usage fees. In response, smaller-scale LLMs (sLLMs) of easy accessibility and low costs have been extensively studied, yet they often suffer from missing key information and entities, i.e., low relevance, in particular, when input documents are long. We hence propose a key-element-informed instruction tuning for summarization, so-called KEITSum, which identifies key elements in documents and instructs sLLM to generate summaries capturing these key elements. Experimental results on dialogue and news datasets demonstrate that sLLM with KEITSum indeed provides high-quality summarization with higher relevance and less hallucinations, competitive to proprietary LLM.
Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation
Authors: Kohei Matsuura; Takanori Ashihara; Takafumi Moriya; Masato Mimura; Takatomo Kano; Atsunori Ogawa; Marc Delcroix
Venue: Interspeech 2024 (main)

Abstract: This paper introduces a novel approach called sentence-wise speech summarization (Sen-SSum), which generates text summaries from a spoken document in a sentence-by-sentence manner. Sen-SSum combines the real-time processing of automatic speech recognition (ASR) with the conciseness of speech summarization. To explore this approach, we present two datasets for Sen-SSum: Mega-SSum and CSJ-SSum. Using these datasets, our study evaluates two types of Transformer-based models: 1) cascade models that combine ASR and strong text summarization models, and 2) end-to-end (E2E) models that directly convert speech into a text summary. While E2E models are appealing to develop compute-efficient models, they perform worse than cascade models. Therefore, we propose knowledge distillation for E2E models using pseudo-summaries generated by the cascade models. Our experiments show that this proposed knowledge distillation effectively improves the performance of the E2E model on both datasets.
An End-to-End Speech Summarization Using Large Language Model
Authors: Hengchao Shang; Zongyao Li; Jiaxin Guo; Shaojun Li; Zhiqiang Rao; Yuanchang Luo; Daimeng Wei; Hao Yang
Venue: Interspeech 2024 (main)

Abstract: Abstractive Speech Summarization (SSum) aims to generate human-like text summaries from spoken content. It encounters difficulties in handling long speech input and capturing the intricate cross-modal mapping between long speech inputs and short text summaries. Research on large language models (LLMs) and multimodal information fusion has provided new insights for addressing these challenges. In this paper, we propose an end-to-end SSum model that utilizes Q-Former as a connector for the audio-text modality and employs LLMs to generate text summaries directly from speech features. We adopt a multi-stage training approach that includes LLM based ASR and Text Summarization (TSum) tasks as auxiliary tasks. ASR tasks are used to align feature spaces and enhance the LLM's ability to handle longer speech. Then, we utilize a curriculum learning strategy to facilitate the model's transition from TSum to SSum. Finally, our model achieves competitive performance on the How-2 dataset.
Prompting Large Language Models with Audio for General-Purpose Speech Summarization
Authors: Wonjune Kang; Deb Roy
Venue: Interspeech 2024 (main)

Abstract: In this work, we introduce a framework for speech summarization that leverages the processing and reasoning capabilities of large language models (LLMs). We propose an end-to-end system that combines an instruction-tuned LLM with an audio encoder that converts speech into token representations that the LLM can interpret. Using a dataset with paired speech-text data, the overall system is trained to generate consistent responses to prompts with the same semantic information regardless of the input modality. The resulting framework allows the LLM to process speech inputs in the same way as text, enabling speech summarization by simply prompting the LLM. Unlike prior approaches, our method is able to summarize spoken content from any arbitrary domain, and it can produce summaries in different styles by varying the LLM prompting strategy. Experiments demonstrate that our approach outperforms a cascade baseline of speech recognition followed by LLM text processing.
Real-time Speech Summarization for Medical Conversations
Authors: Khai Le-Duc; Khai-Nguyen Nguyen; Long Vo-Dang; Truong-Son Hy
Venue: Interspeech 2024 (main)

Abstract: In doctor-patient conversations, identifying medically relevant information is crucial, posing the need for conversation summarization. In this work, we propose the first deployable real-time speech summarization system for real-world applications in industry, which generates a local summary after every N speech utterances within a conversation and a global summary after the end of a conversation. Our system could enhance user experience from a business standpoint, while also reducing computational costs from a technical perspective. Secondly, we present VietMed-Sum which, to our knowledge, is the first speech summarization dataset for medical conversations. Thirdly, we are the first to utilize LLM and human annotators collaboratively to create gold standard and synthetic summaries for medical conversation summarization. Finally, we present baseline results of state-of-the-art models on VietMed-Sum. All code, data (English-translated and Vietnamese) and models are available online.
Itâs Time to Take Action: Acoustic Modeling of Motor Verbs to Detect Parkinsonâs Disease
Authors: Daniel Escobar-Grisales; Cristian David RÃ­os-Urrego; Ilja Baumann; Korbinian Riedhammer; Elmar Noeth; Tobias Bocklet; Adolfo M. Garcia; Juan Rafael Orozco-Arroyave
Venue: Interspeech 2024 (main)

Abstract: Pre-trained models generate speech representations that are used in different tasks, including the automatic detection of Parkinsonâs disease (PD). Although these models can yield high accuracy, their interpretation is still challenging. This paper used a pre-trained Wav2vec 2.0 model to represent speech frames of 25ms length and perform a frame-by-frame discrimination between PD patients and healthy control (HC) subjects. This fine granularity prediction enabled us to identify specific linguistic segments with high discrimination capability. Speech representations of all produced verbs were compared w.r.t. nouns and the first ones yielded higher accuracies. To gain a deeper understanding of this pattern, representations of motor and non-motor verbs were compared and the first ones yielded better results, with accuracies of around 83% in an independent test set. These findings support well-established neurocognitive models about action-related language highlighted as key drivers of PD.
Towards objective and interpretable speech disorder assessment: a comparative analysis of CNN and transformer-based models
Authors: Malo Maisonneuve; Corinne Fredouille; Muriel Lalain; Alain Ghio; Virginie Woisard
Venue: Interspeech 2024 (main)

Abstract: Head and Neck Cancers (HNC) significantly impact patients' ability to speak, affecting their quality of life. Commonly used metrics for assessing pathological speech are subjective, prompting the need for automated and unbiased evaluation methods. This study proposes a self-supervised Wav2Vec2-based model for phone classification with HNC patients, to enhance accuracy and improve the discrimination of phonetic features for subsequent interpretability purpose. The impact of pre-training datasets, model size, and fine-tuning datasets and parameters are explored. Evaluation on diverse corpora reveals the effectiveness of the Wav2Vec2 architecture, outperforming a CNN-based approach, used in previous work. Correlation with perceptual measures also affirms the model relevance for impaired speech analysis. This work paves the way for better understanding of pathological speech with interpretable approaches for clinicians, by leveraging complex self-learnt speech representations.
Macro-descriptors for Alzheimer's disease detection using large language models
Authors: Catarina Botelho; John MendonÃ§a; Anna Pompili; Tanja Schultz; Alberto Abad; Isabel Trancoso
Venue: Interspeech 2024 (main)

Abstract: This work explores the potential of Large Language Models (LLMs) as annotators of high-level characteristics of speech transcriptions, which may be relevant for detecting Alzheimer's disease (AD). These low-dimension interpretable features, here designated as macro-descriptors (e.g. text coherence, lexical diversity), are then used to train a binary classifier. Our experiments compared the extraction of these features from both manual and automatic transcriptions obtained with different types of speech recognition systems, and involved both open and closed source LLMs, with several prompting strategies. The experiments also compared the use of macro-descriptors with the direct prediction of AD by the LLM, given the transcription. Even though LLMs are not trained for this task, our experiments show that they achieve up to 81% accuracy, surpassing the baseline of previous AD detection challenges, particularly when used as extractors of macro-descriptors.
Infusing Acoustic Pause Context into Text-Based Dementia Assessment
Authors: Franziska Braun; Sebastian P. Bayerl; Florian HÃ¶nig; Hartmut Lehfeld; Thomas Hillemacher; Tobias Bocklet; Korbinian Riedhammer
Venue: Interspeech 2024 (main)

Abstract: Speech pauses, alongside content and structure, offer a valuable and non-invasive biomarker for detecting dementia. This work investigates the use of pause-enriched transcripts in transformer-based language models to differentiate the cognitive states of subjects with no cognitive impairment, mild cognitive impairment, and Alzheimer's dementia based on their speech from a clinical assessment. We address three binary classification tasks: Onset, monitoring, and dementia exclusion. The performance is evaluated through experiments on a German Verbal Fluency Test and a Picture Description Test, comparing the model's effectiveness across different speech production contexts. Starting from a textual baseline, we investigate the effect of incorporation of pause information and acoustic context. We show the test should be chosen depending on the task, and similarly, lexical pause information and acoustic cross-attention contribute differently.
Towards Scalable Remote Assessment of Mild Cognitive Impairment Via Multimodal Dialog
Authors: Oliver Roesler; Jackson Liscombe; Michael Neumann; Hardik Kothare; Abhishek Hosamath; Lakshmi Arbatti; Doug Habberstad; Christiane Suendermann-Oeft; Meredith Bartlett; Cathy Zhang; Nikhil Sukhdev; Kolja Wilms; Anusha Badathala; Sandrine Istas; Steve Ruhmel; Bryan Hansen; Madeline Hannan; David Henley; Arthur Wallace; Ira Shoulson; David Suendermann-Oeft; Vikram Ramanarayanan
Venue: Interspeech 2024 (main)

Abstract: Early assessment of mild cognitive impairment (MCI) has the potential to expedite interventions and slow disease progress for people at risk of developing dementia. We investigate the feasibility of administering remote assessments of speech, orofacial and cognitive function to an elderly population with MCI via a cloud-based conversational remote monitoring platform, and the utility of automatically extracted multimodal biomarkers and self-reported problems in identifying MCI patients. We analyzed data from 90 MCI patients and 91 controls who each completed two assessments. 90% of participants reported excellent engagement and liked their overall user experience. Furthermore, combining multiple facial, speech and cognitive markers performed best at distinguishing MCI patients from controls with an AUC of 0.75 using a support vector machine classifier. Finally, we found that MCI patients reported significantly more problems related to memory, falls, anxiety and speech than controls.
Automatic recognition and detection of aphasic natural speech
Authors: Mara Barberis; Pieter De Clercq; Bastiaan Tamm; Hugo Van hamme; Maaike Vandermosten
Venue: Interspeech 2024 (main)

Abstract: Aphasia is a language disorder affecting one third of stroke patients. Current aphasia assessment does not consider natural speech due to the time consuming nature of manual transcriptions and a lack of knowledge on how to analyze such data. Here, we evaluate the potential of automatic speech recognition (ASR) to transcribe Dutch aphasic speech and the ability of natural speech features to detect aphasia. A picture-description task was administered and automatically transcribed in 62 persons with aphasia and 57 controls. Acoustic and linguistic features were semi-automatically extracted and provided as input to a support vector machine (SVM) classifier. Our ASR model obtained a WER of 24.5%, outperforming earlier ASR models for aphasia. The SVM shows high accuracy (86.6%) at the individual level, with fluency features as most dominant to detect aphasia. ASR and semi-automatic feature extraction can thus facilitate natural speech analysis in a time efficient manner in clinical practice.
When Whisper Listens to Aphasia: Advancing Robust Post-Stroke Speech Recognition
Authors: Giulia Sanguedolce; Sophie Brook; Dragos C. Gruia; Patrick A. Naylor; Fatemeh Geranmayeh
Venue: Interspeech 2024 (main)

Abstract: Despite recent advancements in Automatic Speech Recognition (ASR), its accuracy remains low for pathological speech, thereby limiting AI-based healthcare interventions in such settings. This work addresses this challenge by fine-tuning Whisper, an ASR known for its ability to capture high-dimensional features in healthy speech. Using our comprehensive dataset of patients with stroke, we fine-tuned Whisper and significantly reduced Word Error Rate (WER), surpassing previous work on severe aphasia. To demonstrate its generalisability, we tested the model on a separate database, AphasiaBank, and observed a lower WER despite variations in dialect, linguistics, and test protocols. Our result on the AphasiaBank was superior to previous ASRs trained on this database, confirming the generalisability of our approach. These outcomes not only address ASR limitations in impaired speech but also establish the foundations for standardised and versatile AI solutions for remote speech monitoring for timely diagnosis and intervention.
Automatic Prediction of Amyotrophic Lateral Sclerosis Progression using Longitudinal Speech Transformer
Authors: Liming Wang; Yuan Gong; Nauman Dawalatabad; Marco Vilela; Katerina Placek; Brian Tracey; Yishu Gong; Alan Premasiri; Fernando Vieira; James Glass
Venue: Interspeech 2024 (main)

Abstract: Automatic prediction of amyotrophic lateral sclerosis (ALS) disease progression provides a more efficient and objective alternative than manual approaches. We propose ALS longitudinal speech transformer (ALST), a neural network-based automatic predictor of ALS disease progression from longitudinal speech recordings of ALS patients. By taking advantage of high-quality pretrained speech features and longitudinal information in the recordings, our best model achieves 91.0%AUC, improving upon the previous best model by 5.6% relative on the ALS TDI dataset. Careful analysis reveals that ALST is capable of fine-grained, interpretable predictions of ALS progression.
How Consistent are Speech-Based Biomarkers in Remote Tracking of ALS Disease Progression Across Languages? A Case Study of English and Dutch
Authors: Hardik Kothare; Michael Neumann; Cathy Zhang; Jackson Liscombe; Jordi W J van Unnik; Lianne C M Botman; Leonard H van den Berg; Ruben P A van Eijk; Vikram Ramanarayanan
Venue: Interspeech 2024 (main)

Abstract: Previous work has demonstrated the utility of speech-based digital biomarkers for remotely tracking longitudinal progression in people with Amyotrophic Lateral Sclerosis (pALS). Here, we investigate the responsiveness of these biomarkers across languages for consistency. We collected audiovisual data using a cloud-based multimodal dialogue platform, where pALS interacted with a virtual guide to perform several speaking exercises. We automatically extracted speech, linguistic and orofacial metrics from 143 English-speaking pALS (36 bulbar onset, 107 non-bulbar onset) and 26 Dutch-speaking pALS (10 bulbar, 16 non-bulbar onset). We used growth curve models to estimate the trajectory of these metrics over time. We observe that for most of these metrics, English-speaking pALS and Dutch-speaking pALS follow similar trajectories, i.e. the slopes are not statistically different from each other, demonstrating the potential of such speech-based biomarkers for remote monitoring across languages.
âSo . . . my child . . . â â How Child ADHD Influences the Way Parents Talk
Authors: Anika A. Spiesberger; Andreas Triantafyllopoulos; Alexander Kathan; Anastasia Semertzidou; Caterina Gawrilow; Tilman Reinelt; Wolfgang A. Rauch; BjÃ¶rn Schuller
Venue: Interspeech 2024 (main)

Abstract: Attention-deficit/hyperactivity disorder (ADHD) exerts a psychological burden not only on affected individuals but also on their social support systems. Of particular interest are the parents, who often face challenges related to their childâs condition, including its impact on their own mental well-being. The interaction among the childâs symptomatology, parental mental health, and the parent-child relationship is a crucial area of investigation. Expressed Emotion (EE), as assessed through the Preschool Five Minute Speech Sample (PFMSS), serves as a valuable measure. However, manual annotation of EE can be cumbersome and impractical for continuous monitoring. To address this, we propose leveraging machine learning methods. This study presents an initial exploration into predicting childrenâs ADHD diagnosis using linguistic and paralinguistic features derived from the PFMSS. Despite achieving a UAR score of 67.1%, our results have not surpassed the performance of manually annotated EE.
Variability of speech timing features across repeated recordings: a comparison of open-source extraction techniques
Authors: Judith Dineley; Ewan Carr; Lauren L. White; Catriona Lucas; Zahia Rahman; Tian Pan; Faith Matcham; Johnny Downs; Richard J. Dobson; Thomas F. Quatieri; Nicholas Cummins
Venue: Interspeech 2024 (main)

Abstract: Variations in speech timing features have been reliably linked to symptoms of various health conditions, demonstrating clinical potential. However, replication challenges hinder their translation; extracted speech features are susceptible to methodological variations in the recording and processing pipeline. Investigating this, we compared exemplar timing features extracted via three different techniques from recordings of healthy speech. Our results show that features extracted via an intensity-based method differ from those produced by forced alignment. Different extraction methods also led to differing estimates of within-speaker feature variability over time in an analysis of recordings repeated systematically over three sessions in one day (n=26) and in one week (n=28). Our findings highlight the importance of feature extraction in study design and interpretation, and the need for consistent, accurate extraction techniques for clinical research.
Zero-Shot End-To-End Spoken Question Answering In Medical Domain
Authors: Yanis Labrak; Adel Moumen; Richard Dufour; Mickael Rouvier
Venue: Interspeech 2024 (main)

Abstract: In the rapidly evolving landscape of spoken question-answering (SQA), the integration of large language models (LLMs) has emerged as a transformative development. Conventional approaches often entail the use of separate models for question audio transcription and answer selection, resulting in significant resource utilization and error accumulation. To tackle these challenges, we explore the effectiveness of end-to-end (E2E) methodologies for SQA in the medical domain. Our study introduces a novel zero-shot SQA approach, compared to traditional cascade systems. Through a comprehensive evaluation conducted on a new open benchmark of 8 medical tasks and 48 hours of synthetic audio, we demonstrate that our approach requires up to 14.7 times fewer resources than a combined 1.3B parameters LLM with a 1.55B parameters ASR model while improving average accuracy by 0.5%. These findings underscore the potential of E2E methodologies for SQA in resource-constrained contexts.
Perceiver-Prompt: Flexible Speaker Adaptation in Whisper for Chinese Disordered Speech Recognition
Authors: Yicong Jiang; Tianzi Wang; Xurong Xie; Juan Liu; Wei Sun; Nan Yan; Hui Chen; Lan Wang; Xunying Liu; Feng Tian
Venue: Interspeech 2024 (main)

Abstract: Disordered speech recognition profound implications for improving the quality of life for individuals afflicted with, for example, dysarthria. Dysarthric speech recognition encounters challenges including limited data, substantial dissimilarities between dysarthric and non-dysarthric speakers, and significant speaker variations stemming from the disorder. This paper introduces Perceiver-Prompt, a method for speaker adaptation that utilizes P-Tuning on the Whisper large-scale model. We first fine-tune Whisper using LoRA and then integrate a trainable Perceiver to generate fixed-length speaker prompts from variable-length inputs, to improve model recognition of Chinese dysarthric speech. Experimental results from our Chinese dysarthric speech dataset demonstrate consistent improvements in recognition performance with Perceiver-Prompt. Relative reduction up to 13.04% in CER is obtained over the fine-tuned Whisper.
Custom wake word detection
Authors: Kesavaraj V; Charan Devarkonda; Vamshiraghusimha Narasinga; Anil Kumar Vuppala
Venue: Interspeech 2024 (main)

Abstract: In personalizing interactions with smart devices, identifying keywords in an open-vocabulary context is crucial. Previous methods for open-vocabulary keyword spotting relied on a shared embedding space created by audio and text encoders. However, they suffered from heterogeneous modality representations, causing audio-text mismatch. To tackle this issue, our proposed framework utilizes knowledge from a pre-trained text-to-speech (TTS) system. This knowledge transfer incorporates awareness of audio projections into text representations derived from the text encoder. Consequently, this approach aids in preventing false triggers in scenarios such as closely related pronunciations of audio-text pairs. Additionally, our proposed approach benefits from the keyword embedding calculation only during keyword enrollment phase. The proposed system gives consistent performance across all the word lengths.