GenAI4ELab/papercli-papers huggingface dataset:
A Personalised Speech Communication Application for Dysarthric Speakers
Authors: Matthew Gibson; Ievgen Karaulov; Oleksii Zhelo; Filip Jurcicek
Venue: Interspeech 2023 (main)

Abstract: Individuals with impaired speech are often understood only by those familiar with their speech e.g. a care-giver or close family member. These impaired speakers are therefore highly dependent upon those familiar listeners for their spoken communication needs. These needs vary from basic expressions of hunger or thirst to much more advanced requirements like being understood at a work meeting. A significant subset of individuals with impaired speech also have reduced motor function which limits their mobility or dexterity. For this subset of individuals, the ability to communicate via the medium of speech is crucial. This paper describes a personalised speech communication application targeted towards English language speakers with impaired speech. This application enables the user to hold conversations with other humans, dictate text to a machine and participate in meetings via closed captioning.
Video Multimodal Emotion Recognition System for Real World Applications
Authors: Sun-Kyung Lee; Jong-Hwan Kim
Venue: Interspeech 2023 (main)

Abstract: This paper proposes a system capable of recognizing a speaker's utterance-level emotion through multimodal cues in a video. The system seamlessly integrates multiple AI models to first extract and pre-process multimodal information from the raw video input. Next, an end-to-end MER model sequentially predicts the speaker's emotions at the utterance level. Additionally, users can interactively demonstrate the system through the implemented interface.
Promoting Mental Self-Disclosure in a Spoken Dialogue System
Authors: Mahdin Rohmatillah; Bobbi Aditya; Li-Jen Yang; Bryan Gautama Ngo; Willianto Sulaiman; Jen-Tzung Chien
Venue: Interspeech 2023 (main)

Abstract: This paper proposes a mental health spoken dialogue to relax mental illness for university students by acting as an active listener to promote self-disclosure. The proposed system is designed for Mandarin with the specific accent and lexicon in Taiwan which is known as one of the underrepresented spoken languages. To achieve the objective, this work considers three key factors which are high quality speech components including automatic speech recognition and text-to-speech models, and the personalized responses while keeping the trustworthiness and seamless integration among dialogue system components.
"Select language, modality or put on a mask!" Experiments with Multimodal Emotion Recognition
Authors: PaweÅ Bujnowski; BartÅomiej KuÅºma; BartÅomiej Paziewski; Jacek Rutkowski; Joanna Marhula; Zuzanna Bordzicka; Piotr Andruszkiewicz
Venue: Interspeech 2023 (main)

Abstract: We propose a system designed for multimodal emotion recognition. Our research focuses on showing the impact of various signals in the emotion recognition process. Apart from reporting the average results of our models, we would like to encourage individual engagement of conference participants and explore how a unique emotional scene recorded on the spot can be interpreted by the models - for individual modalities as well as their combinations. Our models work for English, German and Korean. We show the comparison of emotion recognition accuracy for these 3 languages, including the influence of each modality. Our second experiment explores emotion recognition for people wearing face masks. We show that the use of face masks affects not only the video signal but also audio and text. To our knowledge, no other study shows the effects of wearing a mask for three modalities. Unlike other studies where masks are added artificially, we use real recordings with actors in masks.
My Vowels Matter: Formant Automation Tools for Diverse Child Speech
Authors: Hannah Valentine; Joel MacAuslan; Maria Grigos; Marisha Speights
Venue: Interspeech 2023 (main)

Abstract: Tools to automate formant measurement in vowels have been developed recently, but they have not been tested on pediatric speech samples. Critically, child speech includes unique acoustic challenges including high fundamental frequencies, wide formant bandwidths, more variable formant values, and increased subglottal coupling relative to adult speech. More importantly, these tools have not been tested on the diverse linguistic variations spoken by children. This study compares three tools for automatic formant estimation: Voweltine, Fast Track, and SpeechMark. The tools are tested on vowel productions from a young child with a speech sound disorder from a Black-identifying family. Benefits and tradeoffs of each automation tool are discussed.
NEMA: An Ecologically Valid Tool for Assessing Hearing Devices, Advanced Algorithms, and Communication in Diverse Listening Environments
Authors: Nicky Chong-White; Arun Sebastian; Jorge Mejia
Venue: Interspeech 2023 (main)

Abstract: Ecological Momentary Assessment (EMA) is valuable research method for evaluating the real-world performance of novel computational algorithms and device technologies, addressing the shortcomings of objective metrics and laboratory assessments. Our customisable, cloud-connected smartphone app, NEMA, gathers repeated self-reports and related acoustic features in users' natural environments, providing personalised insights on how specific technologies impact daily activities. NEMA has proven effective in assessing the real-world performance of novel hearing aid algorithms and features, while also improving our understanding of the challenges faced by those with hearing loss which drive new developments. This paper outlines NEMA's innovative features designed to facilitate efficient data collection and presents findings from a recent clinical trial where NEMA played a key role in providing real-world evidence of user benefits for a medical device seeking FDA approval.
When Words Speak Just as Loudly as Actions: Virtual Agent Based Remote Health Assessment Integrating What Patients Say with What They Do
Authors: Vikram Ramanarayanan; David Pautler; Lakshmi Arbatti; Abhishek Hosamath; Michael Neumann; Hardik Kothare; Oliver Roesler; Jackson Liscombe; Andrew Cornish; Doug Habberstad; Vanessa Richter; David Fox; David Suendermann-Oeft; Ira Shoulson
Venue: Interspeech 2023 (main)

Abstract: We present a unified multimodal dialog platform for the remote assessment and monitoring of patients' neurological and mental health. Tina, a virtual agent, guides participants through an immersive interaction wherein objective speech, facial, linguistic and cognitive biomarkers can be automatically computed from participant speech and video in near real time. Furthermore, Tina encourages participants to describe, in their own words, their most bothersome problems and what makes them better or worse, through the Patient Report of Problems (PROP) instrument. The PROP captures unfiltered verbatim replies of patients, in contrast with traditional patient reported outcomes that typically rely on categorical assessments. We argue that combining these patient reports (i.e., what they say) with objective biomarkers (i.e., how they say it and what they do) can greatly enhance the quality of telemedicine and improve the efficacy of siteless trials and digital therapeutic interventions.
Stuttering Detection Application
Authors: Kowshik Siva Sai Motepalli; Vamshiraghusimha Narasinga; Harsha Pathuri; Hina Khan; Sangeetha Mahesh; Ajish K. Abraham; Anil Kumar Vuppala
Venue: Interspeech 2023 (main)

Abstract: Stuttering is a prevalent speech disorder that affects millions of people worldwide. In this Show and Tell presentation, we demonstrate a novel platform that takes speech samples in English and Kannada to detect and analyze stuttering in patients. The user-friendly interface includes demographic details and speech samples, generating comprehensive reports for different stuttering disfluencies. The platform has four different user types, providing full read-only access for admins and full write access for super admins. Our platform provides valuable assistance for speech-language pathologists to evaluate speech samples. The proposed platform supports both live and recorded speech samples and presents a flexible approach to stuttering detection and analysis. Our research demonstrates the potential of technology to improve speech-language pathology for stuttering. Used F-score as a metric for evaluating the models for the stutter detection task.
Providing Interpretable Insights for Neurological Speech and Cognitive Disorders from Interactive Serious Games
Authors: Mario Zusag; Laurin Wagner
Venue: Interspeech 2023 (main)

Abstract: We propose an automated pipeline for robustly identifying neurological disorders from interactive therapeutic exercises, which are gathered via the mobile therapy app myReha. The app captures speech and cognitive parameters from over 30.000 tasks in various scenarios. Users get immediate and highly accurate feedback for pronunciation and coherency for language tasks, while voice recordings are fed to a feature extraction pipeline in the backend. These features are then used to construct speech characteristics, which are highly indicative of different neurological disorders, such as acquired aphasia after stroke. The data is visually presented in a web application nyra.insights, which allows medical professionals to quickly derive recommendations for treatment and closely monitor outcomes. During the Show and Tell session, users can experiment with the interactive myReha app and experience the real-time speech analysis capabilities via the nyra.insights web platform.
Automated Neural Nursing Assistant (ANNA): An Over-The-Phone System for Cognitive Monitoring
Authors: Jacob Solinsky; Raymond Finzel; Martin Michalowski; Serguei Pakhomov
Venue: Interspeech 2023 (main)

Abstract: ANNA is a telephony-based cognitive assessment tool designed to aid nurses in caring for patients who require close monitoring for the development of confusion or neurological impairment. Of particular concern is the treatment of Immune Effector Cell-Associated Neurotoxicity Syndrome (ICANS), a condition which occurs quite frequently as an adverse outcome of Chimeric Antigen Receptor-T (CAR-T) cancer immunotherapy. ANNA employs both traditional verbal tests for cognitive impairment and novel linguistic methods which identify abnormalities in the patient's speech during ordinary conversation. To collect ordinary speech it uses a lightweight instance of the Facebook's Large Language Model BlenderBot to engage the patient in a partially unscripted conversation. ANNA is designed with easy employment by healthcare providers in mind, being sufficiently lightweight to run on consumer-grade hardware and needing access only to a patient's phone number to interact with them.
5G-IoT Cloud based Demonstration of Real-Time Audio-Visual Speech Enhancement for Multimodal Hearing-aids
Authors: Ankit Gupta; Abhijeet Bishnu; Mandar Gogate; Kia Dashtipour; Tughrul Arslan; Ahsan Adeel; Amir Hussain; Tharmalingam Ratnarajah; Mathini Sellathurai
Venue: Interspeech 2023 (main)

Abstract: Over twenty percent of the world's population suffers from some form of hearing loss, making it one of the most significant public health challenges. Current hearing aids commonly amplify noises while failing to improve speech comprehension in crowded social settings. In this demonstration, we showcase a proof-of-concept implementation of the world's first 5G and Internet of Things (IoT) enabled multi-modal hearing aid (MM HA) prototype. This integrates an innovative 5G cloud-radio access network (C-RAN) and IoT based transceiver model for real-time audio-visual speech enhancement (AVSE). Specifically, we demonstrate a transceiver model for Cloud-based AVSE which satisfies high data rate and low latency requirements for future MM HAs. The innovative 5G-IoT transceiver application is shown to satisfy HA latency limitations while transmitting raw noisy AV data from an MM HA prototype device to the cloud for deep learning-based real-time AVSE processing and obtaining a clean audio signal.
Towards Two-point Neuron-inspired Energy-efficient Multimodal Open Master Hearing Aid
Authors: Mohsin Raza; Adewale Adetomi; Khubaib Ahmed; Amir Hussain; Tughrul Arslan; Ahsan Adeel
Venue: Interspeech 2023 (main)

Abstract: Here we demonstrate a two-point neuron-inspired audio-visual (AV) open Master Hearing Aid (OpenMHA) framework for on-chip energy-efficient speech enhancement (SE). The developed system is compared against state-of-the-art cepstrum-based audio-only (A-only) SE and conventional point-neuron inspired deep neural net (DNN) driven multimodal (MM) SE. Pilot experiments demonstrate that the proposed system outperforms audio-only SE in terms of speech quality and intelligibility and performs comparably to point neuron-inspired DNN with significantly reduced energy consumption at any time --- both during training and inferencing.
FC-MTLF: A Fine- and Coarse-grained Multi-Task Learning Framework for Cross-Lingual Spoken Language Understanding
Authors: Xuxin Cheng; Wanshi Xu; Ziyu Yao; Zhihong Zhu; Yaowei Li; Hongxiang Li; Yuexian Zou
Venue: Interspeech 2023 (main)

Abstract: Currently, zero-shot cross-lingual spoken language understanding (SLU) attracts increasing attention. Most of existing methods construct a mixed-language context via the code-switching approach. However, due to the different syntactic structures of each language, code-switching might fail to perform well and result in the loss of semantics. To address this issue, we propose a novel framework termed FC-MTLF, which applies a multi-task learning by introducing an auxiliary multilingual neural machine translation (NMT) task to compensate for the shortcomings of code-switching. In addition, we also adopt the curriculum learning strategy to further improve the performance. Experimental results show that our framework achieves the new state-of-the-art performance on the MultiATIS++ dataset. Further analysis verifies that our FC-MTLF can effectively transfer knowledge from source languages to target languages.
CÂ²A-SLU: Cross and Contrastive Attention for Improving ASR Robustness in Spoken Language Understanding
Authors: Xuxin Cheng; Ziyu Yao; Zhihong Zhu; Yaowei Li; Hongxiang Li; Yuexian Zou
Venue: Interspeech 2023 (main)

Abstract: Spoken language understanding (SLU) is a critical task in task-oriented dialogue systems. However, automatic speech recognition (ASR) errors often impair the understanding performance. Despite many previous models have obtained promising results for improving ASR robustness in SLU, most of them treat clean manual transcripts and ASR transcripts equally during the fine-tuning stage. To tackle this issue, in this paper, we propose a novel method termed CÂ²A-SLU. Specifically speaking, we add calculated cross attention to the original hidden states and apply contrastive attention to compare the input transcript with clean manual transcripts to distill the contrastive information, which can better capture distinctive features of ASR transcripts. Experiments on three datasets show that CÂ²A-SLU surpasses existing models and achieves a new state-of-the-art performance, with a relative improvement of 3.4% in terms of accuracy over the previous best model on SLURP dataset.
Tri-level Joint Natural Language Understanding for Multi-turn Conversational Datasets
Authors: Henry Weld; Sijia Hu; Siqu Long; Josiah Poon; Soyeon Han
Venue: Interspeech 2023 (main)

Abstract: Natural language understanding typically maps single utterances to a dual level semantic frame, sentence level intent and slot labels at the word level. The best performing models force explicit interaction between intent detection and slot filling. We present a novel tri-level joint natural language understanding approach, adding domain, and explicitly exchange semantic information between all levels. This approach enables the use of multi-turn datasets which are a more natural conversational environment than single utterance. We evaluate our model on two multi-turn datasets for which we are the first to conduct joint slot-filling and intent detection. Our model outperforms state-of-the-art joint models in slot filling and intent detection on multi-turn data sets. We provide an analysis of explicit interaction locations between the layers. We conclude that including domain information improves model performance.
Semantic Enrichment Towards Efficient Speech Representations
Authors: GaÃ«lle LaperriÃ¨re; Ha Nguyen; Sahar Ghannay; Bassam Jabaian; Yannick EstÃ¨ve
Venue: Interspeech 2023 (main)

Abstract: Over the past few years, self-supervised learned speech representations have emerged as fruitful replacements for conventional surface representations when solving Spoken Language Understanding (SLU) tasks. Simultaneously, multilingual models trained on massive textual data were introduced to encode language agnostic semantics. Recently, the SAMU-XLSR approach introduced a way to make profit from such textual models to enrich multilingual speech representations with language agnostic semantics. By aiming for better semantic extraction on a challenging Spoken Language Understanding task and in consideration with computation costs, this study investigates a specific in-domain semantic enrichment of the SAMU-XLSR model by specializing it on a small amount of transcribed data from the downstream task. In addition, we show the benefits of the use of same-domain French and Italian benchmarks for low-resource language portability and explore cross-domain capacities of the enriched SAMU-XLSR.
Tensor decomposition for minimization of E2E SLU model toward on-device processing
Authors: Yosuke Kashiwagi; Siddhant Arora; Hayato Futami; Jessica Huynh; Shih-Lun Wu; Yifan Peng; Brian Yan; Emiru Tsunoo; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: Spoken Language Understanding (SLU) is a critical speech recognition application and is often deployed on edge devices. Consequently, on-device processing plays a significant role in the practical implementation of SLU. This paper focuses on the end-to-end (E2E) SLU model due to its small latency property, unlike a cascade system, and aims to minimize the computational cost. We reduce the model size by applying tensor decomposition to the Conformer and E-Branchformer architectures used in our E2E SLU models. We propose to apply singular value decomposition to linear layers and the Tucker decomposition to convolution layers, respectively. We also compare COMP/PARFAC decomposition and Tensor-Train decomposition to the Tucker decomposition. Since the E2E model is represented by a single neural network, our tensor decomposition can flexibly control the number of parameters without changing feature dimensions. On the STOP dataset, we achieved 70.9% exact match accuracy under the tight constraint of only 15 million parameters.
DiffSLU: Knowledge Distillation Based Diffusion Model for Cross-Lingual Spoken Language Understanding
Authors: Tianjun Mao; Chenghong Zhang
Venue: Interspeech 2023 (main)

Abstract: Spoken language understanding (SLU) has achieved great success in high-resource languages, but it still remains challenging in the low-resource languages due to the scarcity of labeled training data. Hence, there is an increasing interest in zero-shot cross-lingual SLU. SLU typically has two subtasks, including intent detection and slot filling. Slots and intent in the same utterance are correlated, thus it is beneficial to achieve mutual guidance between them. In this paper, we propose a novel cross-lingual SLU framework termed DiffSLU, which leverages powerful diffusion model to enhance the mutual guidance. In addition, we also utilize knowledge distillation to facilitate knowledge transfer. Experimental results demonstrate that our DiffSLU can improve the performance compared with the strong baselines and achieves the new state-of-the-art performance on MultiATIS++ dataset, obtaining a relative improvement of 3.1% over the previous best model in overall accuracy.
Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understanding
Authors: Siddhant Arora; Hayato Futami; Yosuke Kashiwagi; Emiru Tsunoo; Brian Yan; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: There has been an increased interest in the integration of pretrained speech recognition (ASR) and language models (LM) into the SLU framework. However, prior methods often struggle with a vocabulary mismatch between pretrained models, and LM cannot be directly utilized as they diverge from its NLU formulation. In this study, we propose a three-pass end-to-end (E2E) SLU system that effectively integrates ASR and LM subnetworks into the SLU formulation for sequence generation tasks. In the first pass, our architecture predicts ASR transcripts using the ASR subnetwork. This is followed by the LM subnetwork, which makes an initial SLU prediction. Finally, in the third pass, the deliberation subnetwork conditions on representations from the ASR and LM subnetworks to make the final prediction. Our proposed three-pass SLU system shows improved performance over cascaded and E2E SLU models on two benchmark SLU datasets, SLURP and SLUE, especially on acoustically challenging utterances.
Contrastive Learning Based ASR Robust Knowledge Selection For Spoken Dialogue System
Authors: Zhiyuan Zhu; Yusheng Liao; Yu Wang; Yunfeng Guan
Venue: Interspeech 2023 (main)

Abstract: The construction of knowledge-based, task-oriented systems for spoken conversations is a challenging task. Given the spoken dialogue history information, a knowledge selection model selects the appropriate knowledge snippet from an unstructured knowledge base. However, the performance of this model is sensitive to automatic speech recognizer (ASR) recognition errors. To address this problem, we propose a method called CLKS, which develops a knowledge selection model that is robust to ASR recognition errors. This approach involves: 1) To leverage a wide range of information from various ASR outputs, we employ the self-attention mechanism to aggregate the representation of the N-best hypotheses of the dialogue history. 2) We use the written dialogue representation to guide the aggregated spoken dialogue representation to select the correct knowledge candidate through contrastive learning. Experimental results on the DSTC10 dataset demonstrate the effectiveness of our method.
Unsupervised Dialogue Topic Segmentation in Hyperdimensional Space
Authors: Seongmin Park; Jinkyu Seo; Jihwa Lee
Venue: Interspeech 2023 (main)

Abstract: We present HyperSeg, a hyperdimensional computing (HDC) approach to unsupervised dialogue topic segmentation. HDC is a class of vector symbolic architectures that leverages the probabilistic orthogonality of randomly drawn vectors at extremely high dimensions (typically over 10,000). HDC generates rich token representations through its low-cost initialization of many unrelated vectors. This is especially beneficial in topic segmentation, which often operates as a resource-constrained pre-processing step for downstream transcript understanding tasks. HyperSeg outperforms the current state-of-the-art in 4 out of 5 segmentation benchmarks -- even when baselines are given partial access to the ground truth -- and is 10 times faster on average. We show that HyperSeg also improves downstream summarization accuracy. With HyperSeg, we demonstrate the viability of HDC in a major language task. We open-source HyperSeg to provide a strong baseline for unsupervised topic segmentation.