GenAI4ELab/papercli-papers huggingface dataset:
Cognitive Insights Across Languages: Enhancing Multimodal Interview Analysis
Authors: David Ortiz-Perez; Jose Garcia-Rodriguez; David TomÃ¡s
Venue: Interspeech 2024 (main)

Abstract: Cognitive decline is a natural process that occurs as individuals age. Early diagnosis of anomalous decline is crucial for initiating professional treatment that can enhance the quality of life of those affected. To address this issue, we propose a multimodal model capable of predicting Mild Cognitive Impairment and cognitive scores. The TAUKADIAL dataset is used to conduct the evaluation, which comprises audio recordings of clinical interviews. The proposed model demonstrates the ability to transcribe and differentiate between languages used in the interviews. Subsequently, the model extracts audio and text features, combining them into a multimodal architecture to achieve robust and generalized results. Our approach involves in-depth research to implement various features obtained from the proposed modalities.
Combining Acoustic Feature Sets for Detecting Mild Cognitive Impairment in the Interspeech'24 TAUKADIAL Challenge
Authors: GÃ¡bor Gosztolya; LÃ¡szlÃ³ TÃ³th
Venue: Interspeech 2024 (main)

Abstract: Shared tasks or challenges provide valuable opportunities for the machine learning community, as they offer a chance to compare the performance of machine learning approaches without peeking (due to the hidden test set). We present the approach of our team for the Interspeech'24 TAUKADIAL Challenge, where the task is to distinguish patients of Mild Cognitive Impairment (MCI) from healthy controls based on their speech. Our workflow focuses entirely on the acoustics, mixing standard feature sets (ComParE functionals and wav2vec2 embeddings) and custom attributes focusing on the amount of silent and filled pause segments. By training dedicated SVM classifiers on the three speech tasks and combining the predictions over the different speech tasks and feature sets, we obtained F1 values of up to 0.76 for the MCI identification task using cross-validation, while our RMSE scores for the MMSE estimation task were as low as 2.769 (cross-validation) and 2.608 (test).
Pre-trained Feature Fusion and Matching for Mild Cognitive Impairment Detection
Authors: Junwen Duan; Fangyuan Wei; Hong-Dong Li; Jin Liu
Venue: Interspeech 2024 (main)

Abstract: Effective diagnosis of Mild Cognitive Impairment (MCI), a preclinical stage of cognitive decline, is significant for delaying disease progression. While most current spontaneous speech-based diagnostic methods focus on English speech, the Interspeech 2024 TAUKADIAL Challenge proposed an innovative research direction to develop a language-agnostic approach to diagnose MCI. This paper proposes an MCI diagnosis method by analyzing and combining linguistic and acoustic features using the bilingual Chinese-English speech dataset provided by the challenge. We employed a pre-trained multilingual model and expressivity encoder to extract language-agnostic speech features. To overcome the challenges of data scarcity and language diversity, we implemented data augmentation and alignment to enhance the model's generalization. Our approach achieved 77.5% accuracy, demonstrating its effectiveness and potential on cross-lingual data.
The Interspeech 2024 TAUKADIAL Challenge: Multilingual Mild Cognitive Impairment Detection with Multimodal Approach
Authors: Benjamin Barrera-Altuna; Daeun Lee; Zaima Zarnaz; Jinyoung Han; Seungbae Kim
Venue: Interspeech 2024 (main)

Abstract: Mild cognitive impairment (MCI) and dementia significantly impact millions worldwide and rank as a major cause of mortality. Since traditional diagnostic methods are often costly and result in delayed diagnoses, many efforts have been made to propose automatic detection approaches. However, most methods focus on monolingual cases, limiting the scalability of their models to individuals speaking different languages. To understand the common characteristics of people with MCI speaking different languages, we propose a multilingual MCI detection model using multimodal approaches that analyze both acoustic and linguistic features. It outperforms existing machine learning models by identifying universal MCI indicators across languages. Particularly, we find that speech duration and pauses are crucial in detecting MCI in multilingual settings. Our findings can potentially facilitate early intervention in cognitive decline across diverse linguistic backgrounds.
Leveraging Universal Speech Representations for Detecting and Assessing the Severity of Mild Cognitive Impairment Across Languages
Authors: Anna Favaro; Tianyu Cao; Najim Dehak; Laureano Moro-Velazquez
Venue: Interspeech 2024 (main)

Abstract: This study examines the suitability of language-agnostic features for automatically detecting Mild Cognitive Impairment (MCI) and predicting Mini-Mental State Examination (MMSE) scores in a multilingual framework. We explored two methods for feature extraction: traditional feature engineering and pre-trained feature representation. We developed our models using the Interspeech 2024 Taukadial challenge data set, containing audios from subjects with MCI and controls in Chinese and English. Our top ensemble model achieved 75% accuracy in MCI detection and an RMSE of 2.44 in MMSE prediction in the testing set. Our results reveal the complementary nature of acoustic and linguistic representations and the existence of universal features that can be used cross-lingually. However, a statistical analysis of interpretable features did not show any shared speech patterns between the two languages, which can be attributed to differences in disease severity between the two cohorts of participants.
Translingual Language Markers for Cognitive Assessment from Spontaneous Speech
Authors: Bao Hoang; Yijiang Pang; Hiroko Dodge; Jiayu Zhou
Venue: Interspeech 2024 (main)

Abstract: Mild Cognitive Impairment (MCI) is considered a prodromal stage of dementia, including Alzheimerâs disease. It is characterized by behavioral changes and decreased cognitive function, while individuals can still maintain their independence. Early detection of MCI is critical, as it allows for timely intervention, enrichment of clinical trial cohorts, and the development of therapeutic approaches. Recently, language markers have been shown to be a promising approach to identifying MCI in a non-intrusive, affordable, and accessible fashion. In the InterSpeech 2024 TAUKADIAL Challenge, we study language markers from spontaneous speech in English and Chinese and use the bilingual language markers to identify MCI cases and predict the Mini-Mental Status Examination (MMSE) scores. Our proposed framework combines the power from 1) feature extraction of a comprehensive set of bilingual acoustic features, and semantic and syntactic features from language models; 2) careful treatment of model complexity for small sample size; 3) consideration of imbalanced demographic structure, potential outlier removal, and a multi-task treatment that uses the prediction of clinical classification as prior for MMSE prediction. The proposed approach delivers an average of 78.2% Balanced Accuracy in MCI detection and an averaged RMSE of 2.705 in predicting MMSE. Our empirical evaluation shows that translingual language markers can improve the detection of MCI from spontaneous speech. Our codes are provided in https://github.com/illidanlab/translingual-language-markers.
Multilingual Speech and Language Analysis for the Assessment of Mild Cognitive Impairment: Outcomes from the Taukadial Challenge
Authors: Paula Andrea PÃ©rez-Toro; Tomas Arias-Vergara; Philipp Klumpp; Tobias Weise; Maria Schuster; Elmar Noeth; Juan Rafael Orozco-Arroyave; Andreas Maier
Venue: Interspeech 2024 (main)

Abstract: Cognitive decline, a hallmark of several neurological conditions, including dementia and Alzheimer's disease, often manifests in noticeable changes in speech patterns and language use. Speech analysis in this context can serve as a valuable tool for early detection and monitoring of cognitive impairment. In this paper, we present the results of our attempts at the TAUKADIAL Challenge for automatically detecting people with mild cognitive impairment and predicting a cognitive score on English and Chinese speakers. In the classification task, we achieved a UAR of 83% using two language-dependent classifiers trained with timing and acoustic features. In the regression task, we obtained an RMSE of 1.87 using English speakers to train the base model with timing, acoustic, and language-dependent features.
Production of phrases by mechanical models of the human vocal tract
Authors: Takayuki Arai; Ryohei Suzuki; Chandler Earp; Shinya Tsuji; Keiko Ochi
Venue: Interspeech 2024 (main)

Abstract: Three types of mechanical models of the human vocal tract were used to produce utterances in English and Japanese. The first model, VTM-S24-1, is a type of sliding three-tube model in which the inner column only moves inside the outer straight pipe back and forth. In spite of this simple structure, we were able to produce âHow are you?â with this model and a single actuator. The second model, VTM-UT-D11, is based on the model that Umeda and Teranishi made. Eleven actuators control the positions of ten plastic bars in the main vocal tract and the velopharyngeal opening to adjust the vocal-tract configuration dynamically. Not only the English phrase âHow are you?â but also Japanese phrase for âGood morningâ were successfully produced. Finally, the third model, VTM-UT-D6, was newly designed. In this model, there are only six movable blocks inserted in the main vocal tract and no side branch is attached. With this model, a linear cam mechanism successfully
produced five vowels continuously. In addition, a rotating cam mechanism achieved âHow are you?â with high intelligibility.
Faster Vocoder: a multi threading approach to achieve low latency during TTS Inference
Authors: Vishal Gourav; Ankit Tyagi; Phanindra Mankale
Venue: Interspeech 2024 (main)

Abstract: The first question by any customer who wants to buy or rent any service/product is always, âHow fast is it?â. So how fast is our Text-To-Speech(TTS) we get asked, how fast can it turn my text into speech, what is the Customer Perceived Latency (CPL) i.e. total time taken from the time a customer submits text to the time they get the audio file for that text? Like any other service, TTS also comprises of a number of moving parts contributing to CPL, for example, SSML parser, Text Normaliser, G2P, Models (Acoustic Model and Vocoder) and post processing. Each layer contributes in some way to the overall processing time. The goal is to reduce this processing time without any deterioration in the audio quality.
A powerful and modern AAC composition tool for impaired speakers
Authors: Aanchan Mohan; Monideep Chakraborti; Katelyn Eng; Nailia Kushaeva; Mirjana Prpa; Jordan Lewis; Tianyi Zhang; Vince Geisler; Carol Geisler
Venue: Interspeech 2024 (main)

Abstract: Augmentative and alternative communication (AAC) software assists impaired speakers to communicate. Enabling context awareness, authenticity and ease of use goes a long way in empowering communication. Our work presents an easy to use AAC tool that allows for message composition with text or emoji input by typing as well as speaking with contextually relevant word-level suggestions. Any transcription errors are corrected, and contextually relevant phrases with the appropriately chosen emotional tone are suggested by large-language models. The user is then able to use modern text-to-speech synthesis to be able to synthesize the composed message in the userâs own voice. Our system additionally maintains features from AAC software such as word-cards, pre-composed and frequently used messages.
VoxFlow AI: wearable voice converter for atypical speech
Authors: Grzegorz P. Mika; Konrad ZieliÂ´nski; PaweÅ Cyrta; Marek Grzelec
Venue: Interspeech 2024 (main)

Abstract: The central challenges of the embedded voice conversion system lie in the transformation of atypical speech into one resembling the original voice without changing the linguistic content. While various atypical speech input techniques have been developed to aid voice inter-
actions, they are typically challenging to use daily, and the research on their usability could be more robust. In this Show&Tell paper, we
present VoxFlow AI, a wearable loudspeaker integrated with a voice converter designed to improve real-life interactions. During the interactive demonstration, we will show how this device changes interactions between people with oncological and neurological speech disorders, who are the direct beneficiaries of the solution.
Stress transfer in speech-to-speech machine translation
Authors: Sai Akarsh; Vamshiraghusimha Narasinga; Anil Kumar Vuppala
Venue: Interspeech 2024 (main)

Abstract: Indiaâs education sector faces a significant challenge due to its linguistic diversity, hindering inclusivity. The dominance of English on the internet underscores the critical need for translating educational content into Indian languages to enhance accessibility. Although Speech-to-Speech Machine Translation (SSMT) technologies exist, their deficiency in reproducing intonation results in monotonous translations, diminishing audience engagement and interest in the content. To address this issue, this paper demonstrates an SSMT application with a Text-to-Speech (TTS) architecture capable of incorporating stress into synthesized speech to give a more engaging experience. The SSMT pipeline also has components like a stress classifier that captures the stress in the source speech and allows it to be utilized during speech generation. The application takes in a speech file and gives a translated speech file with stress transferred from the source.
Mobile PresenTra: NICT fast neural text-to-speech system on smartphones with incremental inference of MS-FC-HiFi-GAN for law-latency synthesis
Authors: Takuma Okamoto; Yamato Ohtani; Hisashi Kawai
Venue: Interspeech 2024 (main)

Abstract: For achieving fast and high-fidelity neural text-to-speech on edge smartphone devices without network connection, we NICT prototyped Mobile PresenTra by introducing non-autoregressive acoustic model with Transformer encoder and ConvNeXt decoder, and MS-FC-HiFi-GAN neural vocoder.
Additionally, the incremental inference is applied only to neural vocoder for low-latency synthesis without performance degradation. Compared with a previous NICT system with Transformer encoder, Transforme decoder and MS-HiFi-GAN neural vocoder, the proposed Mobile PresenTra can realize high-fidelity and fast synthesis on a middle-range smartphone with a real-time factor of about 0.3 for batch inference, and a latency of less than 0.5 s for incremental inference. In the Show & Tell, attendees can freely experience the demonstration of Mobile PresenTra systems implemented on actual smartphones for English, Japanese and Chinese with arbitrary text input.
Multi-speaker and multi-dialectal Catalan TTS models for video gaming
Authors: Alex PeirÃ³-Lilja; JosÃ© Giraldo; MartÃ­ Llopart-Font; Carme Armentano-Oller; Baybars KÃ¼lebi; Mireia FarrÃºs
Venue: Interspeech 2024 (main)

Abstract: Recently, we explored and trained different state-of-the-art text-to-speech (TTS) architectures for Catalan. We used existing datasets but also produced a new Catalan multi-accent dataset to train these architectures. The objective of our work is to improve the quality of current TTS systems in Catalan and export the resulting models for potential interactive applications and video games. For this reason, our set of multi-speaker and multi-accent Catalan TTS models are presented within a demo made in Unity. The users are able to interact with game characters which are attached to our Catalan TTS. While generated Catalan speech replies are reproduced, execution time, real-time
factor and transcription are shown on screen. Exported weights, such as data and demo source code, are released to the public.
ConnecTone: a modular AAC system prototype with contextual generative text prediction and style-adaptive conversational TTS
Authors: Juliana Francis; Ãva SzÃ©kely; Joakim Gustafson
Venue: Interspeech 2024 (main)

Abstract: Recent developments in generative language modeling and conversational Text-to-Speech present transformative potential for enhancing Augmentative and Alternative Communication (AAC) devices. Practical application of these technologies requires extensive research and testing. To address this, we introduce ConnecTone, a modular platform designed for rapid integration and testing of language generation and speech technology. ConnecTone implements context-sensitive generative text prediction, using conversational context from Automatic
Speech Recognition inputs. The system incorporates a neural TTS that supports interpolation between reading and spontaneous conversational styles, along with adjustable prosodic features. These speech characteristics are predicted using Large Language Models, but can be adjusted by users for individual needs. We anticipate ConnecTone will enable us to rapidly evaluate and implement innovations, thereby contributing to faster benefit delivery to AAC users.
Reliable dialogue system for facilitating student-counselor communication
Authors: Mahdin Rohmatillah; Bryan Gautama Ngo; Willianto Sulaiman; Po-Chuan Chen; Jen-Tzung Chien
Venue: Interspeech 2024 (main)

Abstract: The mental well-being of university students is a critical concern, mainly due to the high student-to-counselor ratio in a university. This often results in lengthy waiting periods for students to receive in-person consultations with counselors, and suboptimal treatment due to counselorsâ unmanageable workloads. This paper proposes the advancement of a mental health dialogue system named NYCUKA to support both students and counselors. NYCUKA provides immediate aid by acting as an active listener to encourage students to share more about themselves during the waiting period. NYCUKA focuses primarily on the traditional Chinese language. All components are thoughtfully developed to ensure reliability aspect leading to safe communication. Conversation histories are stored in the database and analyzed using an advanced continual reasoning prompt for large language models. Conversation histories and analyses are displayed on the counselor dashboard to monitor
studentsâ mental health effectively.
CreakVC: a voice conversion tool for modulating creaky voice
Authors: Harm Lameris; Joakim Gustafson; Ãva SzÃ©kely
Venue: Interspeech 2024 (main)

Abstract: We introduce a human-in-the-loop one-shot voice conversion tool called CreakVC designed to modulate the level of creaky voice in the converted speech. Creaky voice, often used by speakers to convey sociolinguistic cues, presents challenges to speech processing due to its complex phonation characteristics. The primary goal of CreakVC is to enable in-depth research into how these cues are perceived, using systematic perceptual studies. CreakVC provides access to a diverse range of voice identities exhibiting creaky voice, while maintaining consistency in other parameters. We developed a spectrogram-frame level creak representation using CreaPy and finetuned FreeVC,
a one-shot voice conversion tool, by conditioning the speaker embedding and the self-supervised audio representation with the creak representation. An integrated plotting feature allows users to visualize and manipulate portions of speech for precise adjustments of creaky phonation levels. Beyond research, CreakVC has potential applications in voice-interactive systems and multimedia production.
EZTalking: English assessment platform for teachers and students
Authors: Yu-Sheng Tsao; Yung-Chang Hsu; Jiun-Ting Li; Siang-Hong Weng; Tien-Hong Lo; Berlin Chen
Venue: Interspeech 2024 (main)

Abstract: EZTalking is an AI-powered English learning platform designed to provide an effective and comfortable learning experience. Based on computer-assisted pronunciation training (CAPT) techniques, our platform features two main functions: (1) A CAPT system that quickly detects mispronunciations and provides immediate feedback to aid students in their pronunciation. (2) A learning portfolio platform that allows teachers to grasp each studentâs strengths and weaknesses, letting them create tailored exercises with real-time feedback on areas needing improvement. As students work through exercises in the EZTalking, they receive phoneme-level feedback, enabling them to refine their pronunciation and monitor their progress. Additionally, advanced students can take mock exams to assess their language proficiency thoroughly. With these features, EZTalking streamlines the teaching process and empowers students to reach their language learning goals with confidence.
Frontier of Frontend for Conversational Speech Processing
Authors: Shoko Araki
Venue: Interspeech 2024 (main)

Abstract: To deepen and enrich our daily communications, researchers have made significant efforts over several decades to develop technologies that can recognize and understand natural human conversations. Despite significant progress in both speech/language processing and speech enhancement technology, conversational speech processing remains challenging. Recordings of conversations with distant microphones contain ambient noise, reverberation, and speaker overlap that changes as the conversation progresses. Consequently, recognizing conversational speech is much more challenging than single-talker speech recognition, and frontend technologies such as speech enhancement and speaker diarization are essential to achieving highly accurate conversational speech processing. 
For more than two decades, the presenterâs research group has explored frontend techniques (source separation, dereverberation, noise reduction, and diarization) for handling realistic natural conversations with distant microphones. In this talk, I would like to talk about the evolution and frontier of frontend technologies for conversational signal processing. Specifically, we will trace the evolution of multichannel signal processing and neural network techniques, including beamforming and target speaker tracking and extraction, which have always played an important role in successive cutting-edge frontends, along with the latest achievements.
Mmm whatcha say? Uncovering distal and proximal context effects in first and second-language word perception using psychophysical reverse correlation
Authors: Paige TuttÃ¶sÃ­; H. Henny Yeung; Yue Wang; Fenqi Wang; Guillaume Denis; Jean-Julien Aucouturier; Angelica Lim
Venue: Interspeech 2024 (main)

Abstract: Acoustic context effects, where surrounding changes in pitch, rate or timbre influence the perception of a sound, are well documented in speech perception, but how they interact with lan- guage background remains unclear. Using a reverse-correlation approach, we systematically varied the pitch and speech rate in phrases around different pairs of vowels for second language (L2) speakers of English (/i/-/I/) and French (/u/-/y/), thus reconstructing, in a data-driven manner, the prosodic profiles that bias their perception. Testing English and French speakers (n=25), we showed that vowel perception is in fact influenced by conflicting effects from the surrounding pitch and speech rate: a congruent proximal effect 0.2s pre-target and a distal contrastive effect up to 1s before; and found that L1 and L2 speakers exhibited strikingly similar prosodic profiles in perception. We provide a novel method to investigate acoustic context effects across stimuli, timescales, and acoustic domain.