GenAI4ELab/papercli-papers huggingface dataset:
Developing an End-to-End Framework for Predicting the Social Communication Severity Scores of Children with Autism Spectrum Disorder
Authors: Jihyun Mun; Sunhee Kim; Minhwa Chung
Venue: Interspeech 2024 (main)

Abstract: Autism Spectrum Disorder (ASD) is a lifelong condition that significantly influencing an individual's communication abilities and their social interactions. Early diagnosis and intervention are critical due to the profound impact of ASD's characteristic behaviors on foundational developmental stages. However, limitations of standardized diagnostic tools necessitate the development of objective and precise diagnostic methodologies. This paper proposes an end-to-end framework for automatically predicting the social communication severity of children with ASD from raw speech data. This framework incorporates an automatic speech recognition model, fine-tuned with speech data from children with ASD, followed by the application of fine-tuned pre-trained language models to generate a final prediction score. Achieving a Pearson Correlation Coefficient of 0.6566 with human-rated scores, the proposed method showcases its potential as an accessible and objective tool for the assessment of ASD.
Multimodal Fusion for Vocal Biomarkers Using Vector Cross-Attention
Authors: Vladimir Despotovic; Abir ElbÃ©ji; Petr V. Nazarov; Guy Fagherazzi
Venue: Interspeech 2024 (main)

Abstract: Vocal biomarkers are measurable characteristics of person's voice that provide valuable insights into various aspects of their physiological and psychological state, or health status. The use of standardized voice tasks, such as reading, counting, or sustained vowel phonation are common in vocal biomarker research, but semi-spontaneous tasks where the person is instructed to talk about a particular topic, or spontaneous speech are also increasingly used. However, limited efforts were made to combine multiple voice modalities. In this paper, we propose a simple, yet efficient approach of fusing multiple standardized voice tasks based on vector cross-attention, showing improved predictive capacity for derived vocal biomarkers in comparison to single modalities. The multimodal approach is tested on the assessment of respiratory quality of life from reading and sustained vowel phonation recordings, outperforming single modalities up to 4.2% in terms of accuracy (relative increase of 7%).
Revealing Confounding Biases: A Novel Benchmarking Approach for Aggregate-Level Performance Metrics in Health Assessments
Authors: Stefano Goria; Roseline Polle; Salvatore Fara; Nicholas Cummins
Venue: Interspeech 2024 (main)

Abstract: Numerous speech-based health assessment studies report high accuracy rates for machine learning models which detect conditions such as depression and Alzheimerâs disease. There are growing concerns that these reported performances are often overestimated, especially in small-scale cross-sectional studies. Possible causes for this overestimation include overfitting, publication biases and a lack of standard procedures to report findings and testing methodology. Another key source of misrepresentation is the reliance on aggregate-level performance metrics. Speech is a highly variable signal that can be affected by factors including age, sex, and accent, which can easily bias models. We highlight this impact by presenting a simple benchmark model for assessing the extent to which aggregate metrics exaggerate the efficacy of a machine learning model in the presence of confounders. We then demonstrate the usefulness of this model on exemplar speech-health assessment datasets.
Developing Multi-Disorder Voice Protocols: A team science approach involving clinical expertise, bioethics, standards, and DEI.
Authors: AnaÃ¯s Rameau; Satrajit Ghosh; Alexandros Sigaras; Olivier Elemento; Jean-Christophe Belisle-Pipon; Vardit Ravitsky; Maria Powell; Alistair Johnson; David Dorr; Philip Payne; Micah Boyer; Stephanie Watts; Ruth Bahr; Frank Rudzicz; Jordan Lerner-Ellis; Shaheen Awan; Don Bolser; Yael Bensoussan
Venue: Interspeech 2024 (main)

Abstract: The world of voice biomarkers is rapidly evolving thanks to the use of artificial intelligence (AI) allowing large-scale analysis of voice, speech, and respiratory sound data. The Bridge2AI-Voice project aims to build a large-scale, ethically sourced, and diverse voice database of human voices linked to health information to help fuel Voice AI research, dubbed Audiomics. The current paper describes the development of protocols of data acquisition across 4 different adult cohorts of disease (voice, respiratory, neurodegenerative diseases, mood, and anxiety disorders) using a Team Science approach for broader adoption by the research community and feedback. Demographic Surveys, Confounders Assessments, Acoustic tasks, validated patient-reported outcome (PRO) questionnaires and clinician-validated diagnostic questions were grouped in a common PART A across all cohorts and individual PART B, with cohort-specific tasks.
Self-Supervised Embeddings for Detecting Individual Symptoms of Depression
Authors: Sri Harsha Dumpala; Katerina Dikaios; Abraham Nunes; Frank Rudzicz; Rudolf Uher; Sageev Oore
Venue: Interspeech 2024 (main)

Abstract: Depression, a prevalent mental health disorder impacting millions globally, demands reliable assessment systems. Unlike previous studies that focus solely on either detecting depression or predicting its severity, our work identifies individual symptoms of depression while also predicting its severity using speech input. We leverage self-supervised learning (SSL)-based speech models to better utilize the small-sized datasets that are frequently encountered in this task. Our study demonstrates notable performance improvements by utilizing SSL embeddings compared to conventional speech features. We compare various types of SSL pretrained models to elucidate the type of speech information (semantic, speaker, or prosodic) that contributes the most in identifying different symptoms. Additionally, we evaluate the impact of combining multiple SSL embeddings on performance. Furthermore, we show the significance of multi-task learning for identifying depressive symptoms effectively.
Comparing ambulatory voice measures during daily life with brief laboratory assessments in speakers with and without vocal hyperfunction
Authors: Daryush D. Mehta; Jarrad H. Van Stan; Hamzeh Ghasemzadeh; Robert E. Hillman
Venue: Interspeech 2024 (main)

Abstract: The most common types of voice disorders are associated with hyperfunctional voice use in daily life. Although current clinical practice uses measures from brief laboratory recordings to assess vocal function, it is unclear how these relate to an individualâs habitual voice use. The purpose of this study was to quantify the correlation and offset between voice features computed from laboratory and ambulatory recordings in speakers with and without vocal hyperfunction. Features derived from a neck-surface accelerometer included estimates of sound pressure level, fundamental frequency, cepstral peak prominence, and spectral tilt. Whereas some measures from laboratory recordings correlated significantly with those captured during daily life, only approximately 6â52% of the actual variance was accounted for. Thus, brief voice assessments are quite limited in the extent to which they can accurately characterize the daily voice use of speakers with and without vocal hyperfunction.
Predicting Acute Pain Levels Implicitly from Vocal Features
Authors: Jennifer Williams; Eike Schneiders; Henry Card; Tina Seabrooke; Beatrice Pakenham-Walsh; Tayyaba Azim; Lucy Valls-Reed; Ganesh Vigneswaran; John Robert Bautista; Rohan Chandra; Arya Farahi
Venue: Interspeech 2024 (main)

Abstract: Evaluating pain in speech represents a critical challenge in high-stakes clinical scenarios, from analgesia delivery to emergency triage. Clinicians have predominantly relied on direct verbal communication of pain which is difficult for patients with communication barriers, such as those affected by stroke, autism, and learning difficulties. Many previous efforts have focused on multimodal data which does not suit all clinical applications. Our work is the first to collect a new English speech dataset wherein we have induced acute pain in adults using a cold pressor task protocol and recorded subjects reading sentences out loud. We report pain discrimination performance as F1 scores from binary (pain vs. no pain) and three-class (mild, moderate, severe) prediction tasks, and support our results with explainable feature analysis. Our work is a step towards providing medical decision support for pain evaluation from speech to improve care across diverse and remote healthcare settings.
Towards Intelligent Speech Assistants in Operating Rooms: A Multimodal Model for Surgical Workflow Analysis
Authors: Kubilay Can Demir; BelÃ©n Lojo RodrÃ­guez; Tobias Weise; Andreas Maier; Seung Hee Yang
Venue: Interspeech 2024 (main)

Abstract: To develop intelligent speech assistants and integrate them seamlessly with intra-operative decision-support frameworks, accurate and efficient surgical phase recognition is a prerequisite. In this study, we propose a multimodal framework based on Gated Multimodal Units (GMU) and Multi-Stage Temporal Convolutional Networks (MS-TCN) to recognize surgical phases of port-catheter placement operations. Our method merges speech and image models and uses them separately in different surgical phases. Based on the evaluation of 28 operations, we report a frame-wise accuracy of 92.65 Â± 3.52% and an F1-score of 92.30 Â± 3.82%. Our results show approximately 10% improvement in both metrics over previous work and validate the effectiveness of integrating multimodal data for the surgical phase recognition task. We further investigate the contribution of individual data channels by comparing mono-modal models with multimodal models.
A Multimodal Framework for the Assessment of the Schizophrenia Spectrum
Authors: Gowtham Premananth; Yashish M. Siriwardena; Philip Resnik; Sonia Bansal; Deanna L.Kelly; Carol Espy-Wilson
Venue: Interspeech 2024 (main)

Abstract: This paper presents a novel multimodal framework to distinguish between different symptom classes of subjects in the schizophrenia spectrum and healthy controls using audio, video, and text modalities. We implemented Convolution Neural Network and Long Short Term Memory based unimodal models and experimented on various multimodal fusion approaches to come up with the proposed framework. We utilized a minimal Gated multimodal unit (mGMU) to obtain a bi-modal intermediate fusion of the features extracted from the input modalities before finally fusing the outputs of the bimodal fusions to perform subject-wise classifications. The use of mGMU units in the multimodal framework improved the performance in both weighted f1-score and weighted AUC-ROC scores.
Exploring the Complementary Nature of Speech and Eye Movements for Profiling Neurological Disorders
Authors: Yuzhe Wang; Anna Favaro; Thomas Thebaud; Jesus Villalba; Najim Dehak; Laureano Moro-Velazquez
Venue: Interspeech 2024 (main)

Abstract: Subtle changes in both speech and ocular movements often characterize neurodegenerative diseases (NDs). This study aims to profile various NDs using speech and ocular movements collected during the Cookie Thief Picture description task. We designed new multi-modal features centered around the Areas of Interest (AoIs) to quantify attention focus and search patterns. Through statistical analysis, we examined which features differentiated controls (CTLs) from subjects with various NDs, including Alzheimerâs disease (AD), Parkinsonâs disease (PD), and Parkinsonâs disease mimics (PDM). Results showed that subjects with AD and PDM had reduced AoI visits, indicating diminished search behavior. In contrast, subjects with PD displayed increased variability in visual search patterns, possibly reflecting disruptions in integrating visual information with verbal expression and loss of attention during the trial. These features effectively differentiated ND groups from CTLs and among themselves.
Refining Self-supervised Learnt Speech Representation using Brain Activations
Authors: HengYu Li; Kangdi Mei; Zhaoci Liu; Yang Ai; Liping Chen; Jie Zhang; Zhenhua Ling
Venue: Interspeech 2024 (main)

Abstract: It was shown in literature that speech representations extracted by self-supervised pre-trained models exhibit similarities with brain activations of human for speech perception and fine-tuning speech representation models on downstream tasks can further improve the similarity. However, it still remains unclear if this similarity can be used to optimize the pre-trained speech models. In this work, we therefore propose to use the brain activations recorded by fMRI to refine the often-used wav2vec2.0 model by aligning model representations toward human neural responses. Experimental results on SUPERB reveal that this operation is beneficial for several downstream tasks, e.g., speaker verification, automatic speech recognition, intent classification. One can then consider the proposed method as a new alternative to improve self-supervised speech models.
Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disorder
Authors: Yuejiao Wang; Xianmin Gong; Lingwei Meng; Xixin Wu; Helen Meng
Venue: Interspeech 2024 (main)

Abstract: Functional magnetic resonance imaging (fMRI) is essential for developing encoding models that identify functional changes in language-related brain areas of individuals with Neurocognitive Disorders (NCD). While large language model (LLM)-based fMRI encoding has shown promise, existing studies predominantly focus on healthy, young adults, overlooking older NCD populations and cognitive level correlations. This paper explores language-related functional changes in older NCD adults using LLM-based fMRI encoding and brain scores, addressing current limitations. We analyze the correlation between brain scores and cognitive scores at both whole-brain and language-related ROI levels. Our findings reveal that higher cognitive abilities correspond to better brain scores, with correlations peaking in the middle temporal gyrus. This study highlights the potential of fMRI encoding models and brain scores for detecting early functional changes in NCD patients.
From Sound to Meaning in the Auditory Cortex: A Neuronal Representation and Classification Analysis
Authors: Kumar Neelabh; Vishnu Sreekumar
Venue: Interspeech 2024 (main)

Abstract: The neural mechanisms underlying the comprehension of meaningful sounds are yet to be fully understood. While previous work has shown that the auditory cortex can classify auditory stimuli into distinct semantic categories, the specific functions of the primary (A1) and the secondary auditory cortex (A2) are not well understood. We analyzed the neural responses of songbirds as they listened to their entire vocal repertoire. We first show that the distances between semantic categories in acoustic and neural representations are correlated in both A1 and A2. We then show that while both A1 and A2 are equally informative of the acoustic category of the vocalizations, A2 is significantly more informative of their semantic category. Additionally, we show that the semantic categories are more separated in A2. These findings suggest that as the incoming signal moves downstream within the auditory cortex, its acoustic information is preserved, while its semantic information is enhanced.
Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models
Authors: Sheng Feng; Heyang Liu; Yu Wang; Yanfeng Wang
Venue: Interspeech 2024 (main)

Abstract: In this paper, we introduce a groundbreaking end-to-end (E2E) framework for decoding invasive brain signals, marking a significant advancement in the field of speech neuroprosthesis. Our methodology leverages the comprehensive reasoning abilities of large language models (LLMs) to facilitate direct decoding. By fully integrating LLMs, we achieve results comparable to the state-of-the-art cascade models. Our findings underscore the immense potential of E2E frameworks in speech neuroprosthesis, particularly as the technology behind brain-computer interfaces (BCIs) and the availability of relevant datasets continue to evolve. This work not only showcases the efficacy of combining LLMs with E2E decoding for enhancing speech neuroprosthesis but also sets a new direction for future research in BCI applications, underscoring the impact of LLMs in decoding complex neural signals for communication restoration. Code will be made available at https://github.com/FsFrancis15/BrainLLM.
Toward Fully-End-to-End Listened Speech Decoding from EEG Signals
Authors: Jihwan Lee; Aditya Kommineni; Tiantian Feng; Kleanthis Avramidis; Xuan Shi; Sudarsana Reddy Kadiri; Shrikanth Narayanan
Venue: Interspeech 2024 (main)

Abstract: Speech decoding from EEG signals is a challenging task, where brain activity is modeled to estimate salient characteristics of acoustic stimuli. We propose FESDE, a novel framework for Fully-End-to-end Speech Decoding from EEG signals. Our approach aims to directly reconstruct listened speech waveforms given EEG signals, where no intermediate acoustic feature processing step is required. The proposed method consists of an EEG module and a speech module along with a connector. The EEG module learns to better represent EEG signals, while the speech module generates speech waveforms from model representations. The connector learns to bridge the distributions of the latent spaces of EEG and speech. The proposed framework is both simple and efficient, by allowing single-step inference, and outperforms prior works on objective metrics. A fine-grained phoneme analysis is conducted to unveil model characteristics of speech decoding. The source code is available here: github.com/lee-jhwn/fesde.
The Use of Phone Categories and Cross-Language Modeling for Phone Alignment of PanÃ£ra
Authors: Emily P. Ahn; Eleanor Chodroff; Myriam Lapierre; Gina-Anne Levow
Venue: Interspeech 2024 (main)

Abstract: Automating the time-alignment of phonetic labels in speech facilitates research in language documentation, yet such phonetic forced alignment requires pretrained acoustic models. For low-resource languages, this raises the question as to how and on which data the acoustic model should be trained. To align data from PanÃ£ra, an Amazonian indigenous language of Brazil, we investigated three approaches for forced alignment of low-resource languages using the Montreal Forced Aligner. First, we implemented a novel approach of manipulating the acoustic model granularity from phone-specific to increasingly broader natural class categories in training language-specific PanÃ£ra models. Second, we trained cross-language English models under two granularity settings. Third, we compared these models to a large, pretrained Global English acoustic model. Results showed that broadening phone categories can improve language-specific modeling, but cross-language modeling performed the best.
Deciphering Assamese Vowel Harmony with Featural InfoWaveGAN
Authors: Sneha Ray Barman; Shakuntala Mahanta; Neeraj Kumar Sharma
Venue: Interspeech 2024 (main)

Abstract: Traditional approaches for understanding phonological learning have predominantly relied on curated text data. Although insightful, such approaches limit the knowledge captured in textual representations of the spoken language. To overcome this limitation, we investigate the potential of the Featural InfoWaveGAN model to learn iterative long-distance vowel harmony using raw speech data. We focus on Assamese, a language known for its phonologically regressive and word-bound vowel harmony. We demonstrate that the model is adept at grasping the intricacies of Assamese phonotactics, particularly iterative long-distance harmony with regressive directionality. It also produced non-iterative illicit forms resembling speech errors during human language acquisition. Our statistical analysis reveals a preference for a specific [+high,+ATR] vowel as a trigger across novel items, indicative of feature learning. More data and control could improve model proficiency, contrasting the universality of learning.
Phonological Feature Detection for US English using the Phonet Library
Authors: Harsha Veena Tadavarthy; Austin Jones; Margaret E. L. Renwick
Venue: Interspeech 2024 (main)

Abstract: This paper details adaptation of the Phonet Library, a speech technology that uses distinctive features to calculate posterior probabilities for phonological classes, by training it on US English. Phonet's posterior probabilities provide a statistical basis for understanding patterned variability in speech, thus bridging the gap between acoustic data and phonological structures. We train Phonet on an English corpus, and we investigate both its precision in phoneme recognition at designated timestamps and the relationship of its distinctive feature probabilities to linguistic expectations for a subset of vowels and consonants. Phonet shows accuracy of over 90\% in predicting English phonological classes. Our analysis demonstrates Phonet's robustness in capturing basic relationships between theoretical natural classes of sounds in English, highlighting its utility in the broader context of speech analysis and phonetic research.
K-means and hierarchical clustering of f0 contours
Authors: Constantijn Kaland; Jeremy Steffman; Jennifer Cole
Venue: Interspeech 2024 (main)

Abstract: Cluster analysis on time-series f0 data is an increasingly popular method in intonation research. There are a number of methodological decisions to take when applying cluster analysis. Crucially, these decisions may affect the clustering results, potentially also the conclusions of the research. This paper investigates the extent to which the choice for either K-means or hierarchical clustering, two of the most popular clustering methods, leads to grouping differences that are potentially relevant for intonation research. This is tested using a dataset of f0 measures taken from imitated intonation patterns in American English. The analysis concerns a generic correlation test between K-means and hierarchical clustering outcomes as well as a number of specific measures assessing partitioning quality and f0 contour differences. The results show that both cluster methods generally show very similar outcomes, although considerable differences for specific clusterings might occur.
Tradition or Innovation: A Comparison of Modern ASR Methods for Forced Alignment
Authors: Rotem Rousso; Eyal Cohen; Joseph Keshet; Eleanor Chodroff
Venue: Interspeech 2024 (main)

Abstract: Forced alignment (FA) plays a key role in speech research through the automatic time alignment of speech signals with corresponding text transcriptions. Despite the move towards end-to-end architectures for speech technology, FA is still dominantly achieved through a classic GMM-HMM acoustic model. This work directly compares alignment performance from leading automatic speech recognition (ASR) methods, WhisperX and Massively Multilingual Speech Recognition (MMS), against a Kaldi-based GMM-HMM system, the Montreal Forced Aligner (MFA). Performance was assessed on the manually aligned TIMIT and Buckeye datasets, with comparisons conducted only on words correctly recognized by WhisperX and MMS. The MFA outperformed both WhisperX and MMS, revealing a shortcoming of modern ASR systems. These findings highlight the need for advancements in forced alignment and emphasize the importance of integrating traditional expertise with modern innovation to foster progress.