GenAI4ELab/papercli-papers huggingface dataset:
Contrastive Feedback Mechanism for Simultaneous Speech Translation
Authors: Haotian Tan; Sakriani Sakti
Venue: Interspeech 2024 (main)

Abstract: Recent advances in simultaneous speech translation (SST) focus on the decision policies that enable the use of offline-trained ST models for simultaneous inference. These decision policies not only control the quality-latency trade-off in SST but also mitigate the impact of unstable predictions on translation quality by delaying translation for more context or discarding these predictions through stable hypothesis detection. However, these policies often overlook the potential benefits of utilizing unstable predictions. We introduce the contrastive feedback mechanism (CFM) for SST, a novel method that leverages these unstable predictions as feedback to improve translation quality. CFM guides the system to eliminate undesired model behaviors from these predictions through a contrastive objective. The experiments on 3 state-of-the-art decision policies across 8 languages in the MuST-C v1.0 dataset show that CFM effectively improves the performance of SST.
Towards Speech-to-Pictograms Translation
Authors: CÃ©cile Macaire; ChloÃ© Dion; Didier Schwab; Benjamin Lecouteux; Emmanuelle EsperanÃ§a-Rodier
Venue: Interspeech 2024 (main)

Abstract: The automatic translation of speech into pictogram terms (Speech-to-Pictos) represents a novel NLP task with the potential to enhance communication for individuals with language impairments. Recent research has not explored the adaptation of state-of-the-art methods to this task, despite its significance. In this work, we investigate two approaches: (1) the cascade approach, which combines a speech recognition system with a machine translation system, and (2) the end-to-end approach, which tailors a speech translation system. We compare state-of-the-art architectures trained on an aligned speech-to-pictogram dataset, specially created and released for this study. We conduct an in-depth automatic and human evaluation to analyze their behavior on pictogram translation. The results highlight the cascade approachâs ability to generate relevant translations from everyday read speech, while the end-to-end approach achieves competitive results with challenging acoustic data.
Automatic Assessment of Speech Production Skills for Children with Cochlear Implants Using Wav2Vec2.0 Acoustic Embeddings
Authors: Seonwoo Lee; Sunhee Kim; Minhwa Chung
Venue: Interspeech 2024 (main)

Abstract: This study introduces an automatic assessment model for speech production skills of children with cochlear implants (CIs) to support home-based speech therapy. The model employs acoustic embeddings from self-supervised models and considers speech traits of both normal hearing (NH) adults and children, which is a novel method for evaluating speech of children with disorders. It combines phoneme embeddings and two acoustic embeddings from Wav2Vec2.0 models, each trained on the speech of NH adults and children, via multi-head attention. Using a speech corpus of Korean-speaking children with CIs, our model outperforms single-embedding methods in a Pearson correlation coefficient between predicted and expert-rated scores, with a relative improvement of 51%. The results highlight the effectiveness of Wav2Vec2.0 acoustic embeddings and the importance of incorporating both of typical speech patterns of NH adults and children in assessing speech production skills in children with CIs.
SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization
Authors: Young Jin Ahn; Jungwoo Park; Sangha Park; Jonghyun Choi; Kee-Eung Kim
Venue: Interspeech 2024 (main)

Abstract: Visual Speech Recognition (VSR) stands at the intersection of computer vision and speech recognition, aiming to interpret spoken content from visual cues. A prominent challenge in VSR is the presence of homophenesâvisually similar lip gestures that represent different phonemes. Prior approaches have sought to distinguish fine-grained visemes by aligning visual and auditory semantics, but often fell short of full synchronization. To address this, we present SyncVSR, an end-to-end learning framework that leverages quantized audio for frame-level crossmodal supervision. By integrating a projection layer that synchronizes visual representation with acoustic data, our encoder learns to generate discrete audio tokens from a video sequence in a non-autoregressive manner. SyncVSR shows versatility across tasks, languages, and modalities at the cost of a forward pass. Our empirical evaluations show that it not only achieves state-of-the-art results but also reduces data usage by up to ninefold.
Evaluating a 3-factor listener model for prediction of speech intelligibility to hearing-impaired listeners
Authors: Mark Huckvale; Gaston Hilkhuysen
Venue: Interspeech 2024 (main)

Abstract: A speech intelligibility prediction model for hearing impaired listeners would be useful in the development of better signal enhancement methods and for the fitting of hearing aids. Most current prediction models use only information from a pure-tone audiogram to characterise impaired listeners, although evidence suggests that listeners vary in ways not captured by pure-tone thresholds. In this paper we evaluate a model in which each listener is described by three factors: average pure-tone thresholds, sensitivity to phonetic distortion and sensitivity to word likelihood. We build and evaluate the model using the corpus collected by the second Clarity Prediction Challenge, which contains over 13,000 intelligibility judgments by 31 hearing impaired listeners. We describe how the factors were estimated and test their independence. We show that incorporating the listener-dependent factors into an existing intelligibility metric can improve the accuracy of prediction on held-out test data with a 9.8% relative improvement in prediction error.
Production of fricative consonants in French-speaking children with cochlear implants and typical hearing: acoustic and phonological analyses.
Authors: Sophie Fagniart; Brigitte Charlier; VÃ©ronique Delvaux; Bernard Harmegnies; Anne Huberlant; Myriam Piccaluga; Kathy Huet
Venue: Interspeech 2024 (main)

Abstract: The following study investigates fricative consonant production skills in 23 children with cochlear implants (CI group) and 47 children with typical hearing (TH group), matched by chronological and auditory age. The voiceless (/f/,/s/,/Ê/) and voiced (/v/,/z/,/Ê/) fricative consonants of French were studied from children's productions to a picture naming task. The results showed lower percentages of correct fricatives as well as fricativization and stopping errors in the CI group. Acoustic analyses showed productions differing between our two groups, with lower mid-frequency amplitude peak values for the /f,s,z/ phonemes, higher amplitude in the low-frequency bands and lower high-frequency energy in the CI group. Furthermore, links between phonological performance and acoustic productions is demonstrated: higher spectral values distinction are associated with a higher percentage of correct phonological production and fewer stopping/fricativization errors.
Signal processing algorithm effective for sound quality of hearing loss simulators
Authors: Toshio Irino; Shintaro Doan; Minami Ishikawa
Venue: Interspeech 2024 (main)

Abstract: Hearing loss (HL) simulators, which allow normal hearing (NH) listeners to experience HL, have been used in speech intelligibility experiments, but not in sound quality experiments due to perceptible distortion. If they produced less distortion, they might be useful for NH listeners to evaluate the sound quality of, for example, hearing aids. We conducted perceptual sound quality experiments to compare the Cambridge version of HL simulator (CamHLS) and the Wakayama version of the HL simulator (WHIS), which has the two algorithms of filterbank analysis synthesis (FBAS) and direct time-varying filter (DTVF). The experimental results showed that WHIS with DTVF produces less perceptible distortion in speech sounds than CamHLS and WHIS with FBAS, even when the nonlinear process is working. This advantage is mainly due to the use of the DTVF algorithm, which could be applied to various signal synthesis applications with filterbank analysis.
Auditory Spatial Attention Detection Based on Feature Disentanglement and Brain Connectivity-Informed Graph Neural Networks
Authors: Yixiang Niu; Ning Chen; Hongqing Zhu; Zhiying Zhu; Guangqiang Li; Yibo Chen
Venue: Interspeech 2024 (main)

Abstract: Auditory spatial attention detection (ASAD) aims to determine which speaker in a surround sound field a listener is focusing on from a single-trial electroencephalogram (EEG). Latest studies have represented non-Euclidean structured EEGs by graph-based modeling, but how to better incorporate brain connectivity into EEG graphs remains a great challenge. Moreover, due to inter-subject distribution shifts in EEGs, most existing models perform well only on specific subjects. To address these issues, we propose a new ASAD model. EEG graphs are constructed based on brain effective connectivity, and then mapped into embedding spaces by a graph neural network architecture. Meanwhile, feature disentanglement combined with correlation alignment is utilized to learn subject-invariant EEG patterns relevant to ASAD tasks. Experiments on open datasets demonstrate that in cross-subject scenarios, the proposed model outperforms state-of-the-art models, and the algorithmic complexity is relatively low.
Automatic Detection of Hearing Loss from Children's Speech using wav2vec 2.0 Features
Authors: Jessica Monaghan; Arun Sebastian; Nicky Chong-White; Vicky Zhang; Vijayalakshmi Easwar; Padraig Kitterick
Venue: Interspeech 2024 (main)

Abstract: This study explores the feasibility of employing machine learning to classify acoustic features of speech for detecting hearing loss in preschool children. Acknowledging the critical developmental impacts of early hearing loss identification and the challenges associated with traditional testing methods for this age group, we propose a novel, scalable approach leveraging automatic speech analysis. Using speech recordings from children with and without hearing loss, we used wav2vec 2.0 and ComParE feature sets to capture speech characteristics and compared LSTM, DNN, and XGBoost classifiers. Our findings reveal that these models can accurately distinguish between the speech of children with hearing loss and those with normal hearing, achieving up to 96.4% accuracy. This proof-of-concept study indicates the potential of using speech for early hearing loss detection, and a path toward non-intrusive, scalable screening tools that could significantly benefit early developmental outcomes.
Whister: Using Whisperâs representations for Stuttering detection
Authors: Vrushank Changawala; Frank Rudzicz
Venue: Interspeech 2024 (main)

Abstract: In this paper, we empirically investigate the influence of different factors on the performance of dysfluency detection. Specifically, we examine the impact of data splits, data quality, and learned representations of large pre-trained models. To conduct our experiments, we use the frozen Whisper model with two trainable heads, along with MFCC features extracted from input audio. We train on different data splits and evaluate performance using a cross-corpora testing strategy. We find that longer audio segments, specifically 5 seconds as opposed to the conventional 3-second segments, leads to improved performance. We also show that our architecture design is generalizable for multilingual data. We attain a 9.3% and 22% relative improvement in the average F1 score for FluencyBank and KSoF-test datasets, respectively, surpassing the previous state-of-the-art.
Improving Speech-Based Dysarthria Detection using Multi-task Learning with Gradient Projection
Authors: Yan Xiong; Visar Berisha; Julie Liss; Chaitali Chakrabarti
Venue: Interspeech 2024 (main)

Abstract: Speech analytic models based on deep learning are popular in clinical diagnostics. However, constraints on clinical data collection and sharing place limits on available dataset sizes, which adversely impacts trained model performance. Multi-task learning (MTL) has been utilized to mitigate the effect of limited sample size by jointly training on multiple tasks that are considered to be related. However, discrepancies between clinical and non-clinical tasks can reduce MTL efficiency and can even cause it to fail, especially when there are gradient conflicts. In this paper, we enhance the performance of dysarthria detection by using MTL with an auxiliary task of learning speaker embeddings. We propose a task-specific gradient projection method to overcome gradient conflicts. Our evaluation shows that the proposed MTL paradigm outperforms both single-task learning and conventional MTL under different data availability settings.
Cascaded Transfer Learning Strategy for Cross-Domain Alzheimer's Disease Recognition through Spontaneous Speech
Authors: Guanlin Chen; Yun Jin
Venue: Interspeech 2024 (main)

Abstract: In our work, we propose a cascaded transfer learning strategy for cross-domain Alzheimerâs disease (AD) recognition through spontaneous speech. This strategy cascaded a pre-trained GPT-3 model as the first-level and a Random Forest Multi-Source Discriminant Subspace Alignment (RF-MDSA) Algorithm as the second-level. The goal of this strategy is to align feature spaces extracted from different corpora to derive a common projection subspace. We conduct experiments on three corpora in Chinese, English, and Spanish. On single corpus experiments, we achieve accuracy rates of 74.6%, 82.4%, and 62.8%, respectively. In cross-domain experiments, we achieve accuracy rates of 71.8%, 72.9%, and 56.1%, respectively. These results suggest that our cascaded strategy improves the modelâs generalization capabilities, making it effective for cross-domain AD recognition.
A Cross-Attention Layer coupled with Multimodal Fusion Methods for Recognizing Depression from Spontaneous Speech
Authors: Loukas Ilias; Dimitris Askounis
Venue: Interspeech 2024 (main)

Abstract: Depression is a serious mood disorder, which affects the way people feel and perform daily activities. Speech is a reliable biomarker for diagnosing depression, since depressed people present decreased verbal activity productivity and âlifelessâ sounding speech. Existing methods employ unimodal models, use early, intermediate, or late fusion strategies to fuse the different modalities, rely on feature extraction, and perform their approaches only in the English language. This study presents a new method for identifying depression from spontaneous speech in the Italian language, which uses a cross-attention layer for capturing the cross-modal interactions, followed by a variety of multimodal fusion methods. We also perform a multi-task learning framework to explore whether the prediction of age, education level, and gender help in recognizing depression. Findings show that our approach yields multiple advantages over existing approaches reaching Accuracy up to 95.29%.
Segmental and Suprasegmental Speech Foundation Models for Classifying Cognitive Risk Factors: Evaluating Out-of-the-Box Performance
Authors: Si-Ioi Ng; Lingfeng Xu; Kimberly D. Mueller; Julie Liss; Visar Berisha
Venue: Interspeech 2024 (main)

Abstract: Speech foundation models are remarkably successful in various consumer applications, prompting their extension to clinical use-cases. This is challenged by small clinical datasets, which precludes effective fine-tuning. We tested the efficacy of two models to classify participants by segmental (Wav2Vec2.0) and suprasegmental (Trillsson) speech analysis windows. Analysis at both time scales has shown differences in the context of cognitive decline. Speakers were classified as healthy controls (HC), Amyloid-Î²+ (AÎ²+), mild cognitive impairment (MCI), or dementia. A subset of W2V2 and Trillsson representations showed large effect size between HC and each risk factor. Cross-validation showed W2V2 consistently outperforms Trillsson. Mean macro-F1 of 54.1%, 63.5%, and 72.0% in were found for classifying AÎ²+, MCI, and dementia from HC. Repeatability of Trillsson and W2V2 showed intraclass correlations of 0.30 and 0.41. Reliability of such models must be enhanced for clinical speech analysis and longitudinal tracking.
Multimodal Continuous Fingerspelling Recognition via Visual Alignment Learning
Authors: Katerina Papadimitriou; Gerasimos Potamianos
Venue: Interspeech 2024 (main)

Abstract: Continuous fingerspelling recognition from videos is paramount for real-time sign language (SL) interpretation, enhancing accessibility. Despite deep learning progress, challenges persist, especially in signer-independent (SI) scenarios, due to signing variability. To address these, we propose a novel bimodal approach that integrates appearance and skeletal information, focusing solely on the signing hand. Our system relies on two basic modules: (a) a 3D-CNN model capturing spatial features, while adapting to motion variations and (b) a modulated spatio-temporal graph convolutional network (ST-GCN) based on 3D joint-rotation parameterization for skeletal feature modeling. Both modalities are combined with a BiGRU encoder and CTC decoding. To further enhance representation capacity, we introduce an alignment mechanism relying on two auxiliary losses. Through ensemble fusion and language model integration, our method achieves superior performance across three SI fingerspelling datasets.
Contrastive Learning Approach for Assessment of Phonological Precision in Patients with Tongue Cancer Using MRI Data
Authors: Tomas Arias-Vergara; Paula Andrea PÃ©rez-Toro; Xiaofeng Liu; Fangxu Xing; Maureen Stone; Jiachen Zhuo; Jerry L. Prince; Maria Schuster; Elmar Noeth; Jonghye Woo; Andreas Maier
Venue: Interspeech 2024 (main)

Abstract: Magnetic Resonance Imaging (MRI) allows analyzing speech production by capturing high-resolution images of the dynamic processes in the vocal tract. In clinical applications, combining MRI with synchronized speech recordings leads to improved patient outcomes, especially if a phonological-based approach is used for assessment. However, when audio signals are unavailable, the recognition accuracy of sounds is decreased when using only MRI data. We propose a contrastive learning approach to improve the detection of phonological classes from MRI data when acoustic signals are not available at inference time. We demonstrate that frame-wise recognition of phonological classes improves from an f1 of 0.74 to 0.85 when the contrastive loss approach is implemented. Furthermore, we show the utility of our approach in the clinical application of using such phonological classes to assess speech disorders in patients with tongue cancer, yielding promising results in the recognition task.
DysArinVox: DYSphonia & DYSarthria mandARIN speech corpus
Authors: Haojie Zhang; Tao Zhang; Ganjun Liu; Dehui Fu; Xiaohui Hou; Ying Lv
Venue: Interspeech 2024 (main)

Abstract: This paper introduces DysArinVox, a new pathological speech corpus in Chinese. It included 173 participants from 27 healthy individuals and 146 voice disorders, whose various types and severities of vocal impairments as diagnosed by speech pathology experts via auditory perceptual evaluations and laryngoscopic imagery. DysArinVox is designed to provide a high-quality Chinese resource for AI-driven diagnostics and prognostics. To ensure the efficiency of corpus collection, we meticulously crafted recording scripts represent Mandarin phonetically, ensuring comprehensive syllable representation with minimal lexical complexity. Additionally, incorporating laryngoscopic images of patients into the dataset offers extra visual information, facilitating the development of advanced diagnostic frameworks. To our knowledge, this database represents the most comprehensive corpus of Chinese pathological speech to date.
YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection
Authors: Xuanru Zhou; Anshul Kashyap; Steve Li; Ayati Sharma; Brittany Morin; David Baquirin; Jet Vonk; Zoe Ezzes; Zachary Miller; Maria Tempini; Jiachen Lian; Gopala Anumanchipalli
Venue: Interspeech 2024 (main)

Abstract: Dysfluent speech detection is the bottleneck for disordered speech analysis and spoken language learning. Current state-of-the-art models are governed by rule-based systems which lack efficiency and robustness, and are sensitive to template design. In this paper, we propose YOLO-Stutter: a first end-to-end method that detects dysfluencies in a time-accurate manner. YOLO-Stutter takes  imperfect speech-text alignment as input, followed by a spatial feature aggregator, and a temporal dependency extractor to perform region-wise boundary and class predictions. We also introduce two dysfluency corpus, VCTK-Stutter and VCTK-TTS, that simulate natural spoken dysfluencies including repetition, block, missing, replacement, and prolongation. Our end-to-end method achieves state-of-the-art performance with a minimum number of trainable parameters for on both simulated data and real aphasia speech . Code and datasets are open-sourced at https://github.com/rorizzz/YOLO-Stutter
Automatic Longitudinal Investigation of Multiple Sclerosis Subjects
Authors: GÃ¡bor Gosztolya; Veronika Svindt; Judit BÃ³na; IldikÃ³ Hoffmann
Venue: Interspeech 2024 (main)

Abstract: Multiple Sclerosis is a chronic inflammatory disease of the central nervous system. Over time, people with MS may experience significant changes in cognition, language and speech processes. In this study we investigate speech utterances recorded over the course of three years for 16 MS subjects and 12 healthy controls. Our examination is based on speaker category classification (healthy or MS) using wav2vec2 embeddings as features. We found that subject classification performance improved over time: the 0.745-0.844 AUC values from year one increased to 0.891-0.979 in the third year. By analyzing the posterior estimates, we measured a statistically significant improvement in the scores corresponding to the third year for the MS category, while for the control subjects there was no such tendency. This, in our view, indicates that the change is due to a subtle deterioration in the condition of MS patients, which was detected by our machine learning workflow.
Connected Speech-Based Cognitive Assessment in Chinese and English
Authors: Saturnino Luz; Sofia De La Fuente Garcia; Fasih Haider; Davida Fromm; Brian MacWhinney; Alyssa Lanzi; Ya-Ning Chang; Chia-Ju Chou; Yi-Chien Liu
Venue: Interspeech 2024 (main)

Abstract: We present a novel benchmark dataset and prediction tasks for investigating approaches to assess cognitive function through analysis of connected speech. The dataset consists of speech samples and clinical information for speakers of Mandarin Chinese and English with different levels of cognitive impairment as well as individuals with normal cognition. These data have been carefully matched by age and sex by propensity score analysis to ensure balance and representativity in model training. The prediction tasks encompass mild cognitive impairment diagnosis and cognitive test score prediction. This framework was designed to encourage the development of approaches to speech-based cognitive assessment which generalise across languages. We illustrate it by presenting baseline prediction models that employ language-agnostic and comparable features for diagnosis and cognitive test score prediction. Unweighted average recall was 59.2% in diagnosis, and root mean squared error was 2.89 in score prediction.