GenAI4ELab/papercli-papers huggingface dataset:
Speech Recognition Models are Strong Lip-readers
Authors: K R Prajwal; Triantafyllos Afouras; Andrew Zisserman
Venue: Interspeech 2024 (main)

Abstract: In this work, we show that a large pre-trained ASR model can be adapted to perform lip-reading. Our method enables an ASR model like Whisper to interpret lip movements in a video and output text transcriptions. We achieve this by learning a cross-modal mapping from a lip sequence to a speech sequence, allowing a pre-trained ASR model to directly perform lip-reading. The mapping can be learnt simply by backpropagating the cross-entropy loss on the text labels through the pre-trained, frozen ASR model. We achieve an impressive gain of 5.7 WER in the low data regime on the LRS3 benchmark over previous lip-reading methods. Finally, we demonstrate that the same strategy can be extended to other visual speech tasks, such as identifying the spoken language in silent videos.
Towards Self-Attention Understanding for Automatic Articulatory Processes Analysis in Cleft Lip and Palate Speech
Authors: Ilja Baumann; Dominik Wagner; Maria Schuster; Korbinian Riedhammer; Elmar Noeth; Tobias Bocklet
Venue: Interspeech 2024 (main)

Abstract: Cleft lip and palate (CLP) speech presents unique challenges for automatic phoneme analysis due to its distinct acoustic characteristics and articulatory anomalies. We perform phoneme analysis in CLP speech using a pre-trained wav2vec 2.0 model with a multi-head self-attention classification module to capture long-range dependencies within the speech signal, thereby enabling better contextual understanding of phoneme sequences. We demonstrate the effectiveness of our approach in the classification of various articulatory processes in CLP speech. Furthermore, we investigate the interpretability of self-attention to gain insights into the model's understanding of CLP speech characteristics. Our findings highlight the potential of the self-attention mechanisms for improving automatic phoneme analysis in CLP speech, paving the way for enhanced diagnostics, adding interpretability for therapists and affected patients.
Clever Hans Effect Found in Automatic Detection of Alzheimer's Disease through Speech
Authors: Yin-Long Liu; Rui Feng; Jia-Hong Yuan; Zhen-Hua Ling
Venue: Interspeech 2024 (main)

Abstract: We uncover an underlying bias present in the audio recordings produced from the picture description task of the Pitt corpus, the largest publicly accessible database for Alzheimerâs Disease (AD) detection research. Even by solely utilizing the silent segments of these audio recordings, we achieve nearly 100% accuracy in AD detection. However, employing the same methods to other datasets and preprocessed Pitt recordings results in typical levels (approximately 80%) of AD detection accuracy. These results demonstrate a Clever Hans effect in AD detection on the Pitt corpus. Our findings emphasize the crucial importance of maintaining vigilance regarding inherent biases in datasets utilized for training deep learning models, and highlight the necessity for a better understanding of the modelsâ performance.
Leveraging Phonemic Transcription and Whisper toward Clinically Significant Indices for Automatic Child Speech Assessment
Authors: Yeh-Sheng Lin; Shu-Chuan Tseng; Jyh-Shing Roger Jang
Venue: Interspeech 2024 (main)

Abstract: Diagnosing speech sound disorders (SSD) in children requires professional assessment by speech-language pathologists. Detecting and diagnosing a medical condition takes time and is usually expensive in terms of labor. However, early identification and treatment are essential for subsequent care. ASR-based child speech assessment prioritizes semantic understanding over phonetic accuracy, making it unsuitable for pronunciation assessment. This study uses phonemic transcription available in a normative dataset and utilizes pre-trained speech models to develop an automatic phoneme recognition model with a Phoneme Error Rate (PER) as low as 3.76%. Clinically relevant indices calculated from the model prediction are highly correlated with those from the original normative data. We regard these experimental results as solid evidence that validates the feasibility of our evaluation workflow for practical application in early screening for phonological development delays.
Developing vocal system impaired patient-aimed voice quality assessment approach using ASR representation-included multiple features
Authors: Shaoxiang Dang; Tetsuya Matsumoto; Yoshinori Takeuchi; Takashi Tsuboi; Yasuhiro Tanaka; Daisuke Nakatsubo; Satoshi Maesawa; Ryuta Saito; Masahisa Katsuno; Hiroaki Kudo
Venue: Interspeech 2024 (main)

Abstract: The potential of deep learning in clinical speech processing is immense, yet the hurdles of limited and imbalanced clinical data samples loom large. This article addresses these challenges by showcasing the utilization of automatic speech recognition and self-supervised learning representations, pre-trained on extensive datasets of normal speech. This innovative approach aims to estimate voice quality of patients with impaired vocal systems. Experiments involve checks on PVQD dataset, covering various causes of vocal system damage in English, and a Japanese dataset focusing on patients with Parkinson's disease before and after undergoing subthalamic nucleus deep brain stimulation (STN-DBS) surgery. The results on PVQD reveal a notable correlation (>0.8 on PCC) and an extraordinary accuracy (<0.5 on MSE) in predicting Grade, Breathy, and Asthenic indicators. Meanwhile, progress has been achieved in predicting the voice quality of patients in the context of STN-DBS.
A Cluster-based Personalized Federated Learning Strategy for End-to-End ASR of Dementia Patients
Authors: Wei-Tung Hsu; Chin-Po Chen; Yun-Shao Lin; Chi-Chun Lee
Venue: Interspeech 2024 (main)

Abstract: Automatic speech recognition (ASR) is crucial for all users, but adapting it for Alzheimerâs disease (AD) faces challenges due to irregular speech patterns and privacy concerns. Federated learning (FL), a privacy-preserving algorithm, is a solution. However, FL ASR suffers from acoustic and text heterogeneities. While advanced model-based and cluster-based FL methods aim to address the issue, they lack a direct mechanism for high intra-speaker heterogeneity exhibited by AD individuals and ASR-related properties. This study presents cluster-based personalized federated learning (CPFL), a strategy mitigating heterogeneity by clustering ASR output token using the proposed CharDiv, a metric for pause and word usage distributions. Evaluation on the ADReSS challenge dataset shows a 3.6% improvement in word error rate (WER). Analysis of per-cluster WER improvements and CharDiv distributions indicates reduced heterogeneity, emphasizing pause usage as a potential key factor in AD-oriented ASR.
A Comparative Analysis of Federated Learning for Speech-Based Cognitive Decline Detection
Authors: Stefan Kalabakov; Monica Gonzalez-Machorro; Florian Eyben; BjÃ¶rn W. Schuller; Bert Arnrich
Venue: Interspeech 2024 (main)

Abstract: Speech-based machine learning models that can distinguish between a healthy cognitive state and different stages of cognitive decline would enable a more appropriate and timely treatment of patients. However, their development is often hampered by data scarcity. Federated Learning (FL) is a potential solution that could enable entities with limited voice recordings to collectively build effective models. Motivated by this, we compare centralised, local, and federated learning for building speech-based models to discern Alzheimerâs Disease, Mild Cognitive Impairment, and a healthy state. For a more realistic evaluation, we use three independently collected datasets to simulate healthcare institutions employing these strategies. Our initial analysis shows that FL may not be the best solution in every scenario, as performance improvements are not guaranteed even with small amounts of available data, and further research is needed to determine the conditions under which it is beneficial.
Multimodal Digital Biomarkers for Longitudinal Tracking of Speech Impairment Severity in ALS: An Investigation of Clinically Important Differences
Authors: Michael Neumann; Hardik Kothare; Jackson Liscombe; Emma C.L. Leschly; Oliver Roesler; Vikram Ramanarayanan
Venue: Interspeech 2024 (main)

Abstract: Speech biomarkers have shown promise for the remote assessment of ALS. However, to demonstrate clinical utility at tracking longitudinal progress of the disease, one needs to understand how well these biomarkers capture changes that are âclinically meaningfulâ, a concept that is not always clearly defined. Therefore, this paper defines and explores multiple methods of computing minimal clinically important difference (MCID) using ratings of speech impairment severity and listener effort as clinical anchors. We analyze how these methods impact the estimated responsiveness of various metrics collected from 125 ALS patients via a multimodal dialog based remote assessment platform. We find that select biomarkers are more responsive than the clinical standard ALSFRS-R across the board at tracking clinically meaningful changes related to speech severity. We further discuss advantages and disadvantages of different MCID computation methods for assessing ALS disease progression.
Enhancing Voice Wake-Up for Dysarthria: Mandarin Dysarthria Speech Corpus Release and Customized System Design
Authors: Ming Gao; Hang Chen; Jun Du; Xin Xu; Hongxiao Guo; Hui Bu; Jianxing Yang; Ming Li; Chin-Hui Lee
Venue: Interspeech 2024 (main)

Abstract: Smart home technology has gained widespread adoption, facilitating effortless control of devices through voice commands. However, individuals with dysarthria, a motor speech disorder, face challenges due to the variability of their speech. This paper addresses the wake-up word spotting (WWS) task for dysarthric individuals, aiming to integrate them into real-world applications. To support this, we release the open-source Mandarin Dysarthria Speech Corpus (MDSC), a dataset designed for dysarthric individuals in home environments. MDSC encompasses information on age, gender, disease types, and intelligibility evaluations. Furthermore, we perform comprehensive experimental analysis on MDSC, highlighting the challenges encountered. We also develop a customized dysarthria WWS system that showcases robustness in handling intelligibility and achieving exceptional performance. MDSC will be released on https://www.aishelltech.com/AISHELL_6B.
Towards Improving NAM-to-Speech Synthesis Intelligibility using Self-Supervised Speech Models
Authors: Neil Shah; Shirish Karande; Vineet Gandhi
Venue: Interspeech 2024 (main)

Abstract: We propose a novel approach to significantly improve the intelligibility in the Non-Audible Murmur (NAM)-to-speech conversion task, leveraging self-supervision and sequence-to-sequence (Seq2Seq) learning techniques. Unlike conventional methods that explicitly record ground-truth speech, our methodology relies on self-supervision and speech-to-speech synthesis to simulate ground-truth speech. Despite utilizing simulated speech, our method surpasses the current state-of-the-art (SOTA) by 29.08% improvement in the Mel-Cepstral Distortion (MCD) metric. Additionally, we present error rates and demonstrate our modelâs proficiency to synthesize speech in novel voices of interest. Moreover, we present a methodology for augmenting the existing CSTR NAM TIMIT Plus corpus, setting a benchmark with a Word Error Rate (WER) of 42.57% to gauge the intelligibility of the synthesized speech. Speech samples can be found at https://nam2speech.github.io/NAM2Speech/
PARAN: Variational Autoencoder-based End-to-End Articulation-to-Speech System for Speech Intelligibility
Authors: Seyun Um; Doyeon Kim; Hong-Goo Kang
Venue: Interspeech 2024 (main)

Abstract: Deep learning-based articulation-to-speech (ATS) systems designed for individuals with speech disorders have been extensively researched in recent years. However, conventional methods have faced challenges in representing the transformation in latent space across speech and electromagnetic articulography (EMA) domains, resulting in low speech quality. In this paper, we propose a variational autoencoder (VAE)-based end-to-end ATS model called PARAN that efficiently produces high-fidelity speech from EMA signals. Our model adjusts a prior distribution of latent representations from EMA signals to match a posterior distribution derived from speech utilizing a normalizing flow mechanism. To further enhance the clarity and intelligibility of the synthesized speech, we incorporate an additional loss function aimed at predicting phonetic information from EMA signals. Experimental results demonstrate that our model outperforms previous methods in terms of speech quality and intelligibility.
Acoustic changes in speech prosody produced by children with autism after robot-assisted speech training
Authors: Si Chen; Bruce Xiao Wang; Yitian Hong; Fang Zhou; Angel Chan; Po-yi Tang; Bin Li; Chunyi Wen; James Cheung; Yan Liu; Zhuoming Chen
Venue: Interspeech 2024 (main)

Abstract: Children with Autism exhibit distinct speech prosody, perceived as monotone and they are reported to show deficits in focus marking. The current study designed a robot-assisted training with controlled social interactions aiming to enhance the prosody of children with Autism speaking a tonal language, Cantonese, specifically on focus marking. 20 autistic and 23 typically-developing (TD) children participated in this study. Only the autistic group received training. Stimuli were designed for training, pre- and post-training production. Acoustics of target words were extracted and analysed using linear mixed-effects models examining effects of training and clinical status. Children with Autism improved in signalling sentence prominence using duration but not f0 and intensity. Variability suggests that certain acoustic cues are more challenging. Comparing to TD children's focus marking patterns, autistic children's variability may also stem from their ongoing prosodic profile development.
Fine-Tuning Automatic Speech Recognition for People with Parkinson's: An Effective Strategy for Enhancing Speech Technology Accessibility
Authors: Xiuwen Zheng; Bornali Phukon; Mark Hasegawa-Johnson
Venue: Interspeech 2024 (main)

Abstract: This paper enhances dysarthric and dysphonic speech recognition by fine-tuning pretrained automatic speech recognition (ASR) models on the 2023-10-05 data package of the Speech Accessibility Project (SAP), which contains the speech of 253 people with Parkinson's disease. Experiments tested methods that have been effective for Cerebral Palsy, including the use of speaker clustering and severity-dependent models, weighted fine-tuning, and multi-task learning.  Best results were obtained using a multi-task learning model, in which the ASR is trained to produce an estimate of the speaker's impairment severity as an auxiliary output. The resulting word error rates are considerably improved relative to a baseline model fine-tuned using only Librispeech data, with word error rate improvements of 37.62% and 26.97% compared to fine-tuning on 100h and 960h of LibriSpeech data, respectively.
Learnings from curating a trustworthy, well-annotated, and useful dataset of disordered English speech
Authors: Pan-Pan Jiang; Jimmy Tobin; Katrin Tomanek; Robert MacDonald; Katie Seaver; Richard Cave; Marilyn Ladewig; Rus Heywood; Jordan Green
Venue: Interspeech 2024 (main)

Abstract: Project Euphonia, a Google initiative, is dedicated to improving automatic speech recognition (ASR) of disordered speech. A central objective of the project is to create a large, high-quality, and diverse speech corpus. This report describes the projectâs latest advancements in data collection and annotation methodologies, such as expanding speaker diversity in the database, adding human-reviewed transcript corrections and audio quality tags to 350K (of the 1.2M total) audio recordings, and amassing a comprehensive set of metadata (including more than 40 speech characteristic labels) for over 75% of the speakers in the database. We report on the impact of transcript corrections on our machine-learning (ML) research, inter-rater variability of assessments of disordered speech patterns, and our rationale for gathering speech metadata. We also consider the limitations of using automated off-the-shelf annotation methods for assessing disordered speech.
Training Data Augmentation for Dysarthric Automatic Speech Recognition by Text-to-Dysarthric-Speech Synthesis
Authors: Wing-Zin Leung; Mattias Cross; Anton Ragni; Stefan Goetze
Venue: Interspeech 2024 (main)

Abstract: Automatic speech recognition (ASR) research has achieved impressive performance in recent years and has significant potential for enabling access for people with dysarthria (PwD) in augmentative and alternative communication (AAC) and home environment systems. However, progress in dysarthric ASR (DASR) has been limited by high variability in dysarthric speech and limited public availability of dysarthric training data. This paper demonstrates that data augmentation using text-to-dysarthic-speech (TTDS) synthesis for finetuning large ASR models is effective for DASR. Specifically, diffusion-based text-to-speech (TTS) models can produce speech samples similar to dysarthric speech that can be used as additional training data for fine-tuning ASR foundation models, in this case Whisper. Results show improved synthesis metrics and ASR performance for the proposed multi-speaker diffusion-based TTDS data augmentation for ASR fine-tuning compared to current DASR baselines.
Wav2vec 2.0 Embeddings Are No Swiss Army Knife -- A Case Study for Multiple Sclerosis
Authors: GÃ¡bor Gosztolya; Mercedes VetrÃ¡b; Veronika Svindt; Judit BÃ³na; IldikÃ³ Hoffmann
Venue: Interspeech 2024 (main)

Abstract: In the past few years, self-supervised learning has revolutionalized automatic speech recognition. Self-supervised models such as wav2vec2, due to their generalization ability on huge unannotated audio corpora, were claimed to be state-of-the-art feature extractors in paralinguistic and pathological applications as well. In this study we test embeddings extracted from a wav2vec 2.0 model fine-tuned on the target language as features on a multiple sclerosis audio corpus, using three speech tasks. After comparing the resulting classification performances with traditional features such as ComParE functionals, ECAPA-TDNN and activations of a HMM/DNN hybrid acoustic model, we found that wav2vec2-based models, surprisingly, only produced a mediocre classification performance. In contrast, the decade-old ComParE functionals feature set consistently led to high scores. Our results also indicate that the number of features correlates surprisingly well with classification performance.
Interface Design for Self-Supervised Speech Models
Authors: Yi-Jen Shih; David Harwath
Venue: Interspeech 2024 (main)

Abstract: Self-supervised speech (SSL) models have recently become widely adopted for many downstream speech processing tasks. The general usage pattern is to employ SSL models as feature extractors, and then train a downstream prediction head to solve a specific task. However, different layers of SSL models have been shown to capture different types of information, and the methods of combining them are not well studied. To this end, we extend the general framework for SSL model utilization by proposing the interface that connects the upstream and downstream. Under this view, the dominant technique of combining features via a layerwise weighted sum can be regarded as a specific interface. We propose several alternative interface designs and demonstrate that the weighted sum interface is suboptimal for many tasks. In particular, we show that a convolutional interface whose depth scales logarithmically with the depth of the upstream model consistently outperforms many other interface designs.
Comparing Discrete and Continuous Space LLMs for Speech Recognition
Authors: Yaoxun Xu; Shi-Xiong Zhang; Jianwei Yu; Zhiyong Wu; Dong Yu
Venue: Interspeech 2024 (main)

Abstract: This paper investigates discrete and continuous speech representations in Large Language Model (LLM)-based Automatic Speech Recognition (ASR), organizing them by feature continuity and training approach into four categories: supervised and unsupervised for both discrete and continuous types. We further classify LLMs based on their input and autoregressive feedback into continuous and discrete-space models. Using specialized encoders and comparative analysis with a Joint-Training-From-Scratch Language Model (JTFS LM) and pre-trained LLaMA2-7b, we provide a detailed examination of their effectiveness. Our work marks the first extensive comparison of speech representations in LLM-based ASR and explores various modeling techniques. We present an open-sourced achievement of a state-of-the-art Word Error Rate (WER) of 1.69% on LibriSpeech using a HuBERT encoder, offering valuable insights for advancing ASR and natural language processing (NLP) research.
Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text
Authors: Jinpeng Li; Yu Pu; Qi Sun; Wei-Qiang Zhang
Venue: Interspeech 2024 (main)

Abstract: Whisper and other large-scale automatic speech recognition models have made significant progress in performance. However, their performance on many low-resource languages, such as Kazakh, is not satisfactory. It is worth researching how to utilize low-cost data to improve the performance of Whisper on under-represented languages. In this study, we utilized easily accessible unpaired speech and text data and combined the language model GPT with Whisper on Kazakh. We implemented end of transcript (EOT) judgment modification and hallucination penalty to improve the performance of speech recognition. Further, we employed the decoding average token log probability as a criterion to select samples from unlabeled speech data and used pseudo-labeled data to fine-tune the model to further improve its performance. Ultimately, we achieved more than 10% absolute WER reduction in multiple experiments, and the whole process has the potential to be generalized to other under-represented languages.
Empowering Low-Resource Language ASR via Large-Scale Pseudo Labeling
Authors: Kaushal Santosh Bhogale; Deovrat Mehendale; Niharika Parasa; Sathish Kumar Reddy G; Tahir Javed; Pratyush Kumar; Mitesh M. Khapra
Venue: Interspeech 2024 (main)

Abstract: In this study, we tackle the challenge of limited labeled data for low-resource languages in ASR, focusing on Hindi. Specifically, we explore pseudo-labeling, by proposing a generic framework combining multiple ideas from existing works. Our framework integrates multiple base models for transcription and evaluators for assessing audio-transcript pairs, resulting in robust pseudo-labeling for low resource languages. We validate our approach with a new benchmark, IndicYT, comprising diverse YouTube audio files from multiple content categories. Our findings show that augmenting pseudo labeled data from YouTube with existing training data leads to significant performance improvements on IndicYT, without affecting performance on out-of-domain benchmarks, demonstrating the efficacy of pseudo-labeled data in enhancing ASR capabilities for low-resource languages. The benchmark, code and models developed as a part of this work will be made publicly available.