GenAI4ELab/papercli-papers huggingface dataset:
Few-Shot Keyword-Incremental Learning with Total Calibration
Authors: Ilseok Kim; Ju-Seok Seong; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Keyword spotting (KWS) models need to continuously recognize new keywords for user demand. However, two significant challenges exist in satisfying this requirement: catastrophic forgetting, where the model loses its ability to classify previously learned keywords, and insufficient data for new classes. To address these challenges, we propose a Few-shot keyword-Incremental Learning with total caLibration (FILL), a novel few-shot class-incremental learning (FSCIL) approach for KWS. FSCIL trains a model with sufficient data in an initial session, followed by incremental sessions where it learns new classes with limited data. FILL employs prototype calibration throughout total sessions to enhance class separation and mitigate misclassification. Notably, it utilizes manifold mixup in the initial session to generate new classes for prototype calibration. Experimental results on two KWS datasets demonstrate that FILL outperforms three baselines in terms of average accuracy.
Leveraging Speech Data Diversity to Document Indigenous Heritage and Culture
Authors: Allahsera Tapo; Ãric Le Ferrand; Zoey Liu; Christopher Homan; Emily Prud'hommeaux
Venue: Interspeech 2024 (main)

Abstract: The majority of the world's 7,000 languages lack a standardized writing system. In this paper we consider one such language, Bambara, which is rarely written down but is widely spoken in Mali and neighboring countries. We explore the task of using automatic speech recognition (ASR) to transcribe culturally significant recordings focused on two domains: archival linguistic and anthropological fieldwork and contemporary oral histories performed by griots, the traditional Mande history keepers. We describe our two 6.5-hour corpora then experiment with different data configurations and multi-stage tuning from pretrained multilingual models within two neural ASR architectures. We find that while the diversity in content, style, and recording quality across the two corpora presents challenges, their commonalities can sometimes be leveraged to improve ASR accuracy. We note, however, that the diverse qualities of these corpora diminish their utility for cross-domain ASR training.
Missingness-resilient Video-enhanced Multimodal Disfluency Detection
Authors: Payal Mohapatra; Shamika Likhite; Subrata Biswas; Bashima Islam; Qi Zhu
Venue: Interspeech 2024 (main)

Abstract: Most existing speech disfluency detection techniques only rely upon acoustic data. In this work, we present a practical multimodal disfluency detection approach that leverages available video data together with audio. We curate an audio-visual dataset and propose a novel fusion technique with unified weight-sharing modality-agnostic encoders to learn the temporal and semantic context. Our resilient design accommodates real-world scenarios where the video modality may sometimes be missing during inference. We also present alternative fusion strategies when both modalities are assured to be complete. In experiments across five disfluency-detection tasks, our unified multimodal approach significantly outperforms Audio-only unimodal methods, yielding an average absolute improvement of 10% (i.e., 10 percentage point increase) when both video and audio modalities are always available, and 7% even when video modality is missing in half of the samples.
AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection
Authors: Rong Gong; Hongfei Xue; Lezhi Wang; Xin Xu; Qisheng Li; Lei Xie; Hui Bu; Shaomei Wu; Jiaming Zhou; Yong Qin; Binbin Zhang; Jun Du; Jia Bin; Ming Li
Venue: Interspeech 2024 (main)

Abstract: The rapid advancements in speech technologies over the past two decades have led to human-level performance in tasks like automatic speech recognition (ASR) for fluent speech. However, the efficacy of these models diminishes when applied to atypical speech, such as stuttering. This paper introduces AS-70, the first publicly available Mandarin stuttered speech dataset, which stands out as the largest dataset in its category. Encompassing conversational and voice command reading speech, AS-70 includes verbatim manual transcription, rendering it suitable for various speech-related tasks. Furthermore, baseline systems are established, and experimental results are presented for ASR and stuttering event detection (SED) tasks. By incorporating this dataset into the model fine-tuning, significant improvement in the state-of-the-art ASR models, e.g., Whisper and Hubert, are observed, enhancing their inclusivity in addressing stuttered speech.
Analyzing Speech Motor Movement using Surface Electromyography in Minimally Verbal Adults with Autism Spectrum Disorder
Authors: Wazeer Zulfikar; Nishat Protyasha; Camila Canales; Heli Patel; James Williamson; Laura Sarnie; Lisa Nowinski; Nataliya Kosmyna; Paige Townsend; Sophia Yuditskaya; Tanya Talkar; Utkarsh Oggy Sarawgi; Christopher McDougle; Thomas Quatieri; Pattie Maes; Maria Mody
Venue: Interspeech 2024 (main)

Abstract: Adults who are minimally verbal with autism spectrum disorder (mvASD) have pronounced speech difficulties linked to impaired motor skills. Existing research and clinical assessments primarily use indirect methods such as standardized tests, video-based facial features, and handwriting tasks, which may not directly target speech-related motor skills. In this study, we measure activity from eight facial muscles associated with speech using surface electromyography (sEMG), during carefully designed tasks. The findings reveal a higher power in the sEMG signals and a significantly greater correlation between the sEMG channels in mvASD adults (N=12) compared to age and gender-matched neurotypical controls (N=14). This suggests stronger muscle activation and greater synchrony in the discharge patterns of motor units. Further, eigenvalues derived from correlation matrices indicate lower complexity in muscle coordination in mvASD, implying fewer degrees of freedom in motor control.
Prosody of speech production in latent post-stroke aphasia
Authors: Cong Zhang; Tong Li; Gayle DeDe; Christos Salis
Venue: Interspeech 2024 (main)

Abstract: This study explores prosodic production in latent aphasia, a mild form of aphasia associated with left-hemisphere brain damage (e.g. stroke). Unlike prior research on moderate to severe aphasia, we investigated latent aphasia, which can seem to have very similar speech production with neurotypical speech. We analysed the f0, intensity and duration of utterance-initial and utterance-final words of ten speakers with latent aphasia and ten matching controls. Regression models were fitted to improve our understanding of this understudied type of very mild aphasia. The results highlighted varying degrees of differences in all three prosodic measures between groups. We also investigated the diagnostic classification of latent aphasia versus neurotypical control using random forest, aiming to build a fast and reliable tool to assist with the identification of latent aphasia. The random forest analysis also reinforced the significance of prosodic features in distinguishing latent aphasia.
MMSD-Net: Towards Multi-modal Stuttering Detection
Authors: Liangyu Nie; Sudarsana Reddy Kadiri; Ruchit Agrawal
Venue: Interspeech 2024 (main)

Abstract: Stuttering is a common speech impediment that is caused by irregular disruptions in speech production, affecting over 70 million people across the world. Standard automatic speech processing tools do not take speech ailments into account and are thereby not able to generate meaningful results when presented with stuttered speech as input. The automatic detection of stuttering is an integral step towards building efficient, context-aware speech processing systems. While previous approaches explore both statistical and neural approaches for stuttering detection, all of these methods are uni-modal in nature. This paper presents MMSD-Net, the first multi-modal neural framework for stuttering detection. Experiments and results demonstrate that incorporating the visual signal significantly aids stuttering detection, and our model yields an improvement of 2-17% in the F1-score over existing state-of-the-art uni-modal approaches.
Large Language Models for Dysfluency Detection in Stuttered Speech
Authors: Dominik Wagner; Sebastian P. Bayerl; Ilja Baumann; Elmar Noeth; Korbinian Riedhammer; Tobias Bocklet
Venue: Interspeech 2024 (main)

Abstract: Accurately detecting dysfluencies in spoken language can help to improve the performance of automatic speech and language processing components and support the development of more inclusive speech and language technologies. Inspired by the recent trend towards the deployment of large language models (LLMs) as universal learners and processors of non-lexical inputs, such as audio and video, we approach the task of multi-label dysfluency detection as a language modeling problem. We present hypotheses candidates generated with an automatic speech recognition system and acoustic representations extracted from an audio encoder model to an LLM, and finetune the system to predict dysfluency labels on three datasets containing English and German stuttered speech. The experimental results show that our system effectively combines acoustic and lexical information and achieves competitive results on the multi-label stuttering detection task.
Preliminary Investigation of Psychometric Properties of a Novel Multimodal Dialog Based Affect Production Task in Children and Adolescents with Autism
Authors: Carly Demopoulos; Linnea Lampinen; Cristian Preciado; Hardik Kothare; Vikram Ramanarayanan
Venue: Interspeech 2024 (main)

Abstract: Impairments in nonverbal communication are a defining feature of autism spectrum disorder (ASD) and can manifest as difficulty with, or even complete lack of, communication of emotional states via production of facial affect or vocal affect. The purpose of this study was to evaluate psychometric properties of a novel multimodal dialog based Affect Production Task (APT) in children and adolescents (ages 8-17) with a diagnosis of autism (N=72) or neurotypical controls (N=37). Participants completed activities designed to quantify objective facial and vocal affect production ability using audiovisual capture. Criterion, ecological, and discriminant validity were assessed. Psychometric performance across task conditions, age, sex, and race-ethnicity also was examined. Results of this initial psychometric evaluation suggest that the APT is a valid measure of affect production abilities in children and adolescents, and that psychometric performance is invariant to age, sex, or race/ethnicity.
Training speech-breathing coordination in computer-assisted reading
Authors: Delphine Charuau; Andrea Briglia; Erika Godde; GÃ©rard Bailly
Venue: Interspeech 2024 (main)

Abstract: Aims of this study are: 1) identify respiratory features that could serve as objective markers of fluency of aloud reading; 2) investigate the effects of computer-assisted training on the improvement of speech-breathing coordination. Our training method combines the principles of repeated and assisted close-shadowed reading. Reading assistance takes over the principles of karaoke: highlighting text units in sync with pre-recorded adult performance. The aloud reading of 66 French young pupils is studied. They were divided into 3 groups: control (repeated reading only with no assistance), trained with word highlighting vs. trained with word and breath group highlighting. All children were recorded before and after 3 weeks of training while reading both a trained and an untrained text. The results indicated that respiratory planning and pauses management improved by computer-assisted reading training. However, there was no significant transfer of these improvements to the untrained text.
How Does Alignment Error Affect Automated Pronunciation Scoring in Children's Speech?
Authors: Prad Kadambi; Tristan Mahr; Lucas Annear; Henry Nomeland; Julie Liss; Katherine Hustad; Visar Berisha
Venue: Interspeech 2024 (main)

Abstract: Automated goodness of pronunciation scores measure deviation from typical adult speech by first phonetically segmenting speech using forced alignment and then computing phoneme likelihoods. Care must be taken to distinguish between the impact of alignment error (a spurious signal) and true acoustic deviation on the automated score. Using mixed effects modeling, we predict âPLLR, the difference between pronunciation scores computed using manual alignment (PLLRm) versus computed using automatic forced alignments (PLLRa). Pronunciation deviations and alignment error are both magnified in childrenâs speech and may be influenced by factors such as phoneme position and phoneme type. Our methodology shows that alignment error has a moderate effect on âPLLR, and other variables have small to no effect. Manual PLLR closely matches automatically calculated PLLR following cross utterance averaging. Thus, practical comparisons between child speakers should be very comparable across the two methods.
Examining Vocal Tract Coordination in Childhood Apraxia of Speech with Acoustic-to-Articulatory Speech Inversion Feature Sets
Authors: Nina R. Benway; Jonathan L. Preston; Carol Espy-Wilson
Venue: Interspeech 2024 (main)

Abstract: Childhood apraxia of speech is a genetically driven, neurodevelopmental speech sound disorder with speech deficits theorized to reflect difficulty in the spatiotemporal programming of speech movements. Therefore, this work examined how well articulatory coordination features generated from audio-estimated kinematic data distinguished speakers with childhood apraxia of speech versus non-apraxic speech sound disorder. Two correlation-based feature sets motivated by recent literature demonstrated high performance in replicated 6-fold nested cross validated studies, with no statistically significant difference between feature sets (mean AUROC = .90, Ï = .04). An ablation study emphasized the importance of source-filter coordination in this population of apraxic speakers, with the source-ablated feature set performing significantly worse than the lip-ablated, the tongue-ablated, and full feature set (ÎAUROC = -.19, SE = 0.01, p < .001).
Childrenâs Speech Recognition through Discrete Token Enhancement
Authors: Vrunda N. Sukhadia; Shammur Absar Chowdhury
Venue: Interspeech 2024 (main)

Abstract: Childrenâs speech recognition is considered a low-resource task mainly due to the lack of publicly available data. There are several reasons for such data scarcity, including expensive data collection and annotation processes, and data privacy, among others. Transforming speech signals into discrete tokens that do not carry sensitive information but capture both linguistic and acoustic information could be a solution for privacy concerns. In this study, we investigate the integration of discrete speech tokens into childrenâs speech recognition systems as input without significantly degrading the ASR performance. Additionally, we explored single-view and multi-view strategies for creating these discrete labels. Furthermore, we tested the models for generalization capabilities with unseen domain and nativity dataset. Results reveal that the discrete token ASR for children achieves nearly equivalent performance with an approximate 83% reduction in parameters.
Bridging Child-Centered Speech Language Identification and Language Diarization via Phonetics
Authors: Yujia Wang; Hexin Liu; Leibny Paola Garcia
Venue: Interspeech 2024 (main)

Abstract: Language Diarization (LD) can be viewed as an expansion of Language Identification (LID) that removes the monolingual input assumption. Taking inspiration from this connection and the challenges inherent in Code-Switching (CS) child-centered speech, we extended PHO-LID, an LID model that incorporates acoustic and phonotactic information without needing phoneme annotation, to LD. Our method explores three avenues to adapt PHO-LID into LD: a temporal slicing scheme bridging LID and LD, an embedding modification enriching LD message, and a back-end scoring facilitating fine-tuning. Compared to the baseline, trained on a simulated out-of-domain dataset, SEAME_sim, our method shows a 15.82% relative accuracy improvement on MERLIon, a child-centered CS speech corpus. The back-end scoring preserves pre-trained knowledge in fine-tuning, with a 16.93% relative accuracy improvement on pre-trained SEAME_sim test set without compromising the fine-tuning test set performance.
Reading Miscue Detection in Primary School through Automatic Speech Recognition
Authors: Lingyun Gao; Cristian Tejedor-Garcia; Helmer Strik; Catia Cucchiarini
Venue: Interspeech 2024 (main)

Abstract: Automatic reading diagnosis systems can benefit both teachers for more efficient scoring of reading exercises and students for accessing reading exercises with feedback more easily. However, there are limited studies on Automatic Speech Recognition (ASR) for child speech in languages other than English, and limited research on ASR-based reading diagnosis systems. This study investigates how efficiently state-of-the-art (SOTA) pretrained ASR models recognize Dutch native children speech and manage to detect reading miscues. We found that Hubert Large finetuned on Dutch speech achieves SOTA phoneme-level child speech recognition (PER at 23.1%), while Whisper (Faster Whisper Large-v2) achieves SOTA word-level performance (WER at 9.8%). Our findings suggest that Wav2Vec2 Large and Whisper are the two best ASR models for reading miscue detection. Specifically, Wav2Vec2 Large shows the highest recall at 0.83, whereas Whisper exhibits the highest precision at 0.52 and an F1 score of 0.52.
Automatic Evaluation of a Sentence Memory Test for Preschool Children
Authors: Ilja Baumann; Nicole Unger; Dominik Wagner; Korbinian Riedhammer; Tobias Bocklet
Venue: Interspeech 2024 (main)

Abstract: Assessment of memory capabilities in preschool-aged children is crucial for early detection of potential speech development impairments or delays. We present an approach for the automatic evaluation of a standardized sentence memory test specifically for preschool children. Our methodology leverages automatic transcription of recited sentences and evaluation based on natural language processing techniques. We demonstrate the effectiveness of our approach on a dataset comprised of recited sentences from preschool-aged children, incorporating ratings of semantic and syntactic correctness. The best performing systems achieve an F1 score of 91.7% for semantic correctness and 86.1% for syntactic correctness using automatic transcripts. Our results showcase the potential of automated evaluation systems in providing reliable and efficient assessments of memory capabilities in early childhood, facilitating timely interventions and support for children with language development needs.
Enhancing Child Vocalization Classification with Phonetically-Tuned Embeddings for Assisting Autism Diagnosis
Authors: Jialu Li; Mark Hasegawa-Johnson; Karrie Karahalios
Venue: Interspeech 2024 (main)

Abstract: The assessment of children at risk of autism typically involves a clinician observing, taking notes, and rating childrenâs behaviors. A machine learning model that can label adult and child audio may largely save labor in coding children's behaviors, helping clinicians capture critical events and better communicate with parents. In this study, we leverage Wav2Vec 2.0 (W2V2), pre-trained on 4300-hour of home audio of children under 5 years old, to build a unified system for tasks of clinician-child speaker diarization and vocalization classification (VC). To enhance childrenâs VC, we build a W2V2 phoneme recognition system for children under 4 years old, and we incorporate its phonetically-tuned embeddings as auxiliary features or recognize pseudo phonetic transcripts as an auxiliary task. We test our method on two corpora (Rapid-ABC and BabbleCor) and obtain consistent improvements. Additionally, we outperform the state-of-the-art performance on the reproducible subset of BabbleCor.
Self-Supervised Models for Phoneme Recognition: Applications in Children's Speech for Reading Learning
Authors: Lucas Block Medin; Thomas Pellegrini; Lucile Gelin
Venue: Interspeech 2024 (main)

Abstract: Child speech recognition is still an underdeveloped area of research due to the lack of data (especially on non-English languages) and the specific difficulties of this task. Having explored various architectures for child speech recognition in previous work, in this article we tackle recent self-supervised models. We first compare wav2vec 2.0, HuBERT and WavLM models adapted to phoneme recognition in French child speech, and continue our experiments with the best of them, WavLM base+. We then further adapt it by unfreezing its transformer blocks during fine-tuning on child speech, which greatly improves its performance and makes it significantly outperform our base model, a Transformer+CTC. Finally, we study in detail the behaviour of these two models under the real conditions of our application, and show that WavLM base+ is more robust to various reading tasks and noise levels.
Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Models
Authors: Ruchao Fan; Natarajan Balaji Shankar; Abeer Alwan
Venue: Interspeech 2024 (main)

Abstract: Speech foundation models (SFMs) have achieved state-of- the-art results for various speech tasks in supervised (e.g. Whisper) or self-supervised systems (e.g. WavLM). However, the performance of SFMs for child ASR has not been systematically studied. In addition, there is no benchmark for child ASR with standard evaluations, making the comparisons of novel ideas difficult. In this paper, we initiate and present a comprehensive benchmark on several child speech databases based on various SFMs (Whisper, Wav2vec2.0, HuBERT, and WavLM). Moreover, we investigate finetuning strategies by comparing various data augmentation and parameter-efficient finetuning (PEFT) methods. We observe that the behaviors of these methods are different when the model size increases. For example, PEFT matches the performance of full finetuning for large models but worse for small models. To stabilize finetuning using augmented data, we propose a perturbation invariant finetuning (PIF) loss as a regularization.
Introduction To Partial Fine-tuning: A Comprehensive Evaluation Of End-to-end Childrenâs Automatic Speech Recognition Adaptation
Authors: Thomas Rolland; Alberto Abad
Venue: Interspeech 2024 (main)

Abstract: Automatic Speech Recognition (ASR) encounters unique challenges when dealing with children's speech, mainly due to the scarcity of available data. Training large ASR models with constrained data presents a significant challenge. To address this, fine-tuning strategy is frequently employed. However, fine-tuning an entire large pre-trained model with limited children's speech data may overfit leading to decreased performance. This study offers a granular evaluation of children's ASR fine-tuning, departing from conventional whole-network tunning. We present a partial fine-tuning approach spotlighting the importance of the Encoder and Feedforward Neural Network modules in Transformer-based models. Remarkably, this method surpasses the efficacy of whole-model fine-tuning, with a relative word error rate improvement of 9\% when dealing with limited data. Our findings highlight the critical role of partial fine-tuning in advancing children's ASR model development.