GenAI4ELab/papercli-papers huggingface dataset:
BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models
Authors: Marvin Lavechin; Yaya Sy; Hadrien Titeux; MarÃ­a Andrea Cruz BlandÃ³n; Okko RÃ¤sÃ¤nen; HervÃ© Bredin; Emmanuel Dupoux; Alejandrina Cristia
Venue: Interspeech 2023 (main)

Abstract: Self-supervised techniques for learning speech representations have been shown to develop linguistic competence from exposure to speech without the need for human labels. In order to fully realize the potential of these approaches and further our understanding of how infants learn language, simulations must closely emulate real-life situations by training on developmentally plausible corpora and benchmarking against appropriate test sets. To this end, we propose a language-acquisition-friendly benchmark to probe spoken language models at the lexical and syntactic levels, both of which are compatible with the vocabulary typical of children's language experiences. This paper introduces the benchmark and summarizes a range of experiments showing its usefulness. In addition, we highlight two exciting challenges that need to be addressed for further progress: bridging the gap between text and speech and between clean speech and in-the-wild speech.
Data augmentation for children ASR and child-adult speaker classification using voice conversion methods
Authors: Shuyang Zhao; Mittul Singh; Abraham Woubie; Reima Karhila
Venue: Interspeech 2023 (main)

Abstract: Many young children prefer speech based interfaces over text, as they are relatively slow and error-prone with text input. However, children ASR can be challenging due to the lack of transcribed children speech corpora. In this paper, we investigate a voice conversion method based on WORLD vocoder to generate childlike speech for data augmentation. Since noise may lead to severe artifacts in converted speech, we also investigate using speech enhancement to improve the quality of converted speech. On a publicly available children speech corpus, we evaluated the performance of the proposed data augmentation method against existing data augmentation methods based on linear prediction coefficients. Our proposed data augmentation method substantially outperformed the prior work on children ASR. Additionally, on a task to classify the speaker, adult or child, data generated using our proposed method was shown to mimic real children better compared to the reference methods.
Developmental Articulatory and Acoustic Features for Six to Ten Year Old Children
Authors: Vishwas M. Shetty; Steven M. Lulich; Abeer Alwan
Venue: Interspeech 2023 (main)

Abstract: In this paper, we study speech development in children using longitudinal acoustic and articulatory data. Data were collected yearly from grade 1 to grade 4 from four female and four male children. We analyze acoustic and articulatory properties of four corner vowels: /Ã¦/, /i/, /u/, and /É/, each occurring in two different words (different surrounding contexts). Acoustic features include formant frequencies and subglottal resonances (SGRs). Articulatory features include tongue curvature degree (TCD) and tongue curvature position (TCP). Based on the analyses, we observe the emergence of sex-based differences starting from grade 2. Similar to adults, the SGRs divide the vowel space into high, low, front, and back regions at least as early as grade 2. On average, TCD is correlated with vowel height and TCP with vowel frontness. Children in our study used varied articulatory configurations to achieve similar acoustic targets.
Automatically Predicting Perceived Conversation Quality in a Pediatric Sample Enriched for Autism
Authors: Yahan Yang; Sunghye Cho; Maxine Covello; Azia Knox; Osbert Bastani; James Weimer; Edgar Dobriban; Robert Schultz; Insup Lee; Julia Parish-Morris
Venue: Interspeech 2023 (main)

Abstract: Social interaction quality ratings derived from short natural conversations can differentiate children with and without autism at the group level. In this work, we explored conversations between children and an unfamiliar adult who rated their social interaction success on six dimensions. Using hand-crafted acoustic and lexical features, we built different classifiers to predict children's dimensional conversation quality. The best classifier achieved 61% accuracy, which outperformed human raters (49%). Follow-up analyses revealed that a subset of features determined communication quality scores. Additionally, we extracted acoustic features using a pretrained audio transformer and improved our prediction to 68%. This study suggests that automatically predicting conversation quality could be an inexpensive and objective way to monitor intervention progress in children with communication challenges, and could be used to identify intervention targets for improving conversational success.
An Equitable Framework for Automatically Assessing Children's Oral Narrative Language Abilities
Authors: Alexander Johnson; Hariram Veeramani; Natarajan Balaji Shankar; Abeer Alwan
Venue: Interspeech 2023 (main)

Abstract: This work proposes a novel framework for automatically scoring children's oral narrative language abilities. We use audio recordings from 3rd-8th graders of the Atlanta, Georgia area as they take a portion of the Test of Narrative Language. We design a system which extracts linguistic features and fine-tuned BERT-based self-supervised learning representation from state-of-the-art ASR transcripts. We predict manual test scores from the extracted features. This framework significantly outperforms a deterministic method based on the assessment's scoring rubric. Last, we evaluate the system performance across student's reading level, dialect, and diagnosed learning/language disabilities to establish fairness across diverse demographics of students. Using this system, we achieve approximately 98% classification accuracy of student scores. We are also able to identify key areas of improvement for this type of system across demographic areas and reading ability.
An Analysis of Goodness of Pronunciation for Child Speech
Authors: Xinwei Cao; Zijian Fan; TorbjÃ¸rn Svendsen; Giampiero Salvi
Venue: Interspeech 2023 (main)

Abstract: In this paper, we study the use of goodness of pronunciation (GOP) on child speech. We first compare the distributions of GOP scores on several open datasets representing various dimensions of speech variability. We show that the GOP distribution over CMU Kids, corresponding to young age, has larger spread than those on datasets representing other dimensions, i.e., accent, dialect, spontaneity and environmental conditions. We hypothesize that the increased variability of pronunciation in young age may impair the use of traditional mispronunciation detection methods for children. To support this hypothesis, we perform simulated mispronunciation experiments both for children and adults using different variants of the GOP algorithm. We also compare the results to real-case mispronunciations for native children showing that GOP is less effective for child speech than for adult speech.
Measuring Language Development From Child-centered Recordings
Authors: Yaya Sy; William N. Havard; Marvin Lavechin; Emmanuel Dupoux; Alejandrina Cristia
Venue: Interspeech 2023 (main)

Abstract: Standard ways to measure child language development from spontaneous corpora rely on detailed linguistic descriptions of a language as well as exhaustive transcriptions of the child's speech, which today can only be done through costly human labor. We tackle both issues by proposing (1) a new language development metric (based on entropy) that does not require linguistic knowledge other than having a corpus of text in the language in question to train a language model, (2) a method to derive this metric directly from speech based on a smaller text-speech parallel corpus. Here, we present descriptive results on an open archive including data from six English-learning children as a proof of concept. We document that our entropy metric documents a gradual convergence of children's speech towards adults' speech as a function of age, and it also correlates moderately with lexical and morphosyntactic measures derived from morphologically-parsed transcriptions.
Speaking Clearly, Understanding Better: Predicting the L2 Narrative Comprehension of Chinese Bilingual Kindergarten Children Based on Speech Intelligibility Using a Machine Learning Approach
Authors: Hiuching Hung; Paula A. PÃ©rez-Toro; TomÃ¡s Arias-Vergara; Andreas Maier; Elmar NÃ¶th
Venue: Interspeech 2023 (main)

Abstract: This study investigated the relationship of speech intelligibility and the narrative comprehension among bilingual kindergarten children and how well the speech intelligibility of second language (L2) predicted the L2 narrative comprehension using a machine learning approach. Fifty Chinese-English bilingual children aged 5-6 years old participated in this study by taking a narrative comprehension test. Their L2 narrative comprehension was assessed using the MAIN test. The speech intelligibility was assessed in terms of twenty-four features that encode confidence levels with respect to phoneme and word classifiers trained on native speaker speech data. Our hypothesis posits that it is possible to predict L2 narrative comprehension based on speech intelligibility features. By using seven out of the twenty-four considered features we were able to make predictions of the MAIN test scores with an RMSE of 2.13 and a Pearson correlation coefficient of 0.468 based on a data set of 50 bilingual kindergarten children. We conclude the paper by providing pedagogical implications for second language teaching as well as suggestions for future work.
Speech Breathing Behavior During Pauses in Children
Authors: Delphine Charuau; BÃ©atrice Vaxelaire; Rudolph Sock
Venue: Interspeech 2023 (main)

Abstract: The aim of this paper is to investigate speech breathing behaviors in children during the realization of pauses, breathing pauses or non-breathing pauses, depending on the syntactic location of the pause and the speech task. Thus, we will be able to observe the effects of cognitive-linguistic demands on breathing patterns in children. To do so, 10 French speakers, between 8 and 11 years old, were recorded while reading and spontaneous speech. The variation of respiratory movements was measured using inductive respiratory plethysmography. The respiratory signals were synchronized with acoustic data. The results show an effect of the speech task on the duration of inhalation and, to a lesser extent, on its amplitude. 'Partial inhalations' were observed at the syntactic boundaries, suggesting that they are integrated in the child's respiratory patterns. Finally, we observed occasional cessation of rib cage closure, essentially during non-syntactic pauses.
Understanding Spoken Language Development of Children with ASD Using Pre-trained Speech Embeddings
Authors: Anfeng Xu; Rajat Hebbar; Rimita Lahiri; Tiantian Feng; Lindsay Butler; Lue Shen; Helen Tager-Flusberg; Shrikanth Narayanan
Venue: Interspeech 2023 (main)

Abstract: Speech processing techniques are useful for analyzing speech and language development in children with Autism Spectrum Disorder (ASD), who are often varied and delayed in acquiring these skills. Early identification and intervention are crucial, but traditional assessment methodologies such as caregiver reports are not adequate for the requisite behavioral phenotyping. Natural Language Sample (NLS) analysis has gained attention as a promising complement. Researchers have developed benchmarks for spoken language capabilities in children with ASD, obtainable through the analysis of NLS. This paper proposes applications of speech processing technologies in support of automated assessment of children's spoken language development by classification between child and adult speech and between speech and nonverbal vocalization in NLS, with respective F1 macro scores of 82.6% and 67.8%, underscoring the potential for accurate and scalable tools for ASD research and clinical use.
Measuring Phonological Precision in Children with Cleft Lip and Palate
Authors: TomÃ¡s Arias-Vergara; Elizabeth LondoÃ±o-Mora; Paula A. PÃ©rez-Toro; Maria Schuster; Elmar NÃ¶th; Juan Rafael Orozco-Arroyave; Andreas Maier
Venue: Interspeech 2023 (main)

Abstract: Children with Cleft Lip and Palate (CLP) may experience difficulties in oral communication, leading to other developmental problems such as delayed language acquisition and poor social skills; thus, early treatment is essential for successful speech rehabilitation. In this paper, we propose a methodology for automatically assessing the phonological precision of children with CLP. We propose to use the probabilities obtained from a phonological class recognizer to measure phonological precision during connected speech. Furthermore, we compute the nasal-to-sound ratio to improve the automatic detection of the nasality level. For this, we considered speech recordings of 88 children with CLP, assessed by a clinician according to four nasality levels: normal, mild, moderate, and severe. We obtained an F1-score of up to 0.54 for detecting the nasality level automatically. The results suggest that phonological analysis can be used for individualized speech rehabilitation.
A Study on Using Duration and Formant Features in Automatic Detection of Speech Sound Disorder in Children
Authors: Si-Ioi Ng; Cymie Wing-Yee Ng; Tan Lee
Venue: Interspeech 2023 (main)

Abstract: Speech sound disorder (SSD) in children is manifested by persistent articulation and phonological errors on specific phonemes of a language. Automatic SSD detection can be done using features extracted from deep neural network models. Interpretability of such learned features is a major concern. Motivated by clinical knowledge, the use of duration and formant features for SSD detection is investigated in this research. Acoustical analysis is performed to identify the acoustic features that differentiate between the speech of typical and disordered children. On the task of SSD detection in Cantonese-speaking children, the duration features are found to outperform the formant features and surpass previous methods that use paralinguistic feature set and speaker embeddings. Specifically, the duration features achieve a mean unweighted average recall of 71.0%. The results enhance the understanding of SSD, and motivate further use of temporal information of child speech in SSD detection.
Influence of Utterance and Speaker Characteristics on the Classification of Children with Cleft Lip and Palate
Authors: Ilja Baumann; Dominik Wagner; Franziska Braun; Sebastian P. Bayerl; Elmar NÃ¶th; Korbinian Riedhammer; Tobias Bocklet
Venue: Interspeech 2023 (main)

Abstract: Recent findings show that pre-trained wav2vec 2.0 models are reliable feature extractors for various speaker characteristics classification tasks. We show that latent representations extracted at different layers of a pre-trained wav2vec 2.0 system can be used as features for binary classification to distinguish between children with Cleft Lip and Palate (CLP) and a healthy control group. The results indicate that the distinction between CLP and healthy voices, especially with latent representations from the lower and middle encoder layers, reaches an accuracy of 100%. We test the classifier to find influencing factors for classification using unseen out-of-domain healthy and pathologic corpora with varying characteristics: age, spoken content, and acoustic conditions. Cross-pathology and cross-healthy tests reveal that the trained classifiers are unreliable if there is a mismatch between training and out-of-domain test data in, e.g., age, spoken content, or acoustic conditions.
Parameter-Efficient Low-Resource Dialogue State Tracking by Prompt Tuning
Authors: Mingyu Derek Ma; Jiun-Yu Kao; Shuyang Gao; Arpit Gupta; Di Jin; Tagyoung Chung; Nanyun Peng
Venue: Interspeech 2023 (main)

Abstract: Dialogue state tracking (DST) is an important step in dialogue management to keep track of users' beliefs. Existing works fine-tune all language model (LM) parameters to tackle the DST task, which requires significant data and computing resources for training and hosting. The cost grows exponentially in the real-world deployment where dozens of fine-tuned LM are used for different domains and tasks. To reduce parameter size and better utilize cross-task shared information, we propose to use soft prompt token embeddings to learn task properties. Without tuning LM parameters, our method drastically reduces the number of parameters needed to less than 0.5% of prior works while achieving better low-resource DST performance.
An Autoregressive Conversational Dynamics Model for Dialogue Systems
Authors: Matthew McNeill; Rivka Levitan
Venue: Interspeech 2023 (main)

Abstract: Conversational partners adapt their speech to one another in a phenomenon called entrainment. While entrainment behaviors are associated with a variety of positive conversational outcomes, they are rarely implemented in dialogue systems due to their poorly understood mechanics. Conversational dynamics models could discover entrainment behavior in a dialogue corpus, but to date they have not been designed for or evaluated in dialogue systems. In this paper, we propose an autoregressive model specifically for use in a dialogue system. We evaluate its ability to predict features for upcoming conversational turns, and show it outperforms several baseline models. Additionally, we analyze its attention mechanism to explain which turns it finds useful for predicting upcoming speech features. Finally, we discuss its potential for future deployment in a live dialogue system.
Style-transfer based Speech and Audio-visual Scene understanding for Robot Action Sequence Acquisition from Videos
Authors: Chiori Hori; Puyuan Peng; David Harwath; Xinyu Liu; Kei Ota; Siddarth Jain; Radu Corcodel; Devesh Jha; Diego Romeres; Jonathan Le Roux
Venue: Interspeech 2023 (main)

Abstract: To realize human-robot collaboration, robots need to execute actions for new tasks according to human instructions given finite prior knowledge. Human experts can share their knowledge of how to perform a task with a robot through multi-modal instructions in their demonstrations, showing a sequence of shorthorizon steps to achieve a longhorizon goal. This paper introduces a method for robot action sequence generation from instruction videos using (1) an audio-visual Transformer that converts audio-visual features and instruction speech to a sequence of robot actions called dynamic movement primitives (DMPs) and (2) style-transfer-based training that employs multi-task learning with video captioning and weakly-supervised learning with a semantic classifier to exploit unpaired video-action data. We built a system that accomplishes various cooking actions, where an arm robot executes a DMP sequence acquired from a cooking video using the audio-visual Transformer. Experiments with EpicKitchen100, YouCookII, QuerYD, and inhouse instruction video datasets show that the proposed method improves the quality of DMP sequences by 2.3 times the METEOR score obtained with a baseline video-to-action Transformer. The model achieved 32% of the task success rate with the task knowledge of the object.
Speech Aware Dialog System Technology Challenge (DSTC11)
Authors: Hagen Soltau; Izhak Shafran; Mingqiu Wang; Abhinav Rastogi; Jeffrey Zhao; Ye Jia; Wei Han; Yuan Cao; Aramys Miranda
Venue: Interspeech 2023 (main)

Abstract: Most research on task oriented dialog modeling is based on written text input. However, practical dialog systems often use spoken input. Typically, input speech is converted into text using ASR, which are error-prone. Furthermore, most systems don't address the differences in written and spoken language. The research on this topic is stymied by the lack of a public corpus. Motivated by these considerations, we hosted a speech-aware dialog state tracking challenge and created a public corpus which can be used to investigate the performance gap between the written and spoken input. We created three spoken versions of the popular written-domain MultiWoz task and provide waveforms, ASR transcripts, and audio encodings to encourage wider participation from teams that may not have access to ASR systems. In this paper, we describe the corpus, report results from participating teams, provide preliminary analyses of their results, and summarize the current state-of-the-art in this domain.
Knowledge-Retrieval Task-Oriented Dialog Systems with Semi-Supervision
Authors: Yucheng Cai; Hong Liu; Zhijian Ou; Yi Huang; Junlan Feng
Venue: Interspeech 2023 (main)

Abstract: Most existing task-oriented dialog (TOD) systems track dialog states in terms of slots and values and use them to query a database to get relevant knowledge to generate responses. In real life applications, user utterances are noisier, and thus it is more difficult to accurately track dialog states and correctly secure relevant knowledge. Recently, a progress in question answering and document-grounded dialog systems is retrieval-augmented methods with a knowledge retriever. Inspired by such progress, we propose a retrieval-based method to enhance knowledge selection in TOD systems, which significantly outperforms the traditional database query method for real-life dialogs. Further, we develop latent variable model based semi-supervised learning, which can work with the knowledge retriever to leverage both labeled and unlabeled dialog data. Joint Stochastic Approximation (JSA) algorithm is employed for semi-supervised model training, and the whole system is referred to as JSA-KRTOD. Experiments are conducted on a real-life dataset from China Mobile Custom-Service, called MobileCS, and show that JSA-KRTOD achieves superior performances in both labeled-only and semi-supervised settings.
Tracking Must Go On : Dialogue State Tracking with Verified Self-Training
Authors: Jihyun Lee; Chaebin Lee; Yunsu Kim; Gary Geunbae Lee
Venue: Interspeech 2023 (main)

Abstract: In task-oriented dialogues, dialogue state tracking (DST) is a critical component as it identifies specific information for the user's purpose. However, as annotating DST data requires a significant amount of human effort, leveraging raw dialogue is crucial. To address this, we propose a new self-training (ST) framework with a verification model. Unlike previous ST methods that rely on extensive hyper-parameter searching to filter out inaccurate data, our verification methodology ensures the accuracy and validity of the dataset without using a fixed threshold. Furthermore, to mitigate overfitting, we augment the dataset by generating diverse user utterances. Even when using only 10% of the labeled data, our approach achieves comparable results to a fully labeled MultiWOZ2.0 dataset. The evaluation of scalability also demonstrates enhanced robustness in predicting unseen values.
Ordered and Binary Speaker Embedding
Authors: Jiaying Wang; Xianglong Wang; Namin Wang; Lantian Li; Dong Wang
Venue: Interspeech 2023 (main)

Abstract: Modern speaker recognition systems represent utterances by embedding vectors. Conventional embedding vectors are dense and non-structural. In this paper, we propose an ordered binary embedding approach that sorts the dimensions of the embedding vector via a nested dropout and converts the sorted vectors to binary codes via Bernoulli sampling. The resultant ordered binary codes offer some important merits such as hierarchical clustering, reduced memory usage, and fast retrieval. These merits were empirically verified by comprehensive experiments on a speaker identification task with the VoxCeleb and CN-Celeb datasets.