GenAI4ELab/papercli-papers huggingface dataset:
LAMASSU: A Streaming Language-Agnostic Multilingual Speech Recognition and Translation Model Using Neural Transducers
Authors: Peidong Wang; Eric Sun; Jian Xue; Yu Wu; Long Zhou; Yashesh Gaur; Shujie Liu; Jinyu Li
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) and speech translation (ST) can both use neural transducers as the model structure. It is thus possible to use a single transducer model to perform both tasks. In real-world applications, such joint ASR and ST models may need to be streaming and do not require source language identification (i.e. language-agnostic). In this paper, we propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers. Based on the transducer model structure, we propose four methods, a unified joint and prediction network for multilingual output, a clustered multilingual encoder, target language identification for encoder, and connectionist temporal classification regularization. Experimental results show that LAMASSU not only drastically reduces the model size but also reaches the performances of monolingual ASR and bilingual ST models.
DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
Authors: Yifan Peng; Yui Sudo; Shakeel Muhammad; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning (SSL) has achieved notable success in many speech processing tasks, but the large model size and heavy computational cost hinder the deployment. Knowledge distillation trains a small student model to mimic the behavior of a large teacher model. However, the student architecture usually needs to be manually designed and will remain fixed during training, which requires prior knowledge and can lead to suboptimal performance. Inspired by recent success of task-specific structured pruning, we propose DPHuBERT, a novel task-agnostic compression method for speech SSL based on joint distillation and pruning. Experiments on SUPERB show that DPHuBERT outperforms pure distillation methods in almost all tasks. Moreover, DPHuBERT requires little training time and performs well with limited training data, making it suitable for resource-constrained applications. Our method can also be applied to various speech SSL models. Our code and models will be publicly available.
Automatic Data Augmentation for Domain Adapted Fine-Tuning of Self-Supervised Speech Representations
Authors: Salah Zaiem; Titouan Parcollet; Slim Essid
Venue: Interspeech 2023 (main)

Abstract: Self-Supervised Learning (SSL) has allowed leveraging large amounts of unlabeled speech data to improve the performance of speech recognition models even with small annotated datasets. Despite this, speech SSL representations may fail while facing an acoustic mismatch between the pretraining and target datasets. To address this issue, we propose a novel supervised domain adaptation method, designed for cases exhibiting such a mismatch in acoustic domains. It consists in applying properly calibrated data augmentations on a large clean dataset, bringing it closer to the target domain, and using it as part of an initial fine-tuning stage. Augmentations are automatically selected through the minimization of a conditional-dependence estimator, based on the target dataset. The approach is validated during an oracle experiment with controlled distortions and on two amateur-collected low-resource domains, reaching better performances compared to the baselines in both cases.
Dual Acoustic Linguistic Self-supervised Representation Learning for Cross-Domain Speech Recognition
Authors: Zhao Yang; Dianwen Ng; Chong Zhang; Xiao Fu; Rui Jiang; Wei Xi; Yukun Ma; Chongjia Ni; Eng Siong Chng; Bin Ma; Jizhong Zhao
Venue: Interspeech 2023 (main)

Abstract: The integration of well-pre-trained acoustic and linguistic representations boosts the performance of speech-to-text cross-modality tasks. However, the potential of fine-tuning cross-modality integrated model on accented and noisy corpus is still under-explored. To address this gap, we propose an end-to-end acoustic and linguistic integrated representation learning model, namely Dual-w2v-BART. Our model incorporates acoustic representations from wav2vec2.0 and linguistic information from BART model by utilizing the cross-attention mechanism in the decoder, with paired speech-text dual inputs. To enhance model robustness on accent and noise, we propose a text-centric representation consistency component that helps to gain the similarity between different modality inputs while representing the same content. The results on accented and noisy speech recognition tasks demonstrate the effectiveness of the proposed model for reducing error rates compared to baseline and other competitive models.
O-1: Self-training with Oracle and 1-best Hypothesis
Authors: Murali Karthick Baskar; Andrew Rosenberg; Bhuvana Ramabhadran; Kartik Audhkhasi
Venue: Interspeech 2023 (main)

Abstract: We introduce O-1, a new self-training objective to reduce training bias and unify training and evaluation metrics for speech recognition. O-1 is a faster variant of Expected Minimum Bayes Risk (EMBR), that boosts the oracle hypothesis and can accommodate both supervised and unsupervised data. We demonstrate the effectiveness of our approach in terms of recognition on publicly available SpeechStew datasets and a large-scale, in-house data set. On Speechstew, the O-1 objective closes the gap between the actual and oracle performance by 80% relative compared to EMBR which bridges the gap by 43% relative. O-1 achieves 13% to 25% relative improvement over EMBR on the various datasets that SpeechStew comprises of, and a 12% relative gap reduction with respect to the oracle WER over EMBR training on the in-house dataset. Overall, O-1 results in a 9% relative improvement in WER over EMBR, thereby speaking to the scalability of the proposed objective for large-scale datasets.
MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets
Authors: Ziyang Ma; Zhisheng Zheng; Changli Tang; Yujin Wang; Xie Chen
Venue: Interspeech 2023 (main)

Abstract: In this paper, we provide a new perspective on self-supervised speech models from how the training targets are obtained. We generalize the targets extractor into Offline Targets Extractor (Off-TE) and Online Targets Extractor (On-TE). Based on this, we propose a new multi-tasking learning framework for self-supervised learning, MT4SSL, which stands for Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets. MT4SSL uses the K-means algorithm as an Off-TE and a teacher network without gradients as an On-TE, respectively. Our model outperforms previous SSL methods by nontrivial margins on the LibriSpeech benchmark, and is comparable to or even better than the best-performing models with fewer data. Furthermore, we find that using both Off-TE and On-TE results in better convergence in the pre-training phase. With both effectiveness and efficiency, we think doing multi-task learning on self-supervised speech models from our perspective is a promising trend.
Comparing Self-Supervised Pre-Training and Semi-Supervised Training for Speech Recognition in Languages with Weak Language Models
Authors: LÃ©a-Marie Lam-Yee-Mui; Lucas Ondel Yang; OndÅej Klejch
Venue: Interspeech 2023 (main)

Abstract: This paper investigates the potential of improving a hybrid automatic speech recognition model trained on 10 hours of transcribed data with 200 hours of untranscribed data in low-resource languages. First, we compare baseline methods of cross-lingual transfer with MFCC features and features extracted with the multilingual self-supervised model XLSR-53. Subsequently, we compare two approaches that can leverage the untranscribed data: semi-supervised training with LF-MMI and continued self-supervised pre-training of XLSR-53. Our results on well-resourced English broadcast data derived from MGB show that both methods achieve 18% and 27% relative improvements compared to the baseline, respectively. On the low-resource South African Soap Opera dataset, the relative improvement with semi-supervised training is only 3% due to the inherently weak language model. However, continued pre-training achieves 8.6% relative improvement because it does not rely on any external information.
Chinese EFL Learnersâ Perception of English Prosodic Focus
Authors: Xinya Zhang; Ying Chen
Venue: Interspeech 2023 (main)

Abstract: Focus in a sentence can be realized prosodically in speech communication. It has been found not easy for L2 learners to acquire. The present study examines Chinese learners' perception of English prosodic focus, specifically the effects of learners' English proficiency, intonation type, sentence length, and focus location on the perceptual accuracy of English prosodic focus by Chinese EFL learners. Results of two trials in the perception experiment reveal that focus location, intonation type, and English proficiency significantly impacted Chinese learners' perceptual accuracy of both single focus and dual focus in English. Focus in statements was perceived more accurately than that in questions for both single focus and dual focus. Focus located on sentence-final words in questions was perceived more accurately than that on non-final words in questions. Learners' English proficiency positively correlated to the accuracy of focus perception, especially for dual focus.
Pitch Accent Variation and the Interpretation of Rising and Falling Intonation in American English
Authors: Thomas Sostarics; Jennifer Cole
Venue: Interspeech 2023 (main)

Abstract: This study tests the division of labor in the meaning conveyed by pitch accents and edge tones in English intonation. In three perception studies, we investigate where the locus of the contrast between an assertive vs inquisitive interpretation resides. By doing so, we also gain insight into the role of potentially meaningful within- and between-category variation in the phonetic implementation of discrete intonational tunes. We find that the pitch accent does not contribute to assertive interpretation. Rather, the distinction between assertive and inquisitive interpretation is cued primarily by the final F0 of the pitch contour regardless of the pitch accent, but that increased overall pitch prominence may trigger a salient focus interpretation that interferes with judging assertiveness.
Tonal coarticulation as a cue for upcoming prosodic boundary
Authors: Jianjing Kuang; May Pik Yu Chan; Nari Rhee
Venue: Interspeech 2023 (main)

Abstract: It has been established that the lack of tonal coarticulation or pitch reset is a salient cue for the beginning of a large prosodic domain, however, it is yet unclear whether tonal coarticulation can be an informative cue for the end of a prosodic domain. We examined this question with two continuous speech corpora of Mandarin, and both expert and crowd-sourced perceptual annotations were used. The FPCA model of the holistic tonal contours shows that the carry-over effect of the preceding tone is significantly affected by the strength of the following boundaries. Stronger carry-over effects are associated with the end of larger prosodic boundaries. Moreover, machine learning classification shows that the fine-grained tonal coarticulation patterns are salient cues for predicting larger prosodic boundaries. This result is further validated by crowd-sourced boundary perceptual ratings from human listeners. This study has important implications for the understanding of prosodic phrasing.
Alignment of Beat Gestures and Prosodic Prominence in German
Authors: Sophie Repp; Lara Muhtz; Johannes Heim
Venue: Interspeech 2023 (main)

Abstract: We present evidence on the alignment of beat gestures and prosodic prominence from a video corpus consisting of six German educational videos for students from six presenters. Our analysis of 120 beat gestures (with a substantial variety of hand shapes) shows that beat gestures almost always align with prosodically prominent syllables, i.e., syllables carrying a pitch accent. Specifically, the stroke always starts before, or - more often - on, a pitch-accented syllable; the apex mostly falls on the accented syllable (74%) but may also occur in subsequent syllables. The degree of prosodic prominence of the accented syllable (in terms of DIMA-prominence levels) is predictive for the position of the apex, which occurs within rather than after the accented syllable more often for higher degrees of prominence. These findings provide new insights into the alignment of prominence-lending features of prosody and gesture, thereby broadening the empirical landscape for beat gestures.
Creak Prevalence and Prosodic Context in Australian English
Authors: Hannah White; Joshua Penney; Andy Gibson; Anita Szakay; Felicity Cox
Venue: Interspeech 2023 (main)

Abstract: Creaky voice has been found to mark phrase-finality in many varieties of English, as well as in other languages. The present study aims to investigate whether this is also true for Australian English (AusE), a variety that is understudied in creaky voice research. Using automatic creak detection methods, the need for manual annotation of creak is reduced, and we are able to analyse a large dataset of Australian teenagers' speech. As in other varieties, creak is found to be a marker of finality in AusE. Additionally, we find that males use higher rates of creaky voice than females, challenging the widely held assumption that creak is a feature of female speech.
Speech reduction: position within French prosodic structure
Authors: KÃ¼bra Bodur; Roxane Bertrand; James S. German; StÃ©phane Rauzy; Corinne Fredouille; Christine Meunier
Venue: Interspeech 2023 (main)

Abstract: Variation in the speech signal is a characteristic of spoken language, emerging partially as a result of interactions between various linguistic levels. One example of variation is phonetic reduction, where words are produced with missing or underspecified phonetic forms. Using a French conversational corpus, this paper focuses on the relationship between reduction and prosodic structure to see whether certain positions favor the occurrence of reduction. We annotated and observed the distribution of reduced sequences within specific prosodic domains (Intonational and Accentual Phrases). Preliminary analyses revealed that the detected reductions occur mostly mid- IP and very rarely at IP-final. However, this pattern may vary among speakers, as speakers have different patterns in terms of the number of reductions produced and their positions. It is also usually the case that the reduced sequences occurring mid-IP, coincide with the AP level boundaries, extending from one AP to another.
Transvelar Nasal Coupling Contributing to Speaker Characteristics in Non-nasal Vowels
Authors: Ziyu Zhu; Yujie Chi; Zhao Zhang; Kiyoshi Honda; Jianguo Wei
Venue: Interspeech 2023 (main)

Abstract: Nasal-cavity structure is stable in speech and varied across speakers, which potentially gives rise to speaker characteristics. Many studies have reported the acoustic contribution of the nasal cavity for nasal and nasalized sounds with velopharyngeal port opening. However, nasal-cavity resonance does emerge in non-nasal vowels through transvelar nasal coupling, which results in non-negligible modifications to non-nasal vowel spectra. In this study, nasal and oral output sounds were separately recorded during non-nasal utterances, and spectral analysis was conducted. The results indicate clear inter-speaker variability in two spectral measures below 2 kHz: frequency location of double-peaked first nasal-cavity resonance and inconsistent distribution of minor dips above the first resonance. It was also observed that nostril outputs modulate oral output signals to lower the first formant frequency of naturally produced non-low vowels, which also exhibited varied degrees across speakers.
Speech Synthesis from Articulatory Movements Recorded by Real-time MRI
Authors: Yuto Otani; Shun Sawada; Hidefumi Ohmura; Kouichi Katsurada
Venue: Interspeech 2023 (main)

Abstract: Previous speech synthesis models from articulatory movements recorded using real-time MRI (rtMRI) only predicted vocal tract shape parameters and required additional pitch information to generate a speech waveform. This study proposes a two-stage deep learning model composed of CNN-BiLSTM that predicts a mel-spectrogram from a rtMRI video and a HiFi-GAN vocoder that synthesizes a speech waveform. We evaluated our model on two databases: the ATR 503 sentences rtMRI database and the USC-TIMIT database. The experimental results on the ATR 503 sentences rtMRI database show that the PESQ score and the RMSE of F0 are 1.64 and 26.7 Hz. This demonstrates that all acoustic parameters, including fundamental frequency, can be estimated from the rtMRI videos. In the experiment on the USC-TIMIT database, we obtained a good PESQ score and RMSE for F0. However, the synthesized speech is unclear, indicating that the quality of the datasets affects the intelligibility of the synthesized speech.
The ART of Conversation: Measuring Phonetic Convergence and Deliberate Imitation in L2-Speech with a Siamese RNN
Authors: Zheng Yuan; Aldo Pastore; Dorina de Jong; Hao Xu; Luciano Fadiga; Alessandro D'Ausilio
Venue: Interspeech 2023 (main)

Abstract: Phonetic convergence describes the automatic and unconscious speech adaptation of two interlocutors in a conversation. This paper proposes a Siamese recurrent neural network (RNN) architecture to measure the convergence of the holistic spectral characteristics of speech sounds in an L2-L2 interaction. We extend an alternating reading task (the ART) dataset by adding 20 native Slovak L2 English speakers. We train and test the Siamese RNN model to measure phonetic convergence of L2 English speech from three different native language groups: Italian (9 dyads), French (10 dyads) and Slovak (10 dyads). Our results indicate that the Siamese RNN model effectively captures the dynamics of phonetic convergence and the speaker's imitation ability. Moreover, this text-independent model is scalable and capable of handling L1-induced speaker variability.
Did you see that? Exploring the role of vision in the development of consonant feature contrasts in children with cochlear implants
Authors: James Mahshie; Michael Larsen
Venue: Interspeech 2023 (main)

Abstract: This project aimed to explore the potential role of vision in speech contrast production and auditory perception development in children with cochlear implants (CWCI). Ten CWCI between 43 and 61 months of age, with at least 2 years of CI experience, served as participants.  Employing an auditory imitation task, children's ability to auditorily perceive contrasts that are more or less visible was examined both at baseline and one year after the initial assessment. The children's ability to produce these contrasts was also examined through a picture-naming task. The CWCI tended to produce features in both visibility conditions with greater accuracy than they perceived, both at baseline and at 1 year. Production and perception accuracy increased after one year of CI usage, with the mean perceptual gain for the more visible contrasts exceeding that of the less visible contrasts. The implications of the role of vision in contrast development are discussed.
Automatic assessments of dysarthric speech: the usability of acoustic-phonetic features
Authors: Loes van Bemmel; Chiara Pesenti; Xue Wei; Helmer Strik
Venue: Interspeech 2023 (main)

Abstract: Individuals with dysarthria suffer from difficulties in speech production and consequent reductions in speech intelligibility, which is an important concept for diagnosing and assessing effectiveness of speech therapy. In the current study, we investigate which acoustic-phonetic features are most relevant and important in automatically assessing intelligibility and in classifying speech as healthy or dysarthric. After feature selection, we applied a stepwise linear regression to predict intelligibility ratings and a Linear Discriminant Analysis to classify healthy and dysarthric speech. We observed a very strong correlation between actual and predicted intelligibility ratings in the regression analysis. We also observed a high classification accuracy of 98.06% by using 17 features and a comparable, high accuracy of 96.11% with only two features. These results indicate the usefulness of the acoustic-phonetic features in automatic assessments of dysarthric speech
Classification of Multi-class Vowels and Fricatives From Patients Having Amyotrophic Lateral Sclerosis with Varied Levels of Dysarthria Severity
Authors: Chowdam Venkata Thirumala Kumar; Tanuka Bhattacharjee; Yamini Belur; Atchayaram Nalini; Ravi Yadav; Prasanta Kumar Ghosh
Venue: Interspeech 2023 (main)

Abstract: Dysarthria due to Amyotrophic Lateral Sclerosis (ALS) progressively distorts the acoustic space affecting the discriminability of different vowels and fricatives. However, the extent to which this happens with increasing severity is not thoroughly investigated. In this work, we perform automatic 4-class vowel (/a/, /i/, /o/, /u/) and 3-class fricative (/s/, /sh/, /f/) classification at varied severity levels and compare the performances with those from manual classification (through listening tests). Experiments with speech data from 119 ALS and 40 healthy subjects suggest that the manual and automatic classification accuracies reduce with an increase in dysarthria severity reaching 59.22% and 61.67% for vowels and 41.78% and 38.00% for fricatives, respectively, at the most severe cases. While manual classification is better than automatic one for all severity levels except the highest severity case for vowels, the difference between the two gradually reduces with an increase in severity.
Parameter-efficient Dysarthric Speech Recognition Using Adapter Fusion and Householder Transformation
Authors: Jinzi Qi; Hugo Van hamme
Venue: Interspeech 2023 (main)

Abstract: In dysarthric speech recognition, data scarcity and the vast diversity between dysarthric speakers pose significant challenges. While finetuning has been a popular solution, it can lead to overfitting and low parameter efficiency. Adapter modules offer a better solution, with their small size and easy applicability. Additionally, Adapter Fusion can facilitate knowledge transfer from multiple learned adapters, but may employ more parameters. In this work, we apply Adapter Fusion for target speaker adaptation and speech recognition, achieving acceptable accuracy with significantly fewer speaker-specific trainable parameters than classical finetuning methods. We further improve the parameter efficiency of the fusion layer by reducing the size of query and key layers and using Householder transformation to reparameterize the value linear layer. Our proposed fusion layer achieves comparable recognition results to the original method with only one third of the parameters.
Few-shot Dysarthric Speech Recognition with Text-to-Speech Data Augmentation
Authors: Enno Hermann; Mathew Magimai.-Doss
Venue: Interspeech 2023 (main)

Abstract: Speakers with dysarthria could particularly benefit from assistive speech technology, but are underserved by current automatic speech recognition (ASR) systems. The differences of dysarthric speech pose challenges, while recording large amounts of training data can be exhausting for patients. In this paper, we synthesise dysarthric speech with a FastSpeech 2-based multi-speaker text-to-speech (TTS) system for ASR data augmentation. We evaluate its few-shot capability by generating dysarthric speech with as few as 5 words from an unseen target speaker and then using it to train speaker-dependent ASR systems. The results indicated that, while the TTS output is not yet of sufficient quality, this could allow easy development of personalised acoustic models for new dysarthric speakers and domains in the future.