GenAI4ELab/papercli-papers huggingface dataset:
Time-synchronous one-pass Beam Search for Parallel Online and Offline Transducers with Dynamic Block Training
Authors: Yui Sudo; Shakeel Muhammad; Yifan Peng; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: End-to-end automatic speech recognition (ASR) has become an increasingly popular area of research, with two main models being online and offline ASR. Online models aim to provide real-time transcription with minimal latency, whereas offline models wait until the end of the speech utterance before generating a transcription. In this work, we explore three techniques to maximize the performance of each model by 1) proposing a joint parallel online and offline architecture for transducers; 2) introducing dynamic block (DB) training, which allows flexible block size selection and improves the robustness for the offline mode; and, 3) proposing a novel time-synchronous one-pass beam search using the online and offline decoders to further improve the performance of the offline mode. Experimental results show that the proposed method consistently improves the character/word error rates on the CSJ and LibriSpeech datasets.
Prefix Search Decoding for RNN Transducers
Authors: Kiran Praveen; Advait Vinay Dhopeshwarkar; Abhishek Pandey; Balaji Radhakrishnan
Venue: Interspeech 2023 (main)

Abstract: Automatic Speech Recognition (ASR) has seen a surge in popularity for Recurrent Neural Network Transducers (RNN-T) in recent years and shows much promise. RNN-Ts were introduced as an extension of Connectionist Temporal Classification (CTC) models. While CTC models have prefix search as the widely used decoding strategy, it appears to have been overlooked in favour of other decoding strategies such as time-synchronous decoding (TSD) and alignment-synchronous decoding. In this work, we introduce prefix search decoding, looking at all prefixes in the decode lattice to score a candidate. We show that our technique aligns more closely to the training objective compared to the existing strategies. We compare our technique with the originally proposed TSD, using Librispeech and AMI-IHM datasets. We find that while prefix search is closer to the training objective, with larger datasets the performance improves significantly, while with lower size datasets the performance degrades.
WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
Authors: Max Bain; Jaesung Huh; Tengda Han; Andrew Zisserman
Venue: Interspeech 2023 (main)

Abstract: Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, the predicted timestamps corresponding to each utterance are prone to inaccuracies, and word-level timestamps are not available out-of-the-box. Further, their application to long audio via buffered transcription prohibits batched inference due to their sequential nature. To overcome the aforementioned challenges, we present WhisperX, a time-accurate speech recognition system with word-level timestamps utilising voice activity detection and forced phoneme alignment. In doing so, we demonstrate state-of-the-art performance on long-form transcription and word segmentation benchmarks. Additionally, we show that pre-segmenting audio with our proposed VAD Cut & Merge strategy improves transcription quality and enables a twelve-fold transcription speedup via batched inference.
Implementing Contextual Biasing in GPU Decoder for Online ASR
Authors: Iuliia Nigmatulina; Srikanth Madikeri; EsaÃº Villatoro-Tello; Petr Motlicek; Juan Zuluaga-Gomez; Karthik Pandia; Aravind Ganapathiraju
Venue: Interspeech 2023 (main)

Abstract: GPU decoding significantly accelerates the output of ASR predictions. While GPUs are already being used for online ASR decoding, post-processing and rescoring on GPUs have not been properly investigated yet. Rescoring with available contextual information can considerably improve ASR predictions. Previous studies have proven the viability of lattice rescoring in decoding and biasing language model (LM) weights in offline and online CPU scenarios. In real-time GPU decoding, partial recognition hypotheses are produced without lattice generation, which makes the implementation of biasing more complex. The paper proposes and describes an approach to integrate contextual biasing in real-time GPU decoding while exploiting the standard Kaldi GPU decoder. Besides the biasing of partial ASR predictions, our approach also permits dynamic context switching allowing a flexible rescoring per each speech segment directly on GPU. The code is publicly released and tested with open-sourced test sets.
MF-PAM: Accurate Pitch Estimation through Periodicity Analysis and Multi-level Feature Fusion
Authors: Woo-Jin Chung; Doyeon Kim; Soo-Whan Chung; Hong-Goo Kang
Venue: Interspeech 2023 (main)

Abstract: We introduce Multi-level feature Fusion-based Periodicity Analysis Model (MF-PAM), a novel deep learning-based pitch estimation model that accurately estimates pitch trajectory in noisy and reverberant acoustic environments. Our model leverages the periodic characteristics of audio signals and involves two key steps: extracting pitch periodicity using periodic non-periodic convolution (PNP-Conv) blocks and estimating pitch by aggregating multi-level features using a modified bi-directional feature pyramid network (BiFPN). We evaluate our model on speech and music datasets and achieve superior pitch estimation performance compared to state-of-the-art baselines while using fewer model parameters. Our model achieves 99.20 % accuracy in pitch estimation on a clean musical dataset. Overall, our proposed model provides a promising solution for accurate pitch estimation in challenging acoustic environments and has potential applications in audio signal processing.
Enhancing Speech Articulation Analysis Using A Geometric Transformation of the X-ray Microbeam Dataset
Authors: Ahmed Adel Attia; Mark Tiede; Carol Espy-Wilson
Venue: Interspeech 2023 (main)

Abstract: Accurate analysis of speech articulation is crucial for speech analysis. However, X-Y coordinates of articulators strongly depend on the anatomy of the speakers and variability of pellet placements, and existing methods for mapping anatomical landmarks in the X-ray Microbeam Dataset (XRMB) fail to capture the entire anatomy of the vocal tract. In this paper, we propose a new geometric transformation that improves the accuracy of these measurements. Our transformation maps anatomical landmarks' X-Y coordinates along the midsagittal plane onto six relative measures: Lip Aperture (LA), Lip Protrusion (LP), Tongue Body Constriction Location (TBCL), Degree (TBCD), Tongue Tip Constriction Location (TTCL), and Degree (TTCD). Our novel contribution is the extension of the palate trace towards the inferred anterior pharyngeal line, which improves measurements of tongue body constriction.
Matching Acoustic and Perceptual Measures of Phonation Assessment in Disordered Speech - A Case Study
Authors: Melanie Jouaiti; Pippa Kirby; Ravi Vaidyanathan
Venue: Interspeech 2023 (main)

Abstract: Speech/voice disorders are common in People Living with Dementia (PLwD). Fluctuations in speech quality can serve as biomarkers of cognitive deterioration but there is a gap in automated assessment of speech collected in unstructured environs. Our organisation has deployed Alexa in the households of 14 PLwD to track self-reported mental and physical state as well as use of language. In this work, we present a case study analysing highly variable speech over time, providing potential insights into cognitive changes. Alexa data gathered from the participant was manually annotated with speech assessment labels. Those labels are matched to openSMILE features by performing a feature importance analysis to isolate critical features that contribute to the perceptual ratings. We can assess phonation with a F1-score of 0.55, breathiness: 0.71, roughness: 0.60, asthenia: 0.65, strain: 0.74. This work is a first step towards automatic speech assessment to monitor cognitive impairment over time.
Improved Contextualized Speech Representations for Tonal Analysis
Authors: Jiahong Yuan; Xingyu Cai; Kenneth Church
Venue: Interspeech 2023 (main)

Abstract: We propose fine-tuning wav2vec2.0 with a cross-entropy loss to classify tones in an utterance on a frame-by-frame basis. Our study demonstrates that this approach not only improves tone classification accuracy but also generates frame-level representations suitable for tonal analysis. By using these representations, we established that the third-tone-sandhi-rising tone in Mandarin speech differs from the lexical rising tone, and the third tone that doesn't undergo sandhi differs from the third tone that's not in a sandhi context. Our findings suggest that third-tone sandhi in Mandarin Chinese involves a continuous shift from Tone3 to Tone2, rather than a categorical change.
A Study on the Importance of Formant Transitions for Stop-Consonant Classification in VCV Sequence
Authors: Siddarth Chandrasekar; Arvind Ramesh; Tilak Purohit; Prasanta Kumar Ghosh
Venue: Interspeech 2023 (main)

Abstract: This study analyzes formant transitions in six English stop-consonants in vowel-consonant-vowel (VCV) sequences. We investigate whether natural speech preserves formant patterns, and if not, how it affects stop-consonant perception and automatic classification. We specifically ask three questions: 1) To what extent these formant transition patterns are preserved in naturally produced VCV sequences? 2) If not preserved, does it have any effect on the perception of the stop-consonant? 3) How does the classification of stop-consonants by automatic classifiers change when formant transition patterns are not preserved? We found that 33.56% of the corpus deviate from the formant transition pattern. The perception test reveals an Unweighted Average Recall (UAR) of 91.97% in identifying the stop-consonants in the VCV sequences when the pattern is not preserved compared to 93.54% when it is preserved. The best UAR from an automatic classifier is 68.35% and 77.5% in these two cases, respectively.
FusedF0: Improving DNN-based F0 Estimation by Fusion of Summary-Correlograms and Raw Waveform Representations of Speech Signals
Authors: Eray Eren; Lee Ngee Tan; Abeer Alwan
Venue: Interspeech 2023 (main)

Abstract: DSP-based F0 estimation algorithms, such as multi-band summary-correlogram (MBSC), are robust to noisy speech. Recent studies show that mapping from raw waveform segments into F0 estimates by DNNs can outperform DSP-based methods in F0 estimation. However, generalization and noise robustness of DNNs have not been fully addressed previously. We propose a hybrid DSP and DNN based approach to F0 estimation. Key contributions include: (a) a modified version of MBSC that is substantially faster than the original algorithm while maintaining the accuracy of F0 estimates; (b) a method for fusing DSP features with raw waveform representations using a DNN architecture to obtain noise-robust F0 estimation; (c) demonstrating that auxiliary DSP features improve generalization with a relatively small number of DNN parameters. On the PTDB-TUG database, the proposed algorithm outperforms the MBSC and CREPE DNN baselines (including optimized versions) for clean and noisy speech at 20, 10, and 0 dB SNR.
Improving Joint Speech and Emotion Recognition Using Global Style Tokens
Authors: Jehyun Kyung; Ju-Seok Seong; Jeong-Hwan Choi; Ye-Rin Jeoung; Joon-Hyuk Chang
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) and speech emotion recognition (SER) are closely related in that the acoustic features of speech, such as pitch, tone, and intensity, can vary according to the speaker's emotional state. Our study focuses on a joint ASR and SER task, in which an emotion token is tagged and recognized along with the text. To further improve the joint recognition performance, we propose a novel training method that adopts the global style tokens (GSTs). The style embedding is extracted from the GSTs module to enhance the joint ASR and SER model to capture emotional information from speech. Specifically, a conformer-based joint ASR and SER model pre-trained on a large-scale dataset is jointly fine-tuned with style embedding to improve both ASR and SER. The experimental results on the IEMOCAP dataset showed that the proposed model achieves a word error rate of 15.8% and four emotion classification weighted and unweighted accuracy of 75.1% and 76.3%, respectively.
Speech Emotion Recognition by Estimating Emotional Label Sequences with Phoneme Class Attribute
Authors: Ryotaro Nagase; Takahiro Fukumori; Yoichi Yamashita
Venue: Interspeech 2023 (main)

Abstract: In recent years, much research has been into speech emotion recognition (SER) using deep learning to predict emotions conveyed by speech. We studied the method that detected the emotion for the whole utterance using the frame-based SER, which estimates emotions in each frame rather than in a whole utterance. One of the problems with this method is that the emotional label sequence, which is used in training the frame-based SER, does not sufficiently consider phonemic characteristics. To solve this problem, we propose new methods of recognizing the emotion for the whole utterance using frame-based SER that considers the phoneme class attribute such as vowels, voiced consonants, unvoiced consonants, and other symbols in training. As a result, we found that the proposed methods significantly improve the performance of the result for the whole utterance compared to conventional methods.
Unsupervised Transfer Components Learning for Cross-Domain Speech Emotion Recognition
Authors: Shenjie Jiang; Peng Song; Shaokai Li; Keke Zhao; Wenming Zheng
Venue: Interspeech 2023 (main)

Abstract: Cross-domain speech emotion recognition (SER), which utilizes the source domain to recognize the emotions in the target domain, has received significant attention in recent years. In this paper, we propose a novel unsupervised transfer learning method named unsupervised transfer components learning (UTCL) for cross-domain SER. Specifically, we first learn a common projection for the cross-domain data, in which a PCA-like strategy is conducted for the source and target domains separately. Meanwhile, we design a simple strategy to ensure all cross-domain samples share similar manifold structures so that the learned common projection can preserve more transfer components. Furthermore, a novel adaptive structured graph strategy is designed to further narrow the gap between the cross-domain samples.  Comprehensive experimental results on several benchmark datasets demonstrate that our method can achieve better performance in comparison with several state-of-the-art methods.
Dual Memory Fusion for Multimodal Speech Emotion Recognition
Authors: Darshana Prisayad; Tharindu Fernando; Sridha Sridharan; Simon Denman; Clinton Fookes
Venue: Interspeech 2023 (main)

Abstract: Deep learning has been widely used in multi-modal Speech Emotion Recognition (SER) to learn sentiment-related features by aggregating representations from multiple modes. However, most SOTA methods use attentive fusion or late fusion of data which ignores the possibility of long-term dependencies among data. In this study, we propose a transformer-based SER architecture that fuses modality representations through explicit memory modules, where the information from current inputs is integrated with historical information allowing the model to understand the relative importance of modes over time. We have used Wav2Vec2 and BERT models to extract audio and text features which are then fused together by aggregating features from individual modes with information stored in memory, followed by downstream classification. Following state-of-the-art methods, we evaluate our proposed method on the IEMOCAP dataset and results indicate that memory-based fusion can achieve substantial improvements.
Hybrid Dataset for Speech Emotion Recognition in Russian Language
Authors: Vladimir Kondratenko; Nikolay Karpov; Artem Sokolov; Nikita Savushkin; Oleg Kutuzov; Fyodor Minkin
Venue: Interspeech 2023 (main)

Abstract: We present a new data set for speech emotion recognition (SER) tasks called Dusha. The corpus contains approximately 350 hours of data, more than 300 000 audio recordings of Russian speech, and their transcripts. Therefore it is the biggest open bi-modal data collection with an open license for SER tasks nowadays. This data set is the first speech emotion corpus in Russian, including both crowd-sourced acted and real-life emotions from podcasts, with multiple speakers and scalable data set size. Acted subset has a more balanced class distribution than the unbalanced real-life part consisting of audio podcasts. So the first one is suitable for model pre-training, and the second is elaborated for fine-tuning purposes, model approbation, and validation. This paper describes in detail our collecting procedure, pre-processing routine, annotation, and experiment with a baseline model to demonstrate some actual metrics which could be obtained with the Dusha data set.
Speech Emotion Recognition using Decomposed Speech via Multi-task Learning
Authors: Jia-Hao Hsu; Chung-Hsien Wu; Yu-Hung Wei
Venue: Interspeech 2023 (main)

Abstract: In speech emotion recognition, most recent studies used powerful models to obtain robust features without considering the disentangled components, which contain diverse emotion-rich information helpful for speech emotion recognition. In this study, an autoencoder is used as the speech decomposition model to obtain the disentangled components, including content, timbre, pitch, and rhythm features, which are regarded as emotion-rich features, for speech emotion recognition. The mechanism of multi-task training is then used to train the tasks of speech emotion recognition, speaker recognition, speech recognition, and spectral reconstruction at the same time, while exploiting commonalities and differences across tasks. The model proposed in this study achieved an accuracy of 77.50% on the four-classes emotion recognition task of IEMOCAP. Experiments showed that the proposed methods can effectively improve speech emotion recognition performance, outperforming the SOTA approach.
Prospective Validation of Motor-Based Intervention with Automated Mispronunciation Detection of Rhotics in Residual Speech Sound Disorders
Authors: Nina R Benway; Jonathan L Preston
Venue: Interspeech 2023 (main)

Abstract: Because lab accuracy of clinical speech technology systems may be overoptimistic, clinical validation is vital to demonstrate system reproducibility-in this case, the ability of the PERCEPT-R Classifier to predict clinician judgment of American English /É¹/ during ChainingAI motor-based speech sound disorder intervention. All five participants experienced statistically-significant improvement in untreated words following 10 sessions of combined human-ChainingAI treatment. These gains, despite a wide range of PERCEPT-human and human-human (F1-score) agreement, raise questions about best measuring classification performance for clinical speech that may be perceptually ambiguous.
Classifying Rhoticity of /É¹/ in Speech Sound Disorder using Age-and-Sex Normalized Formants
Authors: Nina R Benway; Jonathan L Preston; Asif Salekin; Yi Xiao; Harshit Sharma; Tara McAllister
Venue: Interspeech 2023 (main)

Abstract: Mispronunciation detection tools could increase treatment access for speech sound disorders impacting, e.g., /É¹/. We show age-and-sex normalized formant estimation outperforms cepstral representation for detection of fully rhotic vs. derhotic /É¹/ in the PERCEPT-R Corpus. Gated recurrent neural networks trained on this feature set achieve a mean test participant-specific F1-score =.81 (Ïx=.10, med = .83, n = 48), with post hoc modeling showing no significant effect of child age or sex.
Acoustic-to-Articulatory Speech Inversion Features for Mispronunciation Detection of /É¹/ in Child Speech Sound Disorders
Authors: Nina R Benway; Yashish M Siriwardena; Jonathan L Preston; Elaine Hitchcock; Tara McAllister; Carol Espy-Wilson
Venue: Interspeech 2023 (main)

Abstract: Acoustic-to-articulatory speech inversion could enhance automated clinical mispronunciation detection to provide detailed articulatory feedback unattainable by formant-based mispronunciation detection algorithms; however, it is unclear the extent to which a speech inversion system trained on adult speech performs in the context of (1) child and (2) clinical speech. In the absence of an articulatory dataset in children with rhotic speech sound disorders, we show that classifiers trained on tract variables from acoustic-to-articulatory speech inversion meet or exceed the performance of state-of-the-art features when predicting clinician judgment of rhoticity.
Using Commercial ASR Solutions to Assess Reading Skills in Children: A Case Report
Authors: Timothy Piton; Enno Hermann; Angela Pasqualotto; Marjolaine Cohen; Mathew Magimai.-Doss; DaphnÃ© Bavelier
Venue: Interspeech 2023 (main)

Abstract: Reading is an acquired skill that is essential for integrating and participating in today's society. Yet, becoming literate can be particularly laborious for some children. Identifying reading difficulties early enough is the first, necessary step toward remediation. Here we investigate the opportunities and limitations of integrating commercial, off-the-shelf automatic speech recognition (ASR) services from IBM Watson to ease the administration and evaluation of children's reading assessment tests in French and Italian.
Exploiting Diversity of Automatic Transcripts from Distinct Speech Recognition Techniques for Childrenâs Speech
Authors: Christopher Gebauer; Lars Rumberg; Hanna Ehlert; Ulrike LÃ¼dtke; Joern Ostermann
Venue: Interspeech 2023 (main)

Abstract: The recent advances in automatic speech recognition (ASR) technologies using end-to-end machine learning do not transfer well to children's speech. One cause is the high pronunciation variability and frequent violations of grammatical or lexical rules, which impedes the successful usage of language models or powerful context-representations. Applying these methods affects the nature of the resulting transcript rather than improving the overall recognition performance. In this work we analyze the diversity of the transcripts from distinct ASR-systems for children's speech and exploit it by applying a common combination scheme. We consider systems with various degree of context: Greedily decoded and lexicon-constrained connectionist temporal classification-models, attention-based encoder decoders, and Wav2Vec 2.0, a powerful context-representation. By exploiting their diversity we achieve a relative improvement of 17.8 % on phone recognition compared to the best single system.
Uncertainty Estimation for Connectionist Temporal Classification Based Automatic Speech Recognition
Authors: Lars Rumberg; Christopher Gebauer; Hanna Ehlert; Maren Wallbaum; Ulrike LÃ¼dtke; Joern Ostermann
Venue: Interspeech 2023 (main)

Abstract: Predictive uncertainty estimation of deep neural networks is important when their outputs are used for high stakes decision making. We investigate token-level uncertainty of connectionist temporal classification (CTC) based automatic speech recognition models. We propose an approach, which considers that not all changes at frame-level lead to a change at token-level after CTC decoding. The approach shows promising performance for prediction of recognition errors on TIMIT, Mozilla Common Voice (MCV) and kidsTALC, a corpus of children's speech, using two different model architectures, while introducing only negligible computational overhead. Our approach identifies over 80% of a wav2vec2.0 model's errors on MCV by selecting 10% of the tokens. We further show, that the predictive uncertainty estimate relates to the uncertainty of a human annotator, by re-annotating 500 utterances of kidsTALC.