GenAI4ELab/papercli-papers huggingface dataset:
ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition
Authors: Yuanchao Li; Zeyu Zhao; OndÅej Klejch; Peter Bell; Catherine Lai
Venue: Interspeech 2023 (main)

Abstract: In Speech Emotion Recognition (SER), textual data is often used alongside audio signals to address their inherent variability. However, the reliance on human annotated text in most research hinders the development of practical SER systems. To overcome this challenge, we investigate how Automatic Speech Recognition (ASR) performs on emotional speech by analyzing the ASR performance on emotion corpora and examining the distribution of word errors and confidence scores in ASR transcripts to gain insight into how emotion affects ASR. We utilize four ASR systems, namely Kaldi ASR, wav2vec2, Conformer, and Whisper, and three corpora: IEMOCAP, MOSI, and MELD to ensure generalizability. Additionally, we conduct text-based SER on ASR transcripts with increasing word error rates to investigate how ASR affects SER. The objective of this study is to uncover the relationship and mutual impact of ASR and SER, in order to facilitate ASR adaptation to emotional speech and the use of SER in real world.
BASS: Block-wise Adaptation for Speech Summarization
Authors: Roshan Sharma; Siddhant Arora; Kenneth Zheng; Shinji Watanabe; Rita Singh; Bhiksha Raj
Venue: Interspeech 2023 (main)

Abstract: End-to-end speech summarization has been shown to improve performance over cascade baselines. However, such models are difficult to train on very large inputs (dozens of minutes or hours) owing to computing restrictions and are hence trained with truncated model inputs. Truncation leads to poorer models, and a solution to this problem rests in block-wise modeling, i.e., processing a portion of the input frames at a time. In this paper, we develop a method that allows one to train summarization models on very long sequences in an incremental manner. Speech summarization is realized as a streaming process, where hypothesis summaries are updated every block based on new acoustic information. We devise and test strategies to pass semantic context across the blocks. Experiments on the How2 dataset demonstrate that the proposed block-wise training method improves by 3 points absolute on ROUGE-L over a truncated input baseline.
Speaker Tracking using Graph Attention Networks with Varying Duration Utterances across Multi-Channel Naturalistic Data: Fearless Steps Apollo-11 Audio Corpus
Authors: Meena M. C. Shekar; John H. L. Hansen
Venue: Interspeech 2023 (main)

Abstract: Speaker tracking in spontaneous naturalistic data continues to be a major research challenge, especially for short turn-taking communications. The NASA Apollo-11 space mission brought astronauts to the moon and back, where team based voice communications were captured. Building robust speaker classification models for this corpus has significant challenges due to variability of speaker turns, imbalanced speaker classes, and time-varying background noise/distortions. This study proposes a novel approach for speaker classification and tracking, utilizing a graph attention network framework that builds upon pretrained speaker embeddings. The modelâs robustness is evaluated on a number of speakers (10-140), achieving classification accuracy of 90.78% for 10 speakers, and 79.86% for 140 speakers. Furthermore, a secondary investigation focused on tracking speakers-of-interest(SoI) during mission critical phases, essentially serves as a lasting tribute to the 'Heroes Behind the Heroes'.
Combining language corpora in a Japanese electromagnetic articulography database for acoustic-to-articulatory inversion
Authors: Tianfang Yan; Kikuo Maekawa; Yukiko Nota; Masayuki Hirata
Venue: Interspeech 2023 (main)

Abstract: This paper presents an electromagnetic articulography database of Japanese sentences. The database includes aligned acoustics and articulatory data from seven males and three females, with a total of five recorded hours. The database is now in preparation for public release to further research in areas of acoustic-to-articulatory inversion, brain-machine interface communication systems, artificial speech synthesis, and dialect recognition. Moreover, based on this database we established an acoustic-to-articulatory inversion system using a deep, bidirectional, long short-term memory recurrent neural network structure. The results showed that, for the Japanese language, adding English corpora to the training was not beneficial for this speaker-independent model.
A Dual Attention-based Modality-Collaborative Fusion Network for Emotion Recognition
Authors: Xiaoheng Zhang; Yang Li
Venue: Interspeech 2023 (main)

Abstract: Multi-modal emotion recognition (MER) is an emerging research field in human-computer interactions. However, previous studies have explored several fusion methods to deal with the asynchronism and the heterogeneity of multimodal data but they mostly neglect the importance of discriminative unimodal information resulting in the ignorance of independence of uni-modality. Furthermore, the complementarity among different fusion strategies is seldom taken in consideration. To address these limitations, we propose a modality-collaborative fusion network (MCFN) consisting of three main components: a dual attention-based intra-modal learning module which is devoted to build the initial embedding spaces, a modality-collaborative learning approach is to reconcile the emotional information across modalities, and a two-stage fusion strategy to integrate multimodal features which are improved by a mutual adjustment approach. The proposed framework outperforms the state-of-the-art methods in overall experiments on two well-known public datasets. Our model will be available at https://github.com/zxiaohen/ Speech-emotion-recognition-MCFN
Large Dataset Generation of Synchronized Music Audio and Lyrics at Scale using Teacher-Student Paradigm
Authors: Cristian Chivriga; Rinita Roy
Venue: Interspeech 2023 (main)

Abstract: Large models (e.g., GPT-3, CLIP, DALL-E) show remarkable few-shot and zero-shot capabilities when trained on hundreds of millions of samples. Despite this trend, no publicly available synchronized music audio and lyrics dataset of sufficient scale exists, nor does a reliable evaluation benchmark to assess a model's performance. To address this issue, we build and release MusicLyric, a large public dataset with over 320k audio sequences and lyrics pairs for a total duration of 1,200 hours based on a collection of over 32,000 songs. The generation process is based on the teacher-student paradigm where the student seeks to outclass the teacher with more data available using the newly generated pseudo-alignments. The method is efficient and straightforward with at least 3 iterations needed to create high-quality data that can be scaled to a hundred thousand samples. We make our dataset, toolkit, and pre-trained models open-source.
Enc-Dec RNN Acoustic Word Embeddings learned via Pairwise Prediction
Authors: Adhiraj Banerjee; Vipul Arora
Venue: Interspeech 2023 (main)

Abstract: Learning discriminative Acoustic Word Embeddings (AWEs) summarising variable length spoken word segments brings efficiency in speech retrieval tasks, notably, Query-by-Example (QbE) Speech or Spoken Term Detection (STD). In this paper, we add on to RNN based approaches for generating acoustic word embeddings. The model is trained in an encoder-decoder fashion on pairs of similar word segments by optimizing a pairwise self-supervised loss where the targets are generated offline via clustering. The pairs may be generated with word boundaries (weak supervision) or via augmentation of unlabelled word segments (no supervision). Experiments with word discrimination task on TIMIT and LibriSpeech show state of the art performance of the proposed approach outperforming popular RNN AWE approaches in both weakly supervised and unsupervised settings. The AWEs generated by our model generalise well to OOV words. On STD tasks performed on TIMIT, the proposed approach provides speed advantages.
Query Based Acoustic Summarization for Podcasts
Authors: Samantha Kotey; Rozenn Dahyot; Naomi Harte
Venue: Interspeech 2023 (main)

Abstract: Podcasts are a rich storytelling medium of long diverse conversations. Typically, listeners preview an episode through an audio clip, before deciding to consume the content. An automatic system that produces promotional clips, by supporting acoustic queries would greatly benefit podcasters. Previous text based methods do not use the acoustic signal directly or incorporate acoustic defined queries. Therefore, we propose a query based summarization approach, to produce audio clip summaries from podcast data. Leveraging unsupervised clustering methods, we apply our framework to the Spotify podcasts dataset. Audio signals are transformed into acoustic word embeddings, along with a pre-selected candidate query. We initiate the cluster centroids with the query vector and obtain the final snippets by computing a global and local similarity score. Additionally, we apply our framework to the AMI meeting dataset and demonstrate how audio can successfully be utilized to perform summarization.
Spot Keywords From Very Noisy and Mixed Speech
Authors: Ying Shi; Dong Wang; Lantian Li; Jiqing Han; Shi Yin
Venue: Interspeech 2023 (main)

Abstract: Most existing keyword spotting research focuses on conditions with slight or moderate noise. In this paper, we try to tackle a more challenging task: detecting keywords buried under strong interfering speech (10 times higher than the keyword in amplitude), and even worse, mixed with other keywords. We propose a novel Mix Training (MT) strategy that encourages the model to discover low-energy keywords from noisy and mixed speech. Experiments were conducted with a vanilla CNN and two EfficientNet (B0/B2) architectures. The results evaluated with the Google Speech Command dataset demonstrated that the proposed mix training approach is highly effective and outperforms standard data augmentation and mixup training.
Knowledge Distillation on Joint Task End-to-End Speech Translation
Authors: Khandokar Md. Nayem; Ran Xue; Ching-Yun Chang; Akshaya Vishnu Kudlu Shanbhogue
Venue: Interspeech 2023 (main)

Abstract: An End-to-End Speech Translation (E2E-ST) model takes input audio in one language and directly produces output text in another language. The model requires to learn both speech-to-text modality conversion and translation tasks, which demands a large architecture for effective learning of this joint task. Yet, to the best of our knowledge, we are the first to optimize compression of E2E-ST models. In this work, we explore knowledge distillation for a cross-modality joint-task E2E-ST system from 3 dimensions: 1) student architecture and weight initialization scheme, 2) importance of loss terms associated with different tasks and data modalities, 3) knowledge distillation training scheme customized for the multi-task/module model. Comparing with the full size model, our compressed model's encoder and decoder size are 50% smaller, while it retains 90% and > 95% performance on speech translation task and machine translation task respectively on MUST-C enâde testset.
Investigating Pre-trained Audio Encoders in the Low-Resource Condition
Authors: Hao Yang; Jinming Zhao; Gholamreza Haffari; Ehsan Shareghi
Venue: Interspeech 2023 (main)

Abstract: Pre-trained speech encoders have been central to pushing state-of-the-art results across various speech understanding and generation tasks. Nonetheless, the capabilities of these encoders in low-resource settings are yet to be thoroughly explored. To address this, we conduct a comprehensive set of experiments using a representative set of 3 state-of-the-art encoders (Wav2vec2, WavLM, Whisper) in the low-resource setting across 7 speech understanding and generation tasks. We provide various quantitative and qualitative analyses on task performance, convergence speed, and representational properties of the encoders. We observe a connection between the pre-training protocols of these encoders and the way in which they capture information in their internal layers. In particular, we observe the Whisper encoder exhibits the greatest low-resource capabilities on content-driven tasks in terms of performance and convergence speed.
Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target
Authors: Guan-Wei Wu; Guan-Ting Lin; Shang-Wen Li; Hung-yi Lee
Venue: Interspeech 2023 (main)

Abstract: Spoken Language Understanding (SLU) is a task that aims to extract semantic information from spoken utterances. Previous research has made progress in end-to-end SLU by using paired speech-text data, such as pre-trained Automatic Speech Recognition (ASR) models or paired text as intermediate targets. However, acquiring paired transcripts is expensive and impractical for unwritten languages. On the other hand, Textless SLU extracts semantic information from speech without utilizing paired transcripts. However, the absence of intermediate targets and training guidance for textless SLU often leads to suboptimal performance. In this work, inspired by the content-disentangled discrete units from self-supervised speech models, we proposed to use discrete units as intermediate guidance to improve textless SLU performance. Our method surpasses the baseline method on five SLU benchmark corpora. Additionally, we find that unit guidance facilitates few-shot learning and enhances noise robustness.
Debiased Automatic Speech Recognition for Dysarthric Speech via Sample Reweighting with Sample Affinity Test
Authors: Eungbeom Kim; Yunkee Chae; Jaeheon Sim; Kyogu Lee
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition systems based on deep learning are mainly trained under empirical risk minimization (ERM). Since ERM utilizes the averaged performance on the data samples regardless of a group such as healthy or dysarthric speakers, ASR systems are unaware of the performance disparities across the groups. This results in biased ASR systems whose performance differences among groups are severe. In this study, we aim to improve the ASR system in terms of group robustness for dysarthric speakers. To achieve our goal, we present a novel approach, sample reweighting with sample affinity test (Re-SAT). Re-SAT systematically measures the debiasing helpfulness of the given data sample and then mitigates the bias by debiasing helpfulness-based sample reweighting. Experimental results demonstrate that Re-SAT contributes to improved ASR performance on dysarthric speech without performance degradation on healthy speech.
Multimodal Locally Enhanced Transformer for Continuous Sign Language Recognition
Authors: Katerina Papadimitriou; Gerasimos Potamianos
Venue: Interspeech 2023 (main)

Abstract: In this paper, we propose a novel Transformer-based approach for continuous sign language recognition (CSLR) from videos, aiming to address the shortcomings of traditional Transformers in learning local semantic context of SL. Specifically, the proposed relies on two distinct components: (a) a window-based RNN module to capture local temporal context and (b) a Transformer encoder, enhanced with local modeling via Gaussian bias and relative position information, as well as with global structure modeling through multi-head attention. To further improve model performance, we design a multimodal framework that applies the proposed to both appearance and motion signing streams, aligning their posteriors through a guiding CTC technique. Further, we achieve visual feature and gloss sequence alignment by incorporating a knowledge distillation loss. Experimental evaluation on two popular German CSLR datasets, demonstrates the superiority of our model.
Towards Supporting an Early Diagnosis of Multiple Sclerosis using Vocal Features
Authors: Monica Gonzalez-Machorro; Pascal Hecker; Uwe D. Reichel; Helly N. Hammer; Robert Hoepner; Lisa Pedrotti; Alisha Zmutt; Hesam Sagha; Johan van Beek; Florian Eyben; Dagmar M. Schuller; BjÃ¶rn W. Schuller; Bert Arnrich
Venue: Interspeech 2023 (main)

Abstract: Multiple sclerosis (MS) is a neuroinflammatory disease that affects millions of people worldwide. Since dysarthria is prominent in people with MS (pwMS), this paper aims to identify acoustic features that differ between people with MS and healthy controls (HC). Additionally, we develop automatic classification methods to distinguish between pwMS and HC. In this work, we present a new dataset of a German-speaking cohort which contains 39 patients with low disability of relapsing MS and 16 HC. Findings suggest that certain interpretable speech features could be useful in diagnosing MS, and that machine learning methods could potentially support fast and unobtrusive screening in clinical practice. The study emphasises the importance of analysing free speech compared to read speech.
Whisper Features for Dysarthric Severity-Level Classification
Authors: Siddharth Rathod; Monil Charola; Akshat Vora; Yash Jogi; Hemant A. Patil
Venue: Interspeech 2023 (main)

Abstract: Dysarthria is a speech disorder caused by improper coordination between the brain and the muscles that produce intelligible speech. Accurately diagnosing the severity of dysarthria is critical for determining the appropriate treatment and directing speech to suitable Automatic Speech Recognition systems. Recently, various methods have been employed to investigate the classification of dysarthria severity-levels using advanced features, including STFT and MFCC. This study proposes utilizing Web-scale Supervised Pretraining for Speech Recognition (WSPSR), also known as Whisper, encoder module for dysarthric severity-level classification using transfer learning approach. Whisper model is an advanced machine learning model used for speech recognition, which is trained on a large scale of 680,000 hours of labeled audio data. The proposed approach demonstrated a high accuracy rate of 98.02%, surpassing the accuracies achieved by MFCC (95.2%) and LFCC (96.05%).
A New Benchmark of Aphasia Speech Recognition and Detection Based on E-Branchformer and Multi-task Learning
Authors: Jiyang Tang; William Chen; Xuankai Chang; Shinji Watanabe; Brian MacWhinney
Venue: Interspeech 2023 (main)

Abstract: Aphasia is a language disorder that affects the speaking ability of millions of patients. This paper presents a new benchmark for Aphasia speech recognition and detection tasks using state-of-the-art speech recognition techniques with the AphsiaBank dataset. Specifically, we introduce two multi-task learning methods based on the CTC/Attention architecture to perform both tasks simultaneously. Our system achieves state-of-the-art speaker-level detection accuracy (97.3%), and a relative WER reduction of 11% for moderate Aphasia patients. In addition, we demonstrate the generalizability of our approach by applying it to another disordered speech database, the DementiaBank Pitt corpus. We will make our all-in-one recipes and pre-trained model publicly available to facilitate reproducibility. Our standardized data preprocessing pipeline and open-source recipes enable researchers to compare results directly, promoting progress in disordered speech processing.
Dysarthric Speech Recognition, Detection and Classification using Raw Phase and Magnitude Spectra
Authors: Zhengjun Yue; Erfan Loweimi; Zoran Cvetkovic
Venue: Interspeech 2023 (main)

Abstract: In this paper, we explore the effectiveness of deploying the raw phase and magnitude spectra for dysarthric speech recognition, detection and classification. In particular, we scrutinise the usefulness of various raw phase-based representations along with their combinations with the raw magnitude spectrum and filterbank features. We employed single and multi-stream architectures consisting of a cascade of convolutional, recurrent and fully-connected layers for acoustic modelling. Furthermore, we investigate various configurations and fusion schemes as well as their training dynamics. In addition, the accuracies of the raw phase and magnitude based systems in the detection and classification tasks are studied and discussed. We report the performance on the UASpeech and TORGO dysarthric speech databases and for different severity levels. Our best system achieved WERs of 31.2% and 9.1% for dysarthric and typical speech on TORGO and 30.2% on UASpeech, respectively.
A Stutter Seldom Comes Alone â Cross-Corpus Stuttering Detection as a Multi-label Problem
Authors: Sebastian P. Bayerl; Dominik Wagner; Ilja Baumann; Florian HÃ¶nig; Tobias Bocklet; Elmar NÃ¶th; Korbinian Riedhammer
Venue: Interspeech 2023 (main)

Abstract: Most stuttering detection and classification research has viewed stuttering as a multi-class classification problem or a binary detection task for each dysfluency type; however, this does not match the nature of stuttering, in which one dysfluency seldom comes alone but rather co-occurs with others. This paper explores multi-language and cross-corpus end-to-end stuttering detection as a multi-label problem using a modified wav2vec 2.0 system with an attention-based classification head and multi-task learning. We evaluate the method using combinations of three datasets containing English and German stuttered speech, one containing speech modified by fluency shaping. The experimental results and an error analysis show that multi-label stuttering detection systems trained on cross-corpus and multi-language data achieve competitive results but performance on samples with multiple labels stays below overall detection results.
Transfer Learning to Aid Dysarthria Severity Classification for Patients with Amyotrophic Lateral Sclerosis
Authors: Tanuka Bhattacharjee; Anjali Jayakumar; Yamini Belur; Atchayaram Nalini; Ravi Yadav; Prasanta Kumar Ghosh
Venue: Interspeech 2023 (main)

Abstract: A major challenge involved in automatic dysarthria severity classification for patients with Amyotrophic Lateral Sclerosis (ALS) is the difficulty to build a speech corpus which is large enough to train accurate and generalizable classifiers. To overcome this constraint, we employ transfer learning approaches, specifically, fine-tuning from an auxiliary task and multi-task learning. Input feature reconstruction and gender classification, on the same ALS speech dataset or other healthy speech corpora, are explored as the auxiliary tasks. We use temporal statistics of mel-frequency cepstral coefficients as the features and dense neural networks for performing the primary and auxiliary tasks. Experiments suggest that transfer learning aids severity classification with up to 11.03% absolute increase in the average classification accuracy as compared to direct single task learning. The improvement is attributed mainly to better classification of the mild class than severe/normal classes.
DuTa-VC: A Duration-aware Typical-to-atypical Voice Conversion Approach with Diffusion Probabilistic Model
Authors: Helin Wang; Thomas Thebaud; JesÃºs Villalba; Myra Sydnor; Becky Lammers; Najim Dehak; Laureano Moro-Velazquez
Venue: Interspeech 2023 (main)

Abstract: We present a novel typical-to-atypical voice conversion approach (DuTa-VC), which (i) can be trained with nonparallel data (ii) first introduces diffusion probabilistic model (iii) preserves the target speaker identity (iv) is aware of the phoneme duration of the target speaker. DuTa-VC consists of three parts: an encoder transforms the source mel-spectrogram into a duration-modified speaker-independent mel-spectrogram, a decoder performs the reverse diffusion to generate the target mel-spectrogram, and a vocoder is applied to reconstruct the waveform. Objective evaluations conducted on the UASpeech show that DuTa-VC is able to capture severity characteristics of dysarthric speech, reserves speaker identity, and significantly improves dysarthric speech recognition as a data augmentation. Subjective evaluations by two expert speech pathologists validate that DuTa-VC can preserve the severity and type of dysarthria of the target speakers in the synthesized speech.