GenAI4ELab/papercli-papers huggingface dataset:
CNVVE: Dataset and Benchmark for Classifying Non-verbal Voice
Authors: Ramin Hedeshy; Raphael Menges; Steffen Staab
Venue: Interspeech 2023 (main)

Abstract: Non-verbal voice expressions (NVVEs) have been adopted as a means of human-computer interaction in research studies. However, exploring non-verbal voice-based interactions has been constrained by the limited availability of suitable training data and computational methods for classifying such expressions, leading to a focus on simple binary inputs. We address this issue with a new dataset containing 950 audio samples comprising 6 classes of voice expressions. The data were collected from 42 speakers who donated voice recordings. The classifier was trained on the data using features derived from mel-spectrograms. Furthermore, we studied the effectiveness of data augmentation and improved over the baseline model accuracy significantly with a test accuracy of 96.6% in a 5-fold cross-validation. We have made CNVVE publicly accessible in the hope that it will serve as a benchmark for future research.
Arabic Dysarthric Speech Recognition Using Adversarial and Signal-Based Augmentation
Authors: Massa Baali; Ibrahim Almakky; Shady Shehata; Fakhri Karray
Venue: Interspeech 2023 (main)

Abstract: Despite major advancements in Automatic Speech Recognition (ASR), the state-of-the-art ASR systems struggle to deal with impaired speech even with high-resource languages. In Arabic, this challenge gets amplified, with added complexities in collecting data from dysarthric speakers. In this paper, we aim to improve the performance of Arabic dysarthric automatic speech recognition through a multi-stage augmentation approach. To this effect, we first propose a signal-based approach to generate dysarthric Arabic speech from healthy Arabic speech by modifying its speed and tempo. We also propose a second stage Parallel Wave Generative (PWG) adversarial model that is trained on an English dysarthric dataset to capture language-independant dysarthric speech patterns and further augment the signal-adjusted speech samples. Furthermore, we propose a fine-tuning and text-correction strategies for Arabic Conformer at different dysarthric speech severity levels. Our fine-tuned Conformer achieved 18% Word Error Rate (WER) and 17.2% Character Error Rate (CER) on synthetically generated dysarthric speech from the Arabic common voice speech dataset. This shows significant WER improvement of 81.8% compared to the baseline model trained solely on healthy data. We perform further validation on real English dysarthric speech showing a WER improvement of 124% compared to the baseline trained only on healthy English LJSpeech dataset.
Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling
Authors: Theodoros Kouzelis; Georgios Paraskevopoulos; Athanasios Katsamanis; Vassilis Katsouros
Venue: Interspeech 2023 (main)

Abstract: The study of speech disorders can benefit greatly from time-aligned data. However, audio-text mismatches in disfluent speech cause rapid performance degradation for modern speech aligners, hindering the use of automatic approaches. In this work, we propose a simple and effective modification of alignment graph construction of CTC-based models using Weighted Finite State Transducers. The proposed weakly-supervised approach alleviates the need for verbatim transcription of speech disfluencies for forced alignment. During the graph construction, we allow the modeling of common speech disfluencies, i.e. repetitions and omissions. Further, we show that by assessing the degree of audio-text mismatch through the use of Oracle Error Rate, our method can be effectively used in the wild. Our evaluation on a corrupted version of the TIMIT test set and the UCLASS dataset shows significant improvements, particularly for recall, achieving a 23-25% relative improvement over our baselines.
Glottal source analysis of voice deficits in basal ganglia dysfunction: evidence from de novo Parkinson's disease and Huntington's disease
Authors: Michal NovotnÃ½; Tereza TykalovÃ¡; Michal Å imek; TomÃ¡Å¡ Kouba; Jan Rusz
Venue: Interspeech 2023 (main)

Abstract: Dysphonia is a common speech disruption in people with Parkinson's (PD) and Huntington's (HD). Though the glottal source analysis (GSS) yielded promising results in PD, no study analyzed utility of the GSS in HD. In addition, the potential GSS sex-dependency remains unknown. This study examines sustained vowel phonations provided by 40 PD, 40 HD and 40 age- and sex-matched healthy participants using six GSS features including normalized amplitude quotient, quasi-open quotient, magnitude difference of first two spectral peaks, harmonic richness factor, maximum dispersion quotient (MDQ), and peak slope. Our results showed significant differences in HD men and women compared to the healthy counterpart, suggesting breathiness (p<0.01), tension (p<0.001), and decreased timbre (p<0.01) in HD. Reported sex-related differences highlighted the sensitivity of the GSS towards the speaker's sex. The correlation analysis revealed significant relationship between disease severity and MDQ in HD men.
An Analysis of Glottal Features of Chronic Kidney Disease Speech and Its Application to CKD Detection
Authors: Jihyun Mun; Sunhee Kim; Myeong Ju Kim; Jiwon Ryu; Sejoong Kim; Minhwa Chung
Venue: Interspeech 2023 (main)

Abstract: Chronic kidney disease (CKD) causes a continuous decline in kidney function and structural damage to the kidneys. The speech characteristics of CKD speakers will be different from those of non-CKD speakers because the typical characteristics of CKD, which are impairment of respiratory and laryngeal muscles, can affect respiration, the primary source of speech. In this paper, we identify the glottal characteristics of CKD speech and then investigate whether CKD can be automatically detected using the glottal features. Statistical analysis shows significant differences between groups in glottal source features, representing the breathy characteristic of CKD speech. Through the classification experiment, we compare the performance of solely using voice quality features (baseline) against additional glottal and spectral features. When glottal source features and voice quality features are used together, an F1-score of 0.88 with a 76% relative increase compared to the baseline is obtained.
Weakly supervised glottis segmentation in high-speed videoendoscopy using bounding box labels
Authors: Varun Belagali; Achuth Rao; Prasanta Kumar Ghosh
Venue: Interspeech 2023 (main)

Abstract: Analysis of vocal fold vibration in high-speed videoendoscopy can aid in the assessment of voice disorders. Glottis segmentation is a preliminary step of this analysis. Previous deep learning approaches have focused on fully supervised learning methods for glottis segmentation which require pixel-level annotation. Collection of pixel-level annotated data is time consuming and tedious. To overcome this challenge, in this work, we explore the use of bounding box labels for weakly supervised glottis segmentation. As such, bounding box labels are relatively easier to annotate. The proposed method uses multiple instance learning to leverage bounding box labels in the form of bag labels. The method outperforms the baseline method (trained with bounding box as mask) by 0.20 in terms of dice score, and matches the performance of fully supervised version after fine-tuning.
An Efficient and Noise-Robust Audiovisual Encoder for Audiovisual Speech Recognition
Authors: Zhengyang Li; Chenwei Liang; Timo Lohrenz; Marvin Sach; BjÃ¶rn MÃ¶ller; Tim Fingscheidt
Venue: Interspeech 2023 (main)

Abstract: Boosted by self-supervised learning (SSL) on large amounts of unlabeled data, computationally demanding transformer-based audiovisual ASR (AV-ASR) achieves state-of-the-art performance. In this work, we are the first to propose teacher-student model distillation for an efficient and noise-robust AV encoder for AV-ASR. First, we compare two options for the teacher, a non-task-specific and a task-specific one. Second, we investigate the design and the components in the student neural network. Third, we explore loss function choices during distillation. By distillation with a simplified loss function, the final efficient conformer-based student has 69% fewer parameters and 23% less computational power than the teacher, but excels the baseline student with a WER of 4.6% (11.4%) in clean condition, and with 20.2% (35.7%) in 0dB babble noise. On average over noise types in 0dB SNR, our proposed student even achieves more than 50% relative WER reduction compared to the baseline student.
A Novel Self-training Approach for Low-resource Speech Recognition
Authors: Satwinder Singh; Feng Hou; Ruili Wang
Venue: Interspeech 2023 (main)

Abstract: In this paper, we propose a self-training approach for automatic speech recognition (ASR) for low-resource settings. While self-training approaches have been extensively developed and evaluated for high-resource languages such as English, their applications to low-resource languages like Punjabi have been limited, despite the language being spoken by millions globally. The scarcity of annotated data has hindered the development of accurate ASR systems, especially for low-resource languages (e.g., Punjabi and MÄori languages). To address this issue, we propose an effective self-training approach that generates highly accurate pseudo-labels for unlabeled low-resource speech. Our experimental analysis demonstrates that our approach significantly improves word error rate, achieving a relative improvement of 14.94% compared to a baseline model across four real-speech datasets. Further, our proposed approach reports the best results on the Common Voice Punjabi dataset.
FunASR: A Fundamental End-to-End Speech Recognition Toolkit
Authors: Zhifu Gao; Zerui Li; Jiaming Wang; Haoneng Luo; Xian Shi; Mengzhe Chen; Yabin Li; Lingyun Zuo; Zhihao Du; Shiliang Zhang
Venue: Interspeech 2023 (main)

Abstract: This paper introduces FunASR, an open-source speech recognition toolkit designed to bridge the gap between academic research and industrial applications. FunASR offers models trained on large-scale industrial corpora and the ability to deploy them in applications. The toolkitâs flagship model, Paraformer, is a non-autoregressive end-to-end speech recognition model that has been trained on a manually annotated Mandarin speech recognition dataset that contains 60,000 hours of speech. To improve the performance of Paraformer, we have added timestamp prediction and hotword customization capabilities to the standard Paraformer backbone. In addition, to facilitate model deployment, we have open-sourced a voice activity detection model based on the Feedforward Sequential Memory Network (FSMN-VAD) and a text post-processing punctuation model based on the controllable time-delay Transformer (CT-Transformer), both of which were trained on industrial corpora. These functional modules provide a solid foundation for building high-precision long audio speech recognition services. Compared to other models trained on open datasets, Paraformer demonstrates superior performance.
Streaming Audio-Visual Speech Recognition with Alignment Regularization
Authors: Pingchuan Ma; Niko Moritz; Stavros Petridis; Christian Fuegen; Maja Pantic
Venue: Interspeech 2023 (main)

Abstract: In this work, we propose a streaming AV-ASR system based on a hybrid connectionist temporal classification (CTC)/attention neural network architecture. The audio and the visual encoder neural networks are both based on the conformer architecture, which is made streamable using chunk-wise self-attention (CSA) and causal convolution. Streaming recognition with a decoder neural network is realized by using the triggered attention technique, which performs time-synchronous decoding with joint CTC/attention scoring. Additionally, we propose a novel alignment regularization technique that promotes synchronization of the audio and visual encoder, which in turn results in better word error rates (WERs) at all SNR levels for streaming and offline AV-ASR models. The proposed AV-ASR model achieves WERs of 2.0% and 2.6% on the Lip Reading Sentences 3 (LRS3) dataset in an offline and online setup, respectively, which both present state-of-the-art results when no external training data are used.
SparseVSR: Lightweight and Noise Robust Visual Speech Recognition
Authors: Adriana Fernandez-Lopez; Honglie Chen; Pingchuan Ma; Alexandros Haliassos; Stavros Petridis; Maja Pantic
Venue: Interspeech 2023 (main)

Abstract: Recent advances in deep neural networks have achieved unprecedented success in visual speech recognition. However, there remains substantial disparity between current methods and their deployment in resource-constrained devices. In this work, we explore different magnitude-based pruning techniques to generate a lightweight model that achieves higher performance than its dense model equivalent, especially under the presence of visual noise. Our sparse models achieve state-of-the-art results at 10% sparsity on the LRS3 dataset and outperform the dense equivalent up to 70% sparsity. We evaluate our 50% sparse model on 7 different visual noise types and achieve an overall absolute improvement of more than 2% WER compared to the dense equivalent. Our results confirm that sparse networks are more resistant to noise than dense networks.
Multimodal Speech Recognition for Language-Guided Embodied Agents
Authors: Allen Chang; Xiaoyuan Zhu; Aarav Monga; Seoho Ahn; Tejas Srinivasan; Jesse Thomason
Venue: Interspeech 2023 (main)

Abstract: Benchmarks for language-guided embodied agents typically assume text-based instructions, but deployed agents will encounter spoken instructions. While Automatic Speech Recognition (ASR) models can bridge the input gap, erroneous ASR transcripts can hurt the agentsâ ability to complete tasks. We propose training a multimodal ASR model that utilizes the accompanying visual context to reduce errors in spoken instruction transcripts. We train our model on a dataset of synthetic spoken instructions, derived from the ALFRED household task dataset, where we simulate acoustic noise by systematically masking spoken words. We find that utilizing visual observations facilitates masked word recovery, with multimodal ASR models recovering up to 30% more masked words than unimodal baselines. We also find that spoken instructions transcribed by multimodal ASR models result in higher task completion success rates for a language-guided embodied agent. github.com/Cylumn/embodied-multimodal-asr
Matching Latent Encoding for Audio-Text based Keyword Spotting
Authors: Kumari Nishu; Minsik Cho; Devang Naik
Venue: Interspeech 2023 (main)

Abstract: Using audio and text embeddings jointly for Keyword Spotting (KWS) has shown high-quality results, but the key challenge of how to semantically align two embeddings for multi-word keywords of different sequence lengths remains largely unsolved. In this paper, we propose an audio-text-based end-to-end model architecture for flexible keyword spotting (KWS), which builds upon learned audio and text embeddings. Our architecture uses a novel dynamic programming-based algorithm, Dynamic Sequence Partitioning (DSP), to optimally partition the audio sequence into the same length as the word-based text sequence using the monotonic alignment of spoken content. Our proposed model consists of an encoder block to get audio and text embeddings, a projector block to project individual embeddings to a common latent space, and an audio-text aligner containing a novel DSP algorithm, which aligns the audio and text embeddings to determine if the spoken content is the same as the text. Experimental results show that our DSP is more effective than other partitioning schemes, and the proposed architecture outperformed the state-of-the-art results on the public dataset in terms of Area Under the ROC Curve (AUC) and Equal-Error-Rate (EER) by 14.4 % and 28.9%, respectively.
Self-Paced Pattern Augmentation for Spoken Term Detection in Zero-Resource
Authors: Sudhakar P; Sreenivasa K. Rao; Pabitra Mitra
Venue: Interspeech 2023 (main)

Abstract: The spoken term detection task is challenging when a large volume of spoken content is generated without annotation. The pattern discovery approach aims to overcome the challenges by capturing the pattern similarities directly from the representation of the speech signal. A challenge to the pattern discovery task is handling the variabilities in natural speech. In the proposed approach, we aim to overcome the pattern variability challenges in the spoken term similarity region in three stages. At first, the pattern similarities between two spoken terms were captured using our heuristic search, and the pattern variabilities in the similarity region were observed. In the second stage, the observed pattern variabilities were augmented to the Siamese network to learn the relationship. Finally, the learned network is used to identify the matches between spoken query and document. Based on the experimental studies, it is observed that the proposed approach reduces the false alarms by 17.7% and improves the spoken term detection accuracy by 7.1% against the Microsoft Low-Resource Language corpus.
On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation
Authors: Gene-Ping Yang; Yue Gu; Qingming Tang; Dongsu Du; Yuzong Liu
Venue: Interspeech 2023 (main)

Abstract: Large self-supervised models are effective feature extractors, but their application is challenging under on-device budget constraints and biased dataset collection, especially in keyword spotting. To address this, we proposed a knowledge distillation-based self-supervised speech representation learning (S3RL) architecture for on-device keyword spotting. Our approach used a teacher-student framework to transfer knowledge from a larger, more complex model to a smaller, light-weight model using dual-view cross-correlation distillation and the teacher's codebook as learning objectives. We evaluated our model's performance on an Alexa keyword spotting detection task using a 16.6k-hour in-house dataset. Our technique showed exceptional performance in normal and noisy conditions, demonstrating the efficacy of knowledge distillation methods in constructing self-supervised models for keyword spotting tasks while working within on-device resource constraints.
Online Continual Learning in Keyword Spotting for Low-Resource Devices via Pooling High-Order Temporal Statistics
Authors: Umberto Michieli; Pablo Peso Parada; Mete Ozay
Venue: Interspeech 2023 (main)

Abstract: Keyword Spotting (KWS) models on embedded devices should adapt fast to new user-defined words without forgetting previous ones. Embedded devices have limited storage and computational resources, thus, they cannot save samples or update large models. We consider the setup of embedded online continual learning (EOCL), where KWS models with frozen backbone are trained to incrementally recognize new words from a non-repeated stream of samples, seen one at a time. To this end, we propose Temporal Aware Pooling (TAP) which constructs an enriched feature space computing high-order moments of speech features extracted by a pre-trained backbone. Our method, TAP-SLDA, updates a Gaussian model for each class on the enriched feature space to effectively use audio representations. In experimental analyses, TAP-SLDA outperforms competitors on several setups, backbones, and baselines, bringing a relative average gain of 11.3% on the GSC dataset.
Improving Small Footprint Few-shot Keyword Spotting with Supervision on Auxiliary Data
Authors: Seunghan Yang; Byeonggeun Kim; Kyuhong Shim; Simyoung Chang
Venue: Interspeech 2023 (main)

Abstract: Few-shot keyword spotting (FS-KWS) models usually require large-scale annotated datasets to generalize to unseen target keywords. However, existing KWS datasets are limited in scale and gathering keyword-like labeled data is costly undertaking. To mitigate this issue, we propose a framework that uses easily collectible, unlabeled reading speech data as an auxiliary source. Self-supervised learning has been widely adopted for learning representations from unlabeled data; however, it is known to be suitable for large models with enough capacity and is not practical for training a small footprint FS-KWS model. Instead, we automatically annotate and filter the data to construct a keyword-like dataset, LibriWord, enabling supervision on auxiliary data. We then adopt multi-task learning that helps the model to enhance the representation power from out-of-domain auxiliary data. Our method notably improves the performance over competitive methods in the FS-KWS benchmark.
Robust Keyword Spotting for Noisy Environments by Leveraging Speech Enhancement and Speech Presence Probability
Authors: Chouchang Yang; Yashas Malur Saidutta; Rakshith Sharma Srinivasa; Ching-Hua Lee; Yilin Shen; Hongxia Jin
Venue: Interspeech 2023 (main)

Abstract: Although various deep keyword spotting (KWS) systems have demonstrated promising performance under relatively noiseless environments, accurate keyword detection in the presence of strong noise remains challenging. Room acoustics and noise conditions can be highly diverse, leading to drastic performance degradation if not handled carefully. In this paper, we propose a noise management front-end called SE-SPP Net performing speech enhancement (SE) and speech presence probability (SPP) estimation jointly for robust KWS in noise. The SE-SPP Net estimates both the denoised Mel spectrogram and the position of the speech utterance in the noisy signal, where the latter is estimated as the probability of a particular time-frequency bin containing speech. Further, it comes at relatively no cost in model size when compared to a model estimating the denoised speech. Our SE-SPP Net can improve noisy KWS performance by up to 7% compared to a similar sized state-of-the-art model at SNR -10dB.
Enhancing the Unified Streaming and Non-streaming Model with Contrastive Learning
Authors: Yuting Yang; Yuke Li; Binbin Du
Venue: Interspeech 2023 (main)

Abstract: The unified streaming and non-streaming speech recognition model has achieved great success due to its comprehensive capabilities. In this paper, we propose to improve the accuracy of the unified model by bridging the inherent representation gap between the streaming and non-streaming modes with a contrastive objective. Specifically, the top-layer hidden representation at the same frame of the streaming and non-streaming modes are regarded as a positive pair, encouraging the representation of the streaming mode close to its non-streaming counterpart. The multiple negative samples are randomly selected from the rest frames of the same sample under the non-streaming mode. Experimental results demonstrate that the proposed method achieves consistent improvements toward the unified model in both streaming and non-streaming modes. Our method achieves CER of 4.66% in the streaming mode and CER of 4.31% in the non-streaming mode, which sets a new state-of-the-art on the AISHELL-1 benchmark.