GenAI4ELab/papercli-papers huggingface dataset:
ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMs
Authors: Xingchen Song; Di Wu; Binbin Zhang; Zhendong Peng; Bo Dang; Fuping Pan; Zhiyong Wu
Venue: Interspeech 2023 (main)

Abstract: In this paper, we present ZeroPrompt (Figure 1-(a)) and the corresponding Prompt-and-Refine strategy (Figure 3), two simple but effective training-free methods to decrease the Token Display Time (TDT) of streaming ASR models without any accuracy loss. The core idea of ZeroPrompt is to append zeroed content to each chunk during inference, which acts like a prompt to encourage the model to predict future tokens even before they were spoken. We argue that streaming acoustic encoders naturally have the modeling ability of Masked Language Models and our experiments demonstrate that ZeroPrompt is engineering cheap and can be applied to streaming acoustic encoders on any dataset without any accuracy loss. Specifically, compared with our baseline models, we achieve 350~700ms reduction on First Token Display Time (TDT-F) and 100~400ms reduction on Last Token Display Time (TDT-L), with theoretically and experimentally equal WER on both Aishell-1 and Librispeech datasets.
Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuation
Authors: Hanbyul Kim; Seunghyun Seo; Lukas Lee; Seolki Baek
Venue: Interspeech 2023 (main)

Abstract: Punctuated text prediction is crucial for automatic speech recognition as it enhances readability and impacts downstream natural language processing tasks. In streaming scenarios, the ability to predict punctuation in real-time is particularly desirable but presents a difficult technical challenge. In this work, we propose a method for predicting punctuated text from input speech using a chunk-based Transformer encoder trained with Connectionist Temporal Classification (CTC) loss. The acoustic model trained with long sequences by concatenating the input and target sequences can learn punctuation marks attached to the end of sentences more effectively. Additionally, by combining CTC losses on the chunks and utterances, we achieved both the improved F1 score of punctuation prediction and Word Error Rate (WER).
DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer
Authors: Goeric Huybrechts; Srikanth Ronanki; Xilai Li; Hadis Nosrati; Sravan Bodapati; Katrin Kirchhoff
Venue: Interspeech 2023 (main)

Abstract: Conformer-based end-to-end models have become ubiquitous these days and are commonly used in both streaming and non-streaming automatic speech recognition (ASR). Techniques like dual-mode and dynamic chunk training helped unify streaming and non-streaming systems. However, there remains a performance gap between streaming with a full and limited past context. To address this issue, we propose the integration of a novel dynamic contextual carry-over mechanism in a state-of-the-art (SOTA) unified ASR system. Our proposed dynamic context Conformer (DCTX-Conformer) utilizes a non-overlapping contextual carry-over mechanism that takes into account both the left context of a chunk and one or more preceding context embeddings. We outperform the SOTA by a relative 25.0% word error rate, with a negligible latency impact due to the additional context embeddings.
Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer
Authors: Kyuhong Shim; Jinkyu Lee; Simyoung Chang; Kyuwoong Hwang
Venue: Interspeech 2023 (main)

Abstract: Streaming automatic speech recognition (ASR) models are restricted from accessing future context, which results in worse performance compared to the non-streaming models. To improve the performance of streaming ASR, knowledge distillation (KD) from the non-streaming to streaming model has been studied, mainly focusing on aligning the output token probabilities. In this paper, we propose a layer-to-layer KD from the teacher encoder to the student encoder. To ensure that features are extracted using the same context, we insert auxiliary non-streaming branches to the student and perform KD from the non-streaming teacher layer to the non-streaming auxiliary layer. We design a special KD loss that leverages the autoregressive predictive coding (APC) mechanism to encourage the streaming model to predict unseen future contexts. Experimental results show that the proposed method can significantly reduce the word error rate compared to previous token probability distillation methods.
Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition
Authors: Tianyi Xu; Zhanheng Yang; Kaixun Huang; Pengcheng Guo; Ao Zhang; Biao Li; Changru Chen; Chao Li; Lei Xie
Venue: Interspeech 2023 (main)

Abstract: By incorporating additional contextual information, deep biasing methods have emerged as a promising solution for speech recognition of personalized words. However, for real-world voice assistants, always biasing on personalized words with high prediction scores can degrade the performance of recognizing common words. To address this issue, we propose an adaptive contextual biasing method based on Context-Aware Transformer Transducer (CATT) that utilizes the biased encoder and predictor embeddings to perform streaming prediction of contextual phrase occurrences. This prediction is then used to switch the bias list on and off, enabling the model to adapt to both personalized and common scenarios. Experiments on Librispeech and internal voice assistant datasets show that our approach can achieve up to 6.7% and 20.7% relative reduction in WER and CER compared to the baseline respectively, mitigating up to 96.7% and 84.9% of the relative WER and CER increase for common cases.
Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model
Authors: HÃ©ctor Martel; Julius Richter; Kai Li; Xiaolin Hu; Timo Gerkmann
Venue: Interspeech 2023 (main)

Abstract: We propose Audio-Visual Lightweight ITerative model (AVLIT), an effective and lightweight neural network that uses Progressive Learning (PL) to perform audio-visual speech separation in noisy environments. To this end, we adopt the Asynchronous Fully Recurrent Convolutional Neural Network (A-FRCNN), which has shown successful results in audio-only speech separation. Our architecture consists of an audio branch and a video branch, with iterative A-FRCNN blocks sharing weights for each modality. We evaluated our model in a controlled environment using the NTCD-TIMIT dataset and in-the-wild using a synthetic dataset that combines LRS3 and WHAM!. The experiments demonstrate the superiority of our model in both settings with respect to various audio-only and audio-visual baselines. Furthermore, the reduced footprint of our model makes it suitable for low resource applications.
Remixing-based Unsupervised Source Separation from Scratch
Authors: Kohei Saijo; Tetsuji Ogawa
Venue: Interspeech 2023 (main)

Abstract: We propose an unsupervised approach for training separation models from scratch using RemixIT and Self-Remixing, which are recently proposed self-supervised learning methods for refining pre-trained models. They first separate mixtures with a teacher model and create pseudo-mixtures by shuffling and remixing the separated signals. A student model is then trained to separate the pseudo-mixtures using either the teacher's outputs or the initial mixtures as supervision. To refine the teacher's outputs, the teacher's weights are updated with the student's weights. While these methods originally assumed that the teacher is pre-trained, we show that they are capable of training models from scratch. We also introduce a simple remixing method to stabilize training. Experimental results demonstrate that the proposed approach outperforms mixture invariant training, which is currently the only available approach for training a monaural separation model from scratch.
CAPTDURE: Captioned Sound Dataset of Single Sources
Authors: Yuki Okamoto; Kanta Shimonishi; Keisuke Imoto; Kota Dohi; Shota Horiguchi; Yohei Kawaguchi
Venue: Interspeech 2023 (main)

Abstract: In conventional studies on environmental sound separation and synthesis using captions, datasets consisting of multiple-source sounds with their captions were used for model training. However, when we collect the captions for multiple-source sound, it is not easy to collect detailed captions for each sound source, such as the number of sound occurrences and timbre. Therefore, it is difficult to extract only the single-source target sound by the model-training method using a conventional captioned sound dataset. In this work, we constructed a dataset with captions for a single-source sound named CAPTDURE, which can be used in various tasks such as environmental sound separation and synthesis. Our dataset consists of 1,044 sounds and 4,902 captions. We evaluated the performance of environmental sound extraction using our dataset. The experimental results show that the captions for single-source sounds are effective in extracting only the single-source target sound from the mixture sound.
Recursive Sound Source Separation with Deep Learning-based Beamforming for Unknown Number of Sources
Authors: Hokuto Munakata; Ryu Takeda; Kazunori Komatani
Venue: Interspeech 2023 (main)

Abstract: We propose a recursive separation model for an unknown number of sound sources based on deep learning-based beamforming. Recursive separation models have been investigated as a way to separate a mixture signal composed of an unknown number of sources in a single-channel condition. The mixture signal is separated with residual information in a recursive manner. Although the recursive separation model can be extended to a multi-channel condition using a beamforming-based filter, the separation performance is degraded because the beamforming-based filter tends to accumulate estimation errors in the recursions. To address this problem, we introduce a local Gaussian model (LGM)-based recursive separation model. The proposed method mitigates the accumulation of errors by reusing estimated parameters and applying only one filter to the mixture signal. Experimental results show that our proposed method outperforms a separation model using an accumulative filter.
Multi-Channel Speech Separation with Cross-Attention and Beamforming
Authors: Ladislav Mosner; OldÅich Plchot; Junyi Peng; LukÃ¡Å¡ Burget; Jan "Honza" ÄernockÃ½
Venue: Interspeech 2023 (main)

Abstract: Originally, single-channel source separation gained more research interest. It resulted in immense progress. Multi-channel (MC) separation comes with new challenges posed by adverse indoor conditions making it an important field of study. We seek to combine promising ideas from the two worlds. First, we build MC models by extending current single-channel time-domain separators relying on their strength. Our approach allows reusing pre-trained models by inserting designed lightweight reference channel attention (RCA) combiner, the only trained module. It comprises two blocks: the former allows attending to different parts of other channels w.r.t. the reference one, and the latter provides an attention-based combination of channels. Second, like many successful MC models, our system incorporates beamforming and allows for the fusion of the network and beamformer outputs. We compare our approach with the SOTA models on the SMS-WSJ dataset and show better or similar performance.
Background-Sound Controllable Voice Source Separation
Authors: Deokjun Eom; Woo Hyun Nam; Kyung-Rae Kim
Venue: Interspeech 2023 (main)

Abstract: There have been various approaches to separate mixed voices. In the real world, input voices contain many different kinds of background sounds but existing methods have not considered the background sounds in model architectures. These approaches are difficult to control the background sounds directly and the voice separation results include the background sounds randomly. In this paper, we propose an extended voice separation framework, background-sound controllable voice source separation that can control the degrees of background sounds of voice separation outputs using a control parameter that ranges from 0 to 1 without additional mixing procedures. Several experiments show the controllability of background sounds on various real world datasets with preserving voice separation performances.
An Automatic Multimodal Approach to Analyze Linguistic and Acoustic Cues on Parkinson's Disease Patients
Authors: Daniel Escobar-Grisales; TomÃ¡s Arias-Vergara; Cristian David RÃ­os-Urrego; Elmar NÃ¶th; Adolfo M. GarcÃ­a; Juan Rafael Orozco-Arroyave
Venue: Interspeech 2023 (main)

Abstract: Early detection and monitoring of Parkinson's disease are crucial for properly treating and managing the symptoms. Automatic speech and language analysis has emerged as a promising non-invasive method to monitor the patient's state. This study analyzed different speech and language representations for automatic classification between Parkinson's disease patients and healthy controls. First, each modality is analyzed independently. General representations such as Wav2vec or BETO are used together with representations oriented to model disease traits such as phonemic identifiability in speech modality and grammatical units analysis in language modality. The best speech and language representations were combined using a fusion strategy based on Gated Multimodal Units. The best results are achieved with the multimodal approach, outperforming all results obtained with unimodal representations and the traditional fusion strategy.
Personalization for Robust Voice Pathology Detection in Sound Waves
Authors: Khanh-Tung Tran; Truong Hoang; Duy Khuong Nguyen; Hoang D. Nguyen; Xuan-Son Vu
Venue: Interspeech 2023 (main)

Abstract: Automatic voice pathology detection is promising for non-invasive screening and early intervention using sound signals. Nevertheless, existing methods are susceptible to covariate shifts due to background noises, human voice variations, and data selection biases leading to severe performance degradation in real-world scenarios. Hence, we propose a non-invasive framework that contrastively learns personalization from sound waves as a pre-train and predicts latent-spaced profile features through semi-supervised learning. It allows all subjects from various distributions (e.g., regionality, gender, age) to benefit from personalized predictions for robust voice pathology in a privacy-fulfilled manner. We extensively evaluate the framework on four real-world respiratory illnesses datasets, including Coswara, COUGHVID, ICBHI and our private dataset - ASound, under multiple covariate shift settings (i.e., cross-dataset), improving up to 4.12% in overall performance.
Integrated and Enhanced Pipeline System to Support Spoken Language Analytics for Screening Neurocognitive Disorders
Authors: Helen Meng; Brian Mak; Man-Wai Mak; Helene Fung; Xianmin Gong; Timothy Kwok; Xunying Liu; Vincent Mok; Patrick Wong; Jean Woo; Xixin Wu; Ka Ho Wong; Shensheng Xu; Naijun Zheng; Ranzo Huang; Jiawen Kang; Xiaoquan Ke; Junan Li; Jinchao Li; Yi Wang
Venue: Interspeech 2023 (main)

Abstract: This paper presents an enhanced pipeline system for automated screening of neurocognitive disorders, e.g. Alzheimer's Disease (AD), using spoken language technologies. To ensure local relevance, the pipeline is applied to two-way interactions between clinical assessors and older adult participants in spoken Cantonese, the predominant language used in Hong Kong. The pipeline includes: (i) Speaker diarization using speaker-turn-aware scoring to capture the temporal structure of conversations. (ii) ASR using XLS-R wav2vec 2.0 models further pre-trained on Cantonese speech data and fine-tuned. (iii) Language modelling using RoBERTa with further fine-tuning. (iv) AD screening with neural network classification. A reference benchmark is obtained using the ADReSS corpus where no diarization is needed, and the partial pipeline attained a competitive detection accuracy of 87.5%.
Capturing Mismatch between Textual and Acoustic Emotion Expressions for Mood Identification in Bipolar Disorder
Authors: Minxue Niu; Amrit Romana; Mimansa Jaiswal; Melvin McInnis; Emily Mower Provost
Venue: Interspeech 2023 (main)

Abstract: Emotion is a complex behavioral phenomenon, which is expressed and perceived through various modalities, such as language, vocal and facial expressions. Psychiatric research has suggested that the lack of emotional alignment between modalities is a symptom of emotion disorders. In this work, we quantify the mismatch between emotion expressed through language and acoustics, which we refer to as Emotional MisMatch (EMM), as an intermediate step for mood identification. We use a longitudinal dataset collected from people with Bipolar Disorder (BP) and show that symptomatic mood episodes show significantly more EMM, compared to euthymic moods. We propose a fully automatic mood identification pipeline with automatic speech transcription, emotion recognition, and EMM feature extraction. We find that EMM features, although smaller in size, outperform a language-based baseline, and consistently provide improvement when combined with language and/or raw emotion features on mood classification.
FTA-net: A Frequency and Time Attention Network for Speech Depression Detection
Authors: Qifei Li; Dong Wang; Yiming Ren; Yingming Gao; Ya Li
Venue: Interspeech 2023 (main)

Abstract: Depression is one of the most common mental diseases nowadays, which seriously affects the health of individuals. Some researchers have shown an association between the level of depression and speech features in individuals, so a lot of automatic speech-based depression detection systems have been proposed. A number of studies utilized convolutional neural network (CNN) to realize the speech depression detection. However, most of these studies did not take into account that different frequencies and time steps in the speech spectrum features contribute unequally to the detection of depression. In order to extract more significant and distinctive features, this paper proposes an effective frequency-time attention (FTA) module for CNN, which is based on squeeze and excitation operations and can emphasize the time steps and frequencies associated with depression. Experimental results based on the AVEC 2013 and AVEC 2014 benchmarks demonstrate the effectiveness of our proposed method.
Bayesian Networks for the robust and unbiased prediction of depression and its symptoms utilizing speech and multimodal data
Authors: Salvatore Fara; Orlaith Hickey; Alexandra Georgescu; Stefano Goria; Emilia Molimpakis; Nicholas Cummins
Venue: Interspeech 2023 (main)

Abstract: Predicting the presence of major depressive disorder (MDD) using speech is highly non-trivial. The heterogeneous clinical profile of MDD means that any given speech pattern may be associated with a unique combination of depressive symptoms. Conventional discriminative machine learning models may lack the complexity to robustly model this heterogeneity. Bayesian networks, however, are well-suited to such a scenario. They provide further advantages over standard discriminative modeling by offering the possibility to (i) fuse with other data streams; (ii) incorporate expert opinion into the models; (iii) generate explainable model predictions, inform about the uncertainty of predictions, and (iv) handle missing data. In this study, we apply a Bayesian framework to capture the relationships between depression, depression symptoms, and features derived from speech, facial expression and cognitive game data. Presented results also highlight our model is not subject to demographic biases.
Hyper-parameter Adaptation of Conformer ASR Systems for Elderly and Dysarthric Speech Recognition
Authors: Tianzi Wang; Shoukang Hu; Jiajun Deng; Zengrui Jin; Mengzhe Geng; Yi Wang; Helen Meng; Xunying Liu
Venue: Interspeech 2023 (main)

Abstract: Automatic recognition of disordered and elderly speech remains highly challenging tasks to date due to data scarcity. Parameter fine-tuning is often used to exploit the large quantities of non-aged and healthy speech pre-trained models, while neural architecture hyper-parameters are set using expert knowledge and remain unchanged. This paper investigates hyper-parameter adaptation for Conformer ASR systems that are pre-trained on the Librispeech corpus before being domain adapted to the DementiaBank elderly and UASpeech dysarthric speech datasets. Experimental results suggest that hyper-parameter adaptation produced word error rate (WER) reductions of 0.45% and 0.67% over parameter-only fine-tuning on DBank and UASpeech tasks respectively. An intuitive correlation is found between the performance improvements by hyper-parameter domain adaptation and the relative utterance length ratio between the source and target domain data.
Classifying depression symptom severity: Assessment of speech representations in personalized and generalized machine learning models.
Authors: Edward L. Campbell; Judith Dineley; Pauline Conde; Faith Matcham; Katie M. White; Carolin Oetzmann; Sara Simblett; Stuart Bruce; Amos A. Folarin; Til Wykes; Srinivasan Vairavan; Richard J. B. Dobson; Laura Docio-Fernandez; Carmen Garcia-Mateo; Vaibhav A. Narayan; Matthew Hotopf; Nicholas Cummins
Venue: Interspeech 2023 (main)

Abstract: There is an urgent need for new methods that improve the management and treatment of Major Depressive Disorder (MDD). Speech has long been regarded as a promising digital marker in this regard, with many works highlighting that speech changes associated with MDD can be captured through machine learning models. Typically, findings are based on cross-sectional data, with little work exploring the advantages of personalization in building more robust and reliable models. This work assesses the strengths of different combinations of speech representations and machine learning models, in personalized and generalized settings in a two-class depression severity classification paradigm. Key results on a longitudinal dataset highlight the benefits of personalization. Our strongest performing model set-up utilized self-supervised learning features and convolutional neural network (CNN) and long short-term memory (LSTM) back-end.
Active Learning for Abnormal Lung Sound Data Curation and Detection in Asthma
Authors: Shabnam Ghaffarzadegan; Luca Bondi; Ho-Hsiang Wu; Sirajum Munir; Kelly J. Shields; Samarjit Das; Joseph Aracri
Venue: Interspeech 2023 (main)

Abstract: Existing audio-based asthma monitoring solutions rely on feature engineering designs paired with contact-based auscultation which are brittle in practice and do not scale beyond point of care setups. Data-driven methods utilizing contactless microphones have the potential to address such limitations. These solutions are under-explored in healthcare due to high cost of data curation requiring physicians-in-the-loop. Here, we propose an active learning (AL) system to facilitate audio data collection and annotation. It detects lung sound abnormalities in asthma. AL reduces the annotation cost while increasing the model performance under a constrained annotation budget. It automatically extracts interesting audio segments from the continuous recordings, and efficiently annotates and trains anomaly detector model. The experimental results confirm the effectiveness of the proposed system as an enabler for larger scale data curation on a newly collected audio corpus for pediatric asthma.