GenAI4ELab/papercli-papers huggingface dataset:
Visualizing Data Augmentation in Deep Speaker Recognition
Authors: Pengqi Li; Lantian Li; Askar Hamdulla; Dong Wang
Venue: Interspeech 2023 (main)

Abstract: Visualization is of great value in understanding the internal mechanisms of neural networks. Previous work found that LayerCAM is a reliable visualization tool for deep speaker models. In this paper, we use LayerCAM to analyze the widely-adopted data augmentation (DA) approach, to understand how it leads to model robustness. We conduct experiments on the VoxCeleb1 dataset for speaker identification, which shows that both vanilla and activation-based (Act) DA approaches enhance robustness against interference, with Act DA being consistently superior. Visualization with LayerCAM suggests DA helps models learn to delete temporal-frequency (TF) bins that are corrupted by interference. The âlearn to deleteâ behavior explained why DA models are more robust than clean models, and why the Act DA is superior over the vanilla DA when the interference is non-target speech. However, LayerCAM still cannot clearly explain the superiority of Act DA in other situations, suggesting further research.
Fast and Efficient Multilingual Self-Supervised Pre-training for Low-Resource Speech Recognition
Authors: Zhilong Zhang; Wei Wang; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: Recent advances in self-supervised learning (SSL) have remarkably improved speech recognition performance for low-resource languages. On the other hand, with data of an increasingly larger scale required for SSL, the pretraining process has become extremely time-consuming. To address this problem, we propose an unsupervised data selection method based on utterance-level language similarity and a curriculum learning strategy to boost the efficiency of multilingual SSL pretraining while maintaining performance. We conduct experiments on five languages in COMMONVOICE dataset. Compared to the baseline with all data for pretraining, we pretrained on only 25% of the data and saved 60% of the training steps with even better performance on the target low-resource language.
UniSplice: Universal Cross-Lingual Data Splicing for Low-Resource ASR
Authors: Wei Wang; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: End-to-end (E2E) automatic speech recognition (ASR) has made remarkable progress thanks to the abundant annotated data for a few rich-resource languages. However, data scarcity remains a challenge for the majority of the worldâs languages. To address this issue, we propose UniSplice, a novel cross-lingual speech synthesis framework based on data splicing that leverages self-supervised learning (SSL) units from Hidden Unit BERT (HuBERT) as universal phonetic units. Our approach involves splicing speech fragments from rich-resource languages into complete speech that conforms acoustically to text from low-resource languages. UniSplice eliminates the need for computationally expensive neural text-to-speech (TTS) models, enabling the training of ASR models using on-the-fly synthesized speech. Experimental results on the COMMON-VOICE dataset show 20-30% relative improvement for four Indo-European languages and about 15% for Turkish with a 4-gram language model for rescoring, in a 10-hour low-resource setup.
Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes
Authors: Kevin Glocker; Aaricia Herygers; Munir Georges
Venue: Interspeech 2023 (main)

Abstract: This paper proposes Allophant, a multilingual phoneme recognizer. It requires only a phoneme inventory for cross-lingual transfer to a target language, allowing for low-resource recognition. The architecture combines a compositional phone embedding approach with individually supervised phonetic attribute classifiers in a multi-task architecture. We also introduce Allophoible, an extension of the PHOIBLE database. When combined with a distance based mapping approach for grapheme-to-phoneme outputs, it allows us to train on PHOIBLE inventories directly. By training and evaluating on 34 languages, we found that the addition of multi-task learning improves the model's capability of being applied to unseen phonemes and phoneme inventories. On supervised languages we achieve phoneme error rate improvements of 11 percentage points (pp.) compared to a baseline without multi-task learning. Evaluation of zero-shot transfer on 84 languages yielded a decrease in PER of 2.63 pp. over the baseline.
Phonetic-assisted Multi-Target Units Modeling for Improving Conformer-Transducer ASR system
Authors: Li Li; Dongxing Xu; Haoran Wei; Yanhua Long
Venue: Interspeech 2023 (main)

Abstract: Exploiting effective target modeling units is very important and has always been a concern in end-to-end automatic speech recognition (ASR). In this work, we propose a phonetic-assisted multi-target units (PMU) modeling approach, to enhance the Conformer-Transducer ASR system in a progressive representation learning manner. Specifically, PMU first uses the pronunciation-assisted subword modeling (PASM) and byte pair encoding (BPE) to produce phonetic-induced and text-induced target units separately; Then, three new frameworks are investigated to enhance the acoustic encoder, including a basic PMU, a paraCTC and a paCTC, they integrate the PASM and BPE units at different levels for CTC and transducer multi-task training. Experiments on both LibriSpeech and accented ASR tasks show that, the proposed PMU significantly outperforms the conventional BPE, it reduces the WER of LibriSpeech clean, other, and six accented ASR testsets by relative 12.7%, 4.3% and 7.7%, respectively.
Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages
Authors: Andrew Rouditchenko; Sameer Khurana; Samuel Thomas; Rogerio Feris; Leonid Karlinsky; Hilde Kuehne; David Harwath; Brian Kingsbury; James Glass
Venue: Interspeech 2023 (main)

Abstract: Recent models such as XLS-R and Whisper have made multilingual speech technologies more accessible by pre-training on audio from around 100 spoken languages each. However, there are thousands of spoken languages worldwide, and adapting to new languages is an important problem. In this work, we aim to understand which model adapts better to languages unseen during pre-training. We fine-tune both models on 13 unseen languages and 18 seen languages. Our results show that the number of hours seen per language and language family during pre-training is predictive of how the models compare, despite the significant differences in the pre-training methods.
DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model
Authors: Haoyu Wang; Siyuan Wang; Wei-Qiang Zhang; Jinfeng Bai
Venue: Interspeech 2023 (main)

Abstract: Multilingual self-supervised speech representation models have greatly enhanced the speech recognition performance for low-resource languages, and the compression of these huge models has also become a crucial prerequisite for their industrial application. In this paper, we propose DistilXLSR, a distilled cross-lingual speech representation model. By randomly shuffling the phonemes of existing speech, we reduce the linguistic information and distill cross-lingual models using only English data. We also design a layer-jumping initialization method to fully leverage the teacher's pre-trained weights. Experiments on 2 kinds of teacher models and 15 low-resource languages show that our method can reduce the parameters by 50% while maintaining cross-lingual representation ability. Our method is proven to be generalizable to various languages/teacher models and has the potential to improve the cross-lingual performance of the English pre-trained models.
Emotional Voice Conversion with Semi-Supervised Generative Modeling
Authors: Hai Zhu; Huayi Zhan; Hong Cheng; Ying Wu
Venue: Interspeech 2023 (main)

Abstract: Emotional Voice Conversion (EVC) is a task that aims to convert the emotional state of speech from one to another while preserving the linguistic information and identity of the speaker. However, many studies are limited by the requirement for parallel speech data between different emotional patterns, which is not widely available in real-life applications. Furthermore, the annotation of emotional data is highly time-consuming and labor-intensive. To address these problems, in this paper, we propose SGEVC, a novel semi-supervised generative model for emotional voice conversion. This paper demonstrates that using as little as 1% supervised data is sufficient to achieve EVC. Experimental results show that our proposed model achieves state-of-the-art (SOTA) performance and consistently outperforms EVC baseline frameworks.
Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation
Authors: Ha-Yeong Choi; Sang-Hoon Lee; Seong-Whan Lee
Venue: Interspeech 2023 (main)

Abstract: Although voice conversion (VC) systems have shown a remarkable ability to transfer voice style, existing methods still have an inaccurate pitch and low speaker adaptation quality. To address these challenges, we introduce Diff-HierVC, a hierarchical VC system based on two diffusion models. We first introduce DiffPitch, which can effectively generate F0 with the target voice style. Subsequently, the generated F0 is fed to DiffVoice to convert the speech with a target voice style. Furthermore, using the source-filter encoder, we disentangle the speech and use the converted Mel-spectrogram as a data-driven prior in DiffVoice to improve the voice style transfer capacity. Finally, by using the masked prior in diffusion models, our model can improve the speaker adaptation quality. Experimental results verify the superiority of our model in pitch generation and voice style transfer performance, and our model also achieves a CER of 0.83% and EER of 3.29% in zero-shot VC scenarios.
S2CD: Self-heuristic Speaker Content Disentanglement for Any-to-Any Voice Conversion
Authors: Pengfei Wei; Xiang Yin; Chunfeng Wang; Zhonghao Li; Xinghua Qu; Zhiqiang Xu; Zejun Ma
Venue: Interspeech 2023 (main)

Abstract: In this paper, we propose a Selfheuristic Speaker Content Disentanglement (S2CD) model for any to any voice conversion without using any external resources, e.g., speaker labels or vectors, linguistic models, and transcriptions. S2CD is built on the disentanglement sequential variational autoencoder (DSVAE), but improves DSVAE at the model architecture level from three perspectives. Specifically, we develop different structures for speaker and content encoders based on their underlying static/dynamic property. We further propose a generative graph, modelled by S2CD, so as to make S2CD well mimic the multi-speaker speech generation process. Finally, we propose a self-heuristic way to introduce bias to the prior modelling. Extensive empirical evaluations show the effectiveness of S2CD for any to any voice conversion.
Flow-VAE VC: End-to-End Flow Framework with Contrastive Loss for Zero-shot Voice Conversion
Authors: Le Xu; Rongxiu Zhong; Ying Liu; Huibao Yang; Shilei Zhang
Venue: Interspeech 2023 (main)

Abstract: Voice conversion (VC) seeks to modify one speakerâs voice to generate speech as if it came from another speaker. It is challenging especially when source and target speakers are unseen during training (zero-shot VC). Recent work in this area made progress with disentanglement methods that separate utterance content and speaker characteristics from speech audio recordings. However, these models either lack adequate disentanglement ability or rely on the use of a trained vocoder to reconstruct the speech from acoustic features. We propose Flow-VAE VC, which is an end-to-end system processing directly on the raw audio waveform for zero-shot tasks. Flow-VAE VC adopts a conditional Variational Autoencoder (VAE) with normalizing flows and an adversarial training process to improve the expressive power of generative modeling. Specifically, we learn context-invariant representations by applying frame-level contrastive loss to speech different augment samples. The experiments show that the proposed method achieves a decent performance on zero-shot voice conversion and significantly improves converted speech naturalness and speaker similarity. Readers can get the source code and listen to some audio samples on the demo webpage.
Automatic Speech Disentanglement for Voice Conversion using Rank Module and Speech Augmentation
Authors: Zhonghua Liu; Shijun Wang; Ning Chen
Venue: Interspeech 2023 (main)

Abstract: Voice Conversion (VC) converts the voice of a source speech to that of a target while maintaining the source's content. Speech can be mainly decomposed into four components: content, timbre, rhythm and pitch. Unfortunately, most related works only take into account content and timbre, which results in less natural speech. Some recent works are able to disentangle speech into several components, but they require laborious bottleneck tuning or various hand-crafted features, each assumed to contain disentangled speech information. In this paper, we propose a VC model that can automatically disentangle speech into four components using only two augmentation functions, without the requirement of multiple hand-crafted features or laborious bottleneck tuning. The proposed model is straightforward yet efficient, and the empirical results demonstrate that our model can achieve a better performance than the baseline, regarding disentanglement effectiveness and speech naturalness.
End-to-End Zero-Shot Voice Conversion with Location-Variable Convolutions
Authors: Wonjune Kang; Mark Hasegawa-Johnson; Deb Roy
Venue: Interspeech 2023 (main)

Abstract: Zero-shot voice conversion is becoming an increasingly popular research topic, as it promises the ability to transform speech to sound like any speaker. However, relatively little work has been done on end-to-end methods for this task, which are appealing because they remove the need for a separate vocoder to generate audio from intermediate features. In this work, we propose LVC-VC, an end-to-end zero-shot voice conversion model that uses location-variable convolutions (LVCs) to jointly model the conversion and speech synthesis processes. LVC-VC utilizes carefully designed input features that have disentangled content and speaker information, and it uses a neural vocoder-like architecture that utilizes LVCs to efficiently combine them and perform voice conversion while directly synthesizing time domain audio. Experiments show that our model achieves especially well balanced performance between voice style transfer and speech intelligibility compared to several baselines.
Classifying Dementia in the Presence of Depression: A Cross-Corpus Study
Authors: Franziska Braun; Sebastian P. Bayerl; Paula A. PÃ©rez-Toro; Florian HÃ¶nig; Hartmut Lehfeld; Thomas Hillemacher; Elmar NÃ¶th; Tobias Bocklet; Korbinian Riedhammer
Venue: Interspeech 2023 (main)

Abstract: Automated dementia screening enables early detection and intervention, reducing costs to healthcare systems and increasing quality of life for those affected. Depression has shared symptoms with dementia, adding complexity to diagnoses. The research focus so far has been on binary classification of dementia (DEM) and healthy controls (HC) using speech from picture description tests from a single dataset. In this work, we apply established baseline systems to discriminate cognitive impairment in speech from the semantic Verbal Fluency Test and the Boston Naming Test using text, audio and emotion embeddings in a 3-class classification problem (HC vs. MCI vs. DEM). We perform cross-corpus and mixed-corpus experiments on two independently recorded German datasets to investigate generalization to larger populations and different recording conditions. In a detailed error analysis, we look at depression as a secondary diagnosis to understand what our classifiers actually learn.
Exploiting Cross-Domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition
Authors: Shujie Hu; Xurong Xie; Mengzhe Geng; Mingyu Cui; Jiajun Deng; Guinan Li; Tianzi Wang; Helen Meng; Xunying Liu
Venue: Interspeech 2023 (main)

Abstract: Articulatory features (AFs) are inherently invariant to acoustic signal distortion. Their practical application to atypical domains such as elderly, disordered speech across languages is limited by data scarcity. This paper presents a cross-domain and cross-lingual Acoustic-to-Articulatory (A2A) inversion approach that utilizes the parallel audio and ultrasound tongue imaging (UTI) data of the 24-hour TaL corpus in A2A model training before being adapted to three datasets: the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech corpora; and the English TORGO dysarthric speech data, to produce UTI based AFs. Experiments suggest incorporating the generated AFs consistently outperforms the baseline TDNN/Conformer ASR systems using acoustic features only by statistically significant word/character error rate reductions up to 4.75%, 2.59% and 2.07% absolute (14.69%, 10.64% and 22.72% relative) after data augmentation, speaker adaptation and cross system multi-pass decoding.
Multi-class Detection of Pathological Speech with Latent Features: How does it perform on unseen data?
Authors: Dominik Wagner; Ilja Baumann; Franziska Braun; Sebastian P. Bayerl; Elmar NÃ¶th; Korbinian Riedhammer; Tobias Bocklet
Venue: Interspeech 2023 (main)

Abstract: The detection of pathologies from speech features is usually defined as a binary classification task with one class representing a specific pathology and the other class representing healthy speech. In this work, we train neural networks, large margin classifiers, and tree boosting machines to distinguish between four pathologies: Parkinson's disease, laryngeal cancer, cleft lip and palate, and oral squamous cell carcinoma. We show that latent representations extracted at different layers of a pre-trained wav2vec 2.0 system can be effectively used to classify these types of pathological voices. We evaluate the robustness of our classifiers by adding room impulse responses to the test data and by applying them to unseen speech corpora. Our approach achieves unweighted average F1-Scores between 74.1% and 97.0%, depending on the model and the noise conditions used. The systems generalize and perform well on unseen data of healthy speakers sampled from a variety of different sources.
Responsiveness, Sensitivity and Clinical Utility of Timing-Related Speech Biomarkers for Remote Monitoring of ALS Disease Progression
Authors: Hardik Kothare; Michael Neumann; Jackson Liscombe; Jordan Green; Vikram Ramanarayanan
Venue: Interspeech 2023 (main)

Abstract: In this study, we describe the responsiveness of timing-related measures extracted from read speech in persons with ALS (pALS) collected via a remote patient monitoring platform in an effort to quantify how long it takes to detect a clinically-meaningful change associated with disease progression. We found that the timing alignment of pALS speech relative to a canonical elicitation of the same prompt is the most responsive measure, of the ones considered in this study, at detecting such change in both pALS with bulbar (n = 35) and non-bulbar onset (n = 94). We further evaluated the sensitivity of speech metrics in tracking disease progression in pALS while their ALSFRS-R speech score remained unchanged at 3 out of a total possible score of 4. We observed that timing-related speech metrics showed significant longitudinal changes even after accounting for learning effects. The findings of this study have the potential to inform disease prognosis and functional outcomes of clinical trials.
Use of Speech Impairment Severity for Dysarthric Speech Recognition
Authors: Mengzhe Geng; Zengrui Jin; Tianzi Wang; Shujie Hu; Jiajun Deng; Mingyu Cui; Guinan Li; Jianwei Yu; Xurong Xie; Xunying Liu
Venue: Interspeech 2023 (main)

Abstract: A key challenge in dysarthric speech recognition is the speaker-level diversity attributed to both speaker-identity related factors (e.g. gender) and speech impairment severity. Most prior researches on addressing this issue focused on using speaker-identity only. To this end, this paper proposes a novel set of techniques to use both severity and speaker-identity in dysarthric speech recognition: a) multitask training incorporating severity prediction error; b) speaker-severity aware auxiliary feature adaptation; and c) structured LHUC transforms separately conditioned on speaker-identity and severity. Experiments conducted on UASpeech suggest incorporating additional speech impairment severity into state-of-the-art hybrid DNN, E2E Conformer and pre-trained Wav2vec 2.0 ASR systems produced statistically significant WER reductions up to 4.78% (14.03% relative). Using the best system the lowest published WER of 17.82% (51.25% on very low intelligibility) was obtained on UASpeech.
MMLung: Moving Closer to Practical Lung Health Estimation using Smartphones
Authors: Mohammed Mosuily; Lindsay Welch; Jagmohan Chauhan
Venue: Interspeech 2023 (main)

Abstract: Long-term respiratory illnesses like Chronic Obstructive Pulmonary Disease (COPD) and Asthma are commonly diagnosed with the gold standard spirometry, which is a lung health test that requires specialized equipment and trained healthcare experts, making it expensive and difficult to scale. Moreover, blowing into a spirometer can be quite hard for people suffering from pulmonary illnesses. To solve the aforementioned limitations, we introduce MMLung, an approach that leverages information obtained from multiple audio signals by combining multiple tasks and different modalities performed on the microphone of a smartphone to estimate lung function. Our proposed approach achieves the best mean absolute percentage error (MAPE) of 1.3% on a cohort of 40 participants. Compared to the reported performances (5%-10% MAPE) on lung health estimation using smartphones, MMLung shows that practical lung health estimation is viable by combining multiple tasks utilizing multiple modalities.
Investigating the Utility of Synthetic Data for Doctor-Patient Conversation Summarization
Authors: Siyuan Chen; Colin A. Grambow; Mojtaba Kadkhodaie Elyaderani; Alireza Sadeghi; Federico Fancellu; Thomas Schaaf
Venue: Interspeech 2023 (main)

Abstract: Large-scale pre-training has been a successful strategy for training transformer models. However, maintaining a large clinical dataset for pre-training is not always possible, and access to data in this domain can be time-limited and costly. We explore using synthetic data in pre-training sequence-to-sequence (seq-to-seq) transformer models to generate clinical notes from Doctor-Patient-Conversations (DoPaCos). Using a generative language model fine-tuned on authentic conversations, a synthetic DoPaCo dataset was created and used with a corpus of clinical notes to pre-train a Longformer-Encoder-Decoder (LED) model. Results show that synthetic data leads to comparable performance in the downstream summarization task compared to pre-training with authentic data. Pre-training on synthetic conversations first, followed by clinical notes, yields higher performance across most of our evaluation metrics.