GenAI4ELab/papercli-papers huggingface dataset:
Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understanding
Authors: Umberto Cappellazzo; Muqiao Yang; Daniele Falavigna; Alessio Brutti
Venue: Interspeech 2023 (main)

Abstract: The ability to learn new concepts sequentially is a major weakness for modern neural networks, which hinders their use in non-stationary environments. Their propensity to fit the current data distribution to the detriment of the past acquired knowledge leads to the catastrophic forgetting issue. In this work we tackle the problem of Spoken Language Understanding applied to a continual learning setting. We first define a class-incremental scenario for the SLURP dataset. Then, we propose three knowledge distillation (KD) approaches to mitigate forgetting for a sequence-to-sequence transformer model: the first KD method is applied to the encoder output (audio-KD), and the other two work on the decoder output, either directly on the token-level (tok-KD) or on the sequence-level (seq-KD) distributions. We show that the seq-KD substantially improves all the performance metrics, and its combination with the audio-KD further decreases the average WER and enhances the entity prediction metric.
Contrastive Disentangled Learning for Memory-Augmented Transformer
Authors: Jen-Tzung Chien; Shang-En Li
Venue: Interspeech 2023 (main)

Abstract: This paper developed a new memory-augmented sequential learning based on a contrastive disentangled transformer. Conventionally, transformer is insufficient to characterize long sequences since the sequence length is restricted to avoid the requirement of overlarge memory. A direct solution to handle this issue is to divide long sequence into short segments, but the context fragmentation will happen. In this paper, the contrastive disentangled memory is exploited to deal with the increasing computation cost as well as the overlarge memory requirement due to long sequences. In particular, an informative selection over the disentangled memory slots is proposed for iterative updating in a large-span sequence representation. This paper maximizes the semantic diversity of memory slots and captures the contextual semantics via contrastive learning. The experiments on language understanding show that the context fragmentation is mitigated by the proposed method with reduced computation.
ProsAudit, a prosodic benchmark for self-supervised speech models
Authors: Maureen de Seyssel; Marvin Lavechin; Hadrien Titeux; Arthur Thomas; Gwendal Virlet; Andrea Santos Revilla; Guillaume Wisniewski; Bogdan Ludusan; Emmanuel Dupoux
Venue: Interspeech 2023 (main)

Abstract: We present ProsAudit, a benchmark in English to assess structural prosodic knowledge in self-supervised learning (SSL) speech models. It consists of two subtasks, their corresponding metrics, and an evaluation dataset. In the protosyntax task, the model must correctly identify strong versus weak prosodic boundaries. In the lexical task, the model needs to correctly distinguish between pauses inserted between words and within words. We also provide human evaluation scores on this benchmark. We evaluated a series of SSL models and found that they were all able to perform above chance on both tasks, even when evaluated on an unseen language. However, non-native models performed significantly worse than native ones on the lexical task, highlighting the importance of lexical knowledge in this task. We also found a clear effect of size with models trained on more data performing better in the two subtasks.
Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces
Authors: Oli Danyi Liu; Hao Tang; Sharon Goldwater
Venue: Interspeech 2023 (main)

Abstract: Self-supervised speech representations are known to encode both speaker and phonetic information, but how they are distributed in the high-dimensional space remains largely unexplored. We hypothesize that they are encoded in orthogonal subspaces, a property that lends itself to simple disentanglement. Applying principal component analysis to representations of two predictive coding models, we identify two subspaces that capture speaker and phonetic variances, and confirm that they are nearly orthogonal. Based on this property, we propose a new speaker normalization method which collapses the subspace that encodes speaker information, without requiring transcriptions. Probing experiments show that our method effectively eliminates speaker information and outperforms a previous baseline in phone discrimination tasks. Moreover, the approach generalizes and can be used to remove information of unseen speakers.
Evaluating context-invariance in unsupervised speech representations
Authors: Mark Hallap; Emmanuel Dupoux; Ewan Dunbar
Venue: Interspeech 2023 (main)

Abstract: Unsupervised speech representations have taken off with benchmarks demonstrating major progress on semi-supervised speech recognition, speech synthesis, and speech-only language modelling. Inspiration comes from the promise of discovering the phonemes of a language or a similar low-bitrate encoding. However, one of the critical properties of phoneme transcriptions is context-invariance: the phonetic context of a speech sound can have massive influence on the way it is pronounced while text remains stable. This is why tokens of the same word have the same transcriptions---key to language understanding. Current benchmarks do not measure context-stability. We develop a new version of the ZeroSpeech ABX benchmark that does, and apply it to recent self-supervised representations. We show that context-independence of representations is predictive of the stability of word-level representations. We suggest research concentrate on improving context-independence of unsupervised representations.
CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning
Authors: Chutong Meng; Junyi Ao; Tom Ko; Mingxuan Wang; Haizhou Li
Venue: Interspeech 2023 (main)

Abstract: Speech is the surface form of a finite set of phonetic units, which can be represented by discrete codes. We propose the Code BERT (CoBERT) approach for self-supervised speech representation learning. The idea is to convert an utterance to a sequence of discrete codes, and perform code representation learning, where we predict the code representations based on a masked view of the original speech input. Unlike the prior self-distillation approaches of which the teacher and the student are of the same modality, our target model predicts representations from a different modality. CoBERT outperforms the most recent state-of-the-art performance on the ASR task and brings significant improvements on the SUPERB speech translation (ST) task. Our code and models are released at https://github.com/mct10/CoBERT.
Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering
Authors: Heng-Jui Chang; Alexander H. Liu; James Glass
Venue: Interspeech 2023 (main)

Abstract: Self-supervised speech representation models have succeeded in various tasks, but improving them for content-related problems using unlabeled data is challenging. We propose speaker-invariant clustering (Spin), a novel self-supervised learning method that clusters speech representations and performs swapped prediction between the original and speaker-perturbed utterances. Spin disentangles speaker information and preserves content representations with just 45 minutes of fine-tuning on a single GPU. Spin improves pre-trained networks and outperforms prior methods in speech recognition and acoustic unit discovery.
Self-Supervised Acoustic Word Embedding Learning via Correspondence Transformer Encoder
Authors: Jingru Lin; Xianghu Yue; Junyi Ao; Haizhou Li
Venue: Interspeech 2023 (main)

Abstract: Acoustic word embeddings (AWEs) aims to map a variable-length speech segment into a fixed-dimensional representation. High-quality AWEs should be invariant to variations, such as duration, pitch and speaker. In this paper, we introduce a novel self-supervised method to learn robust AWEs from a large-scale unlabelled speech corpus. Our model, named Correspondence Transformer Encoder (CTE), employs a teacher-student learning framework. We train the model based on the idea that different realisations of the same word should be close in the underlying embedding space. Specifically, we feed the teacher and student encoder with different acoustic instances of the same word and pre-train the model with a word-level loss. Our experiments show that the embeddings extracted from the proposed CTE model are robust to speech variations, e.g. speakers and domains. Additionally, when evaluated on Xitsonga, a low-resource cross-lingual setting, the CTE model achieves new state-of-the-art performance.
A Pipeline to Evaluate the Effects of Noise on Machine Learning Detection of Laryngeal Cancer
Authors: Mary Paterson; James Moor; Luisa Cutillo
Venue: Interspeech 2023 (main)

Abstract: Cases of laryngeal cancer are rising, with diagnosis often involving invasive biopsy procedures. An alternate approach is to identify high-risk patients by analysis of voice recordings which can alert clinical teams to those patients that need prioritisation. We propose a pipeline for evaluating speech classifier performance in the presence of noise. We perform experiments using the pipeline with several classifiers and denoising techniques. Random forest classifier performed best with an accuracy of 81.2% on clean data dropping to 63.8% when noise was added to recordings. The accuracy of all classifiers was reduced by added noise, signal denoising improved classifier accuracy but could not fully reverse the effects of noise. The effects of noise on classification is a complex issue which must be resolved for these detection systems to be implemented in clinical practice. We show that the proposed pipeline allows for the evaluation of classifier performance in the presence of noise.
ReCLR: Reference-Enhanced Contrastive Learning of Audio Representation for Depression Detection
Authors: Pingyue Zhang; Mengyue Wu; Kai Yu
Venue: Interspeech 2023 (main)

Abstract: Contrastive self-supervised learning has seen great success in computer vision while been less investigated in the audio processing field, in particular depression detection, a socially critical challenge. Detecting depression from one's speech has been examined via various audio representations, including acoustic feature combinations and model-based ones. This paper proposes to obtain depressive audio representations by departing speech via reference features from an emotion recognition model. Furthermore, we propose a reference-enhanced contrastive learning (ReCLR) to select fine-grained positive instances and allocate weight to negative instances. The depression detection results indicate that contrastive learning is effective in such an audio task. Moreover, our modified ReCLR strategy has outperformed contrastive training without references.
Automated Multiple Sclerosis Screening Based on Encoded Speech Representations
Authors: JosÃ© Egas-LÃ³pez; Veronika Svindt; Judit BÃ³na; IldikÃ³ Hoffmann; GÃ¡bor Gosztolya
Venue: Interspeech 2023 (main)

Abstract: Multiple Sclerosis (MS) is a chronic disease affecting over 2.5 million people worldwide. Its early detection is crucial for the management and treatment of the disease. Here we present an approach for automatic MS screening based on encoded speech representations. Our methods rely on Wav2Vec2 models to extract relevant traits from speech recordings of patients, which are then fed into a Support Vector Machine. Besides employing Wav2Vec2 models pre-trained on large public corpora, we also fine-tune them on 85 hours of the target language (Hungarian) in two distinct ways: for ASR and for speaker identification. Both variations outperformed the original models and conventional methods (ComParE functionals, x-vectors, and ECAPA-TDNN). Our findings suggest that fine-tuning for the actual speaker provides more advantages than the typical approach of fine-tuning for ASR purposes. Still, we improved our best MS discrimination performance when we fused features from our two fine-tuned models.
Cross-Lingual Features for Alzheimerâs Dementia Detection from Speech
Authors: Thomas Melistas; Lefteris Kapelonis; Nikos Antoniou; Petros Mitseas; Dimitris Sgouropoulos; Theodoros Giannakopoulos; Athanasios Katsamanis; Shrikanth Narayanan
Venue: Interspeech 2023 (main)

Abstract: Alzheimer's dementia is a neurodegenerative disease that affects millions of people worldwide. Early detection of Alzheimer's dementia is crucial for effective treatment and management of the disease. In this paper, we present a cross-lingual approach for detecting Alzheimer's dementia from speech, based on multiple feature streams that capture the individual's speech and conversational interactions. In order to validate the ability of the features to perform well in cross-linguistic scenarios, we evaluate in a zero-shot setup, where the target domain is a language that was not available during training and a few-shot setup, where only limited data is available. Experimental results show that an ensemble system using the features trained on English and evaluated on Greek outperforms the baseline system by 4.4 %. Further experiments show promising zero-shot and few-shot performance on a similar Spanish task.
Careful Whisper - leveraging advances in automatic speech recognition for robust and interpretable aphasia subtype classification
Authors: Mario Zusag; Laurin Wagner; Theresa Bloder
Venue: Interspeech 2023 (main)

Abstract: This paper presents a fully automated approach for identifying speech anomalies from voice recordings to aid in the as- sessment of speech impairments. By combining Connectionist Temporal Classification (CTC) and encoder-decoder-based automatic speech recognition models, we generate rich acoustic and clean transcripts. We then apply several natural language processing methods to extract features from these transcripts to produce prototypes of healthy speech. Basic distance measures from these prototypes serve as input features for standard machine learning classifiers, yielding human-level accuracy for the distinction between recordings of people with aphasia and a healthy control group. Furthermore, the most frequently occurring aphasia types can be distinguished with 90% accuracy. The pipeline is directly applicable to other diseases and languages, showing promise for robustly extracting diagnostic speech biomarkers.
Behavioral Analysis of Pathological Speaker Embeddings of Patients During Oncological Treatment of Oral Cancer
Authors: Jenthe Thienpondt; Caroline M. Speksnijder; Kris Demuynck
Venue: Interspeech 2023 (main)

Abstract: In this paper, we analyze the behavior of speaker embeddings of patients during oral cancer treatment. First, we found that pre- and post-treatment speaker embeddings differ significantly, notifying a substantial change in voice characteristics. However, a partial recovery to pre-operative voice traits is observed after 12 months post-operation. Secondly, the same-speaker similarity at distinct treatment stages is similar to healthy speakers, indicating that the embeddings can capture characterizing features of even severely impaired speech. Finally, a speaker verification analysis signifies a stable false positive rate and variable false negative rate when combining speech samples of different treatment stages. This indicates robustness of the embeddings towards other speakers, while still capturing the changing voice characteristics during treatment. To the best of our knowledge, this is the first analysis of speaker embeddings during oral cancer treatment of patients.
Adversarial Learning of Intermediate Acoustic Feature for End-to-End Lightweight Text-to-Speech
Authors: Hyungchan Yoon; Seyun Um; Changhwan Kim; Hong-Goo Kang
Venue: Interspeech 2023 (main)

Abstract: To simplify the generation process, several text-to-speech (TTS) systems implicitly learn intermediate latent representations instead of relying on predefined features (e.g., mel-spectrogram). However, their generation quality is unsatisfactory as these representations lack speech variances. In this paper, we improve TTS performance by adding prosody embeddings to the latent representations. During training, we extract reference prosody embeddings from mel-spectrograms, and during inference, we estimate these embeddings from text using generative adversarial networks (GANs). Using GANs, we reliably estimate the prosody embeddings in a fast way, which have complex distributions due to the dynamic nature of speech. We also show that the prosody embeddings work as efficient features for learning a robust alignment between text and acoustic features. Our proposed model surpasses several publicly available models with less parameters and computational complexity in comparative experiments.
Adapter-Based Extension of Multi-Speaker Text-To-Speech Model for New Speakers
Authors: Cheng-Ping Hsieh; Subhankar Ghosh; Boris Ginsburg
Venue: Interspeech 2023 (main)

Abstract: Fine-tuning is a popular method for adapting text-to-speech (TTS) models to new speakers. However, this approach has some challenges. Usually, fine-tuning requires several hours of high quality speech per speaker. Fine-tuning might negatively affect the quality of speech synthesis for previously learned speakers. In this paper, we propose an alternative approach for TTS adaptation based on using parameter-efficient adapter modules. In the proposed approach, a few adapter modules are added between the layers of the pretrained network. The pretrained model is frozen, and only the adapters are fine-tuned to the speech of a new speaker. Our approach will produce a new model with a high level of parameter sharing with the original model. Our experiments on LibriTTS, HiFi-TTS and VCTK datasets validate our adapter-based method through objective and subjective metrics. The code is open-sourced and the audio samples are available on our demo page.
SALTTS: Leveraging Self-Supervised Speech Representations for improved Text-to-Speech Synthesis
Authors: Ramanan Sivaguru; Vasista Sai Lodagala; S Umesh
Venue: Interspeech 2023 (main)

Abstract: While FastSpeech2 aims to integrate aspects of speech such as pitch, energy, and duration as conditional inputs, it still leaves scope for richer representations. As a part of this work, we leverage representations from various Self-Supervised Learning (SSL) models to enhance the quality of the synthesized speech. In particular, we pass the FastSpeech2 encoder's length-regulated outputs through a series of encoder layers with the objective of reconstructing the SSL representations. In the SALTTS-parallel implementation, the representations from this second encoder are used for an auxiliary reconstruction loss with the SSL features. The SALTTS-cascade implementation, however, passes these representations through the decoder in addition to having the reconstruction loss. The richness of speech characteristics from the SSL features reflects in the output speech quality, with the objective and subjective evaluation measures of the proposed approach outperforming the baseline FastSpeech2.
UnitSpeech: Speaker-adaptive Speech Synthesis with Untranscribed Data
Authors: Heeseung Kim; Sungwon Kim; Jiheum Yeom; Sungroh Yoon
Venue: Interspeech 2023 (main)

Abstract: We propose UnitSpeech, a speaker-adaptive speech synthesis method that fine-tunes a diffusion-based text-to-speech (TTS) model using minimal untranscribed data. To achieve this, we use the self-supervised unit representation as a pseudo transcript and integrate the unit encoder into the pre-trained TTS model. We train the unit encoder to provide speech content to the diffusion-based decoder and then fine-tune the decoder for speaker adaptation to the reference speaker using a singlepair. UnitSpeech performs speech synthesis tasks such as TTS and voice conversion (VC) in a personalized manner without requiring model re-training for each task. UnitSpeech achieves comparable and superior results on personalized TTS and any-to-any VC tasks compared to previous baselines. Our model also shows widespread adaptive performance on real-world data and other tasks that use a unit sequence as input.
LightVoc: An Upsampling-Free GAN Vocoder Based On Conformer And Inverse Short-time Fourier Transform
Authors: Dinh Son Dang; Tung Lam Nguyen; Bao Thang Ta; Tien Thanh Nguyen; Thi Ngoc Anh Nguyen; Dang Linh Le; Nhat Minh Le; Van Hai Do
Venue: Interspeech 2023 (main)

Abstract: Most neural vocoders based on generative adversarial networks (GANs) rely on iterative upsampling to generate audio sequences from mel-spectrograms as well as dilated convolution to expand their receptive fields. Nevertheless, iterative upsampling increases the network's complexity and thus decreases the inference speed. Moreover, convolution neural networks are geared towards extracting fine-grained local information and still struggle to capture long-term dependencies. In this work, we propose LightVoc, an efficient and high-quality GAN-based neural vocoder that replaces all upsampling blocks with a stack of Conformer blocks and uses a novel combination of discriminators to generate high-resolution waveforms over the full-band. From our experiments on LJSpeech dataset, LightVoc produces comparable audio quality while being 52.5 times faster in terms of CPU-based inference speed in comparison to HiFi-GAN V1.
ChatGPT-EDSS: Empathetic Dialogue Speech Synthesis Trained from ChatGPT-derived Context Word Embeddings
Authors: Yuki Saito; Shinnosuke Takamichi; Eiji Iimori; Kentaro Tachibana; Hiroshi Saruwatari
Venue: Interspeech 2023 (main)

Abstract: We propose ChatGPT-EDSS, an empathetic dialogue speech synthesis (EDSS) method using ChatGPT for extracting dialogue context. ChatGPT is a chatbot that can deeply understand the content and purpose of an input prompt and appropriately respond to the user's request. We focus on ChatGPT's reading comprehension and introduce it to EDSS, a task of synthesizing speech that can empathize with the interlocutor's emotion. Our method first gives chat history to ChatGPT and asks it to generate three words representing the intention, emotion, and speaking style for each line in the chat. Then, it trains an EDSS model using the embeddings of ChatGPT-derived context words as the conditioning features. The experimental results demonstrate that our method performs comparably to ones using emotion labels or neural network-derived context embeddings learned from chat histories. The collected ChatGPT-derived context information is available at our project page.
Human Transcription Quality Improvement
Authors: Jian Gao; Hanbo Sun; Cheng Cao; Zheng Du
Venue: Interspeech 2023 (main)

Abstract: High quality transcription data is crucial for training automatic speech recognition (ASR) systems. However, the existing industry-level data collection pipelines are expensive to researchers, while the quality of crowdsourced transcription is low. In this paper, we propose a reliable method to collect speech transcriptions. We introduce two mechanisms to improve transcription quality: confidence estimation based reprocessing at labeling stage, and automatic word error correction at post-labeling stage. We collect and release LibriCrowd - a large-scale crowdsourced dataset of audio transcriptions on 100 hours of English speech. Experiment shows the Transcription WER is reduced by over 50%. We further investigate the impact of transcription error on ASR model performance and found a strong correlation. The transcription quality improvement provides over 10% relative WER reduction for ASR models. We release the dataset and code to benefit the research community.