GenAI4ELab/papercli-papers huggingface dataset:
LiteFocus: Accelerated Diffusion Inference for Long Audio Synthesis
Authors: Zhenxiong Tan; Xinyin Ma; Gongfan Fang; Xinchao Wang
Venue: Interspeech 2024 (main)

Abstract: Latent diffusion models have shown promising results in audio generation, making notable advancements over traditional methods. However, their performance, while impressive with short audio clips, faces challenges when extended to longer audio sequences. These challenges are due to modelâs selfattention mechanism and training predominantly on 10-second clips, which complicates the extension to longer audio without adaptation. In response to these issues, we introduce a novel approach, LiteFocus that enhances the inference of existing audio latent diffusion models in long audio synthesis. Observed the attention pattern in self-attention, we employ a dual sparse form for attention calculation, designated as same-frequency focus and cross-frequency compensation, which curtails the attention computation under same-frequency constraints, while enhancing audio quality through cross-frequency refillment. LiteFocus demonstrates substantial reduction on inference time with diffusion-based TTA model by 1.99Ã in synthesizing 80-second audio clips while also obtaining improved audio quality.
Speak in the Scene: Diffusion-based Acoustic Scene Transfer toward Immersive Speech Generation
Authors: Miseul Kim; Soo-Whan Chung; Youna Ji; Hong-Goo Kang; Min-Seok Choi
Venue: Interspeech 2024 (main)

Abstract: This paper introduces a novel task in generative speech processing, Acoustic Scene Transfer (AST), which aims to transfer acoustic scenes of speech signals to diverse environments. AST promises an immersive experience in speech perception by adapting the acoustic scene behind speech signals to desired environments. We propose AST-LDM for the AST task, which generates speech signals accompanied by the target acoustic scene of the reference prompt. Specifically, AST-LDM is a latent diffusion model conditioned by CLAP embeddings that describe target acoustic scenes in either audio or text modalities. The contributions of this paper include introducing the AST task and implementing its baseline model. For AST-LDM, we emphasize its core framework, which is to preserve the input speech and generate audio consistently with both the given speech and the target acoustic environment. Experiments, including objective and subjective tests, validate the feasibility and efficacy of our approach.
PL-TTS: A Generalizable Prompt-based Diffusion TTS Augmented by Large Language Model
Authors: Shuhua Li; Qirong Mao; Jiatong Shi
Venue: Interspeech 2024 (main)

Abstract: With the increasing demand for style-controlled speech synthesis, traditional TTS methods for controlling acoustic features clearly have significant limitations. Therefore, using text style descriptions to achieve style-controlled TTS has become a current hot topic. However, existing methods often have unsatisfactory results when dealing with unseen style descriptions and ignore the issue of adding various style conditions to the model, which can lead to poor training performance of the original model. In this context, we propose PL-TTS, an enhanced diffusion-based TTS combined with prompts embedded by a large language model. In order to improve synthesis quality and style control ability, an enhanced diffusion-based framework and a method for fine-tuning large language models have been proposed. Experimental results in LibriTTS-R validate the effectiveness of PL-TTS in fine grained style control and generalization.
Towards realtime co-speech gestures synthesis using STARGATE
Authors: Louis Abel; Vincent Colotte; Slim Ouni
Venue: Interspeech 2024 (main)

Abstract: The field of co-speech gestures synthesis is gaining more and more interest. However, many new systems utilize complex or resource-intensive architectures, making them impractical for integration into Embodied Conversational Agents (ECAs) or for exploration in fields like linguistics, where understanding the connection between speech and gestures is challenging. This paper introduces STARGATE, a novel architecture for Spatio-Temporal Autoregressive Graph from Audio-Text Embeddings. The model leverages autoregression for fast gestures generation, alongside graph convolutions and attention to integrate explicit structural knowledge and facilitate efficient spatial and temporal processing. Through both subjective and objective assessments against state-of-the-art models, our research demonstrates our model capabilities of generating convincing gestures fast. It also achieves slightly better scores in terms of credibility and coherence of generated gestures in relation to speech.
PPPR: Portable Plug-in Prompt Refiner for Text to Audio Generation
Authors: Shuchen Shi; Ruibo Fu; Zhengqi Wen; Jianhua Tao; Tao Wang; Chunyu Qiang; Yi Lu; Xin Qi; Xuefei Liu; Yukun Liu; Yongwei Li; Zhiyong Wang; Xiaopeng Wang
Venue: Interspeech 2024 (main)

Abstract: Text-to-Audio (TTA) aims to generate audio that corresponds to the given text description, playing a crucial role in media production. The text descriptions in TTA datasets lack rich variations and diversity, resulting in a drop in TTA model performance when faced with complex text. To address this issue, we propose a method called Portable Plug-in Prompt Refiner, which utilizes rich knowledge about textual descriptions inherent in large language models to effectively enhance the robustness of TTA acoustic models without altering the acoustic training set. Furthermore, a Chain-of-Thought that mimics human verification is introduced to enhance the accuracy of audio descriptions, thereby improving the accuracy of generated content in practical applications. The experiments show that our method achieves a state-of-the-art Inception Score (IS) of 8.72, surpassing AudioGen, AudioLDM and Tango.
Neural ATSM: Fully Neural Network-based Adaptive Time-Scale Modification Using Sentence-Specific Dynamic Control
Authors: Jaeuk Lee; Sohee Jang; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Adaptive time-scale modification (ATSM) adaptively adjusts audio speed and improves upon previous systems by tailoring the scale for each phoneme in two steps: phoneme positioning via Montreal forced aligner (MFA) and reconstruction with adaptive speaking rate. However, ATSMâs phoneme-specific rate is constant regardless of sentences, and MFA struggles with precise phoneme alignment in synthetic speech. Driven by this, we propose a fully neural networks-based ATSM (Neural ATSM) that dynamically controls each phonemeâs speaking rate to vary from sentence to sentence. It predicts phonemelevel rates using a speaking rate predictor and flexibly modifies the scales to fit sentence context using Gaussian upsampling and attention mechanism, ensuring feature similarity with Softdynamic time warping (DTW) loss. We also integrate a variational autoencoder (VAE) and flow models for enhanced timescaled signals. Experimental results show that Neural ATSM outperforms ATSM for real and synthesized speech.
FLY-TTS: Fast, Lightweight and High-Quality End-to-End Text-to-Speech Synthesis
Authors: Yinlin Guo; Yening Lv; Jinqiao Dou; Yan Zhang; Yuehai Wang
Venue: Interspeech 2024 (main)

Abstract: While recent advances in Text-To-Speech synthesis have yielded remarkable improvements in generating high-quality speech, research on lightweight and fast models is limited. This paper introduces FLY-TTS, a new fast, lightweight and high-quality speech synthesis system based on VITS. Specifically, 1) We replace the decoder with ConvNeXt blocks that generate Fourier spectral coefficients followed by the inverse short-time Fourier transform to synthesize waveforms; 2) To compress the model size, we introduce the grouped parameter-sharing mechanism to the text encoder and flow-based model; 3) We further employ the large pre-trained WavLM model for adversarial training to improve synthesis quality. Experimental results show that our model achieves a real-time factor of 0.0139 on an Intel Core i9 CPU, 8.8x faster than the baseline (0.1221), with a 1.6x parameter compression. Objective and subjective evaluations indicate that FLY-TTS exhibits comparable speech quality to the strong baseline.
Zero-shot Out-of-domain is No Joke: Lessons Learned in the VoiceMOS 2023 MOS Prediction Challenge
Authors: Marie KuneÅ¡ovÃ¡; Jan LeheÄka; Josef MichÃ¡lek; Jindrich Matousek; Jan Å vec
Venue: Interspeech 2024 (main)

Abstract: This paper describes our teamâs experiences in the VoiceMOS Challenge 2023 - a challenge centered around the evaluation of the quality of synthetic or noisy speech. Inspired by our success with an ensemble approach in the first VoiceMOS Challenge in 2022, we submitted an ensemble of four models this time, based on wav2vec 2.0, QuartzNet, CNN-RNN, and LDNet. This was enough to win one of the two tracks we participated in (Track 1b). However, post-challenge analysis shows that only two of the models offer a meaningful contribution in any of the VoiceMOS 2023 tracks, while the other two only degrade the ensembleâs overall performance. On the other hand, post-challenge results on Track 2 (singing voice conversion data) surpassed all our expectations. In the paper, we explain how we tried to deal with the new zero-shot out-of-domain scenarios, analyze the results, and discuss the lessons learned.
Towards a General-Purpose Model of Perceived Pragmatic Similarity
Authors: Nigel G. Ward; Andres Segura; Alejandro Ceballos; Divette Marco
Venue: Interspeech 2024 (main)

Abstract: Models for estimating the similarity between two utterances are fundamental in speech technology.  While fairly good automatic measures exist for semantic similarity, pragmatic similarity has not been previously explored. Using a new collection of thousands of human judgments of the pragmatic similarity between utterance pairs, we train and evaluate various predictive models. The best performing model, which uses 103 features selected from HuBert's 24th layer, correlates on average 0.74 with human judges for the highest-quality data subset, and it sometimes approaches human inter-annotator agreement. We also find evidence for some degree of generality across languages.
Enabling Conversational Speech Synthesis using Noisy Spontaneous Data
Authors: Liisa RÃ¤tsep; Rasmus Lellep; Mark Fishel
Venue: Interspeech 2024 (main)

Abstract: In recent years, the quality of text-to-speech models has increased significantly, but most text-to-speech solutions are trained on datasets of read speech and do not cover conversational speaking styles due to the lack of suitable training data. This paper explores options for creating multi-style speech synthesis using speech recognition datasets that contain samples of spontaneous speech and dialogues but may also include background noise and an insufficient number of samples per speaker. We develop an Estonian multi-speaker TTS system that increases prosodic variability on conversational inputs while still being able to synthesize read speech. We show that our proposed approach can be used to train models that can be controlled to produce conversational speech with little compromise on audio quality. We also highlight a potential multilingual use case to achieve cross-lingual speaker and style transfer to low-resource languages that lack stylistically diverse speech corpora.
Frame-Wise Breath Detection with Self-Training: An Exploration of Enhancing Breath Naturalness in Text-to-Speech
Authors: Dong Yang; Tomoki Koriyama; Yuki Saito
Venue: Interspeech 2024 (main)

Abstract: Developing Text-to-Speech (TTS) systems that can synthesize natural breath is essential for human-like voice agents but requires extensive manual annotation of breath positions in training data. To this end, we propose a self-training method for training a breath detection model that can automatically detect breath positions in speech. Our method trains the model using a large speech corpus and involves: 1) annotation of limited breath sounds utilizing a rule-based approach, and 2) iterative augmentation of these annotations through pseudo-labeling based on the model's predictions. Our detection model employs Conformer blocks with down-/up-sampling layers, enabling accurate frame-wise breath detection. We investigate its effectiveness in multi-speaker TTS using text transcripts with detected breath marks. The results indicate that using our proposed model for breath detection and breath mark insertion synthesizes breath-contained speech more naturally than a baseline model.
H4C-TTS: Leveraging Multi-Modal Historical Context for Conversational Text-to-Speech
Authors: Donghyun Seong; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Conversational text-to-speech (TTS) aims to synthesize natural voices appropriate to a situation by considering the context of past conversations as well as the current text. However, analyzing and modeling the context of a conversation remains challenging. Most conversational TTS use the content of historical and recent conversations without distinguishing between them and often generate speech that does not fit the situation. Hence, we introduce a novel conversational TTS, H4C-TTS, that leverages multi-modal historical context to realize contextually appropriate natural speech synthesis. To facilitate conversational context modeling, we design a context encoder that incorporates historical and recent contexts and a multi-modal encoder that processes textual and acoustic inputs. Experimental results demonstrate that the proposed model significantly improves the naturalness and quality of speech in conversational contexts compared with existing conversational TTS.
Bilingual and Code-switching TTS Enhanced with Denoising Diffusion Model and GAN
Authors: Huai-Zhe Yang; Chia-Ping Chen; Shan-Yun He; Cheng-Ruei Li
Venue: Interspeech 2024 (main)

Abstract: In this paper, we propose a Mandarin-English bilingual and code-switching text-to-speech (TTS) system featuring a diffusion model and generative adversarial network (GAN) to improve the output speech. To address speaker consistency, we employ a feature separation architecture that converts language and speaker IDs into embeddings as input to the encoder. Subsequently, we employ two adversarial classifiers and two classifiers to separate language and speaker features. We integrate a modified diffusion model and discriminators to push for better speech quality and speaker consistency, especially for code-swtiching scenarios. On the MOS measure, the performance of the proposed TTS system differs only slightly from the ground truth data in monolingual speech and achieves MOS of 3.83 in the synthesis of code-switching speech.
SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics
Authors: Takaaki Saeki; Soumi Maiti; Shinnosuke Takamichi; Shinji Watanabe; Hiroshi Saruwatari
Venue: Interspeech 2024 (main)

Abstract: While subjective assessments have been the gold standard for evaluating speech generation, there is a growing need for objective metrics that are highly correlated with human subjective judgments due to their cost efficiency. This paper proposes reference-aware automatic evaluation methods for speech generation inspired by evaluation metrics in natural language processing. The proposed SpeechBERTScore computes the BERTScore for self-supervised dense speech features of the generated and reference speech, which can have different sequential lengths. We also propose SpeechBLEU and SpeechTokenDistance, which are computed on speech discrete tokens. The evaluations on synthesized speech show that our method correlates better with human subjective ratings than mel cepstral distortion and a recent mean opinion score prediction model. Also, they are effective in noisy speech evaluation and have cross-lingual applicability.
UNIQUE : Unsupervised Network for Integrated Speech Quality Evaluation
Authors: Juhwan Yoon; WooSeok Ko; Seyun Um; Sungwoong Hwang; Soojoong Hwang; Changhwan Kim; Hong-Goo Kang
Venue: Interspeech 2024 (main)

Abstract: The significance of an objective metric for evaluating synthetic speech lies in its ability to provide a quantitative measure for systematic assessment of speech quality. However, previous works have focused on predicting subjective quality scores in a supervised manner, requiring a large amount of paired data comprising speech and perceived quality scores. In this work, we introduce a novel metric, the UNIQUE score, that integrates the concept of anomaly detection to systematically evaluate input speech in an unsupervised manner. By leveraging speech features from a self-supervised model, the system can learn a sophisticated speech distribution that enables it to detect differences between real and synthesized speech. By comparing the UNIQUE score of synthetic speech across various text-to-speech models and datasets with other objective measures, we demonstrate that our metric provides an effective evaluation of speech quality that shows a higher correlation with human perceptions.
FVTTS : Face Based Voice Synthesis for Text-to-Speech
Authors: Minyoung Lee; Eunil Park; Sungeun Hong
Venue: Interspeech 2024 (main)

Abstract: A face is expressive of individual identity and used in various studies such as identification, authentication, and personalization. Similarly, a voice is a means of expressing individuals, and personalized voice synthesis based on voice reference is active. However, the voice-based method confronts voice sample dependency limitations. We propose Face-based Voice synthesis for Text-To-Speech (FVTTS) to synthesize voice from face images that are more expressive of personal identity than voice samples. A major challenge in face-based TTS methods is extracting distinct voice features highly related to voice from the face image. Our face encoder is designed to tackle this by integrating global facial attributes with voice-related features to represent personalized characteristics. FVTTS has shown superiority in various metrics and adaptability across different data domains. We establish a new standard in face-based TTS, leading the way in personalized voice synthesis.
Meta Learning Text-to-Speech Synthesis in over 7000 Languages
Authors: Florian Lux; Sarina Meyer; Lyonel Behringer; Frank Zalkow; Phat Do; Matt Coler; EmanuÃ«l A. P. Habets; Ngoc Thang Vu
Venue: Interspeech 2024 (main)

Abstract: In this work, we take on the challenging task of building a single text-to-speech synthesis system that is capable of generating speech in over 7000 languages, many of which lack sufficient data for traditional TTS development. By leveraging a novel integration of massively multilingual pretraining and meta learning to approximate language representations, our approach enables zero-shot speech synthesis in languages without any available data. We validate our system's performance through objective measures and human evaluation across a diverse linguistic landscape. By releasing our code and models publicly, we aim to empower communities with limited linguistic resources and foster further innovation in the field of speech technology.
An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios
Authors: Cheng Gong; Erica Cooper; Xin Wang; Chunyu Qiang; Mengzhe Geng; Dan Wells; Longbiao Wang; Jianwu Dang; Marc Tessier; Aidan Pine; Korin Richmond; Junichi Yamagishi
Venue: Interspeech 2024 (main)

Abstract: Self-supervised learning (SSL) representations from massively multilingual models offer a promising solution for low-resource language speech tasks. Despite advancements, language adaptation in TTS systems remains an open problem. This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system proposed in our previous work. We conducted experiments on 12 languages using limited data with various fine-tuning configurations. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target languageâs adaptation performance. Additionally, we find that the fine-tuning dataset size and number of speakers influence adaptability. Surprisingly, we also observed that using paired data for fine-tuning is not always optimal compared to audio-only data. Beyond speech intelligibility, our analysis covers speaker similarity, language identification, and predicted MOS.
Improving Multilingual Text-to-Speech with Mixture-of-Language-Experts and Accent Disentanglement
Authors: Jing Wu; Ting Chen; Minchuan Chen; Wei Hu; Shaojun Wang; Jing Xiao
Venue: Interspeech 2024 (main)

Abstract: Code-switching and accent control is particularly valuable in multilingual text-to-speech (TTS) systems as both of them contribute to improving the authenticity and comprehensibility. However, the issues of seamless integration of languages within a single utterance and the thorough disentanglement of different attributes without bilingual data remains to be solved. To conquer these problems, a computation-efficient model is proposed in this paper. Firstly, the Mixture of Language Experts (MoLE) module is introduced as the encoder to extract language-specific features and fuse intra-utterance semantic information. Secondly, embedding methods together with several regularization strategies and speaker consistency constraints are utilized to ensure that the generated speech aligns with the desired accent. Experiments show that the proposed model can improve the performance of code-switching accent-controllable multilingual TTS over the baseline model in terms of fluency and naturalness.
Seamless Language Expansion: Enhancing Multilingual Mastery in Self-Supervised Models
Authors: Jing Xu; Minglin Wu; Xixin Wu; Helen Meng
Venue: Interspeech 2024 (main)

Abstract: Self-supervised (SSL) models have shown great performance in various downstream tasks. However, they are typically developed for limited languages, and may encounter new languages in real-world. Developing a SSL model for each new language is costly. Thus, it is vital to figure out how to efficiently adapt existed SSL models to a new language without impairing its original abilities. We propose adaptation methods which integrate LoRA to existed SSL models to extend new language. We also develop preservation strategies which include data combination and re-clustering to retain abilities on existed languages. Applied to mHuBERT, we investigate their effectiveness on speech re-synthesis task. Experiments show that our adaptation methods enable mHuBERT to be applied to a new language (Mandarin) with MOS value increased about 1.6 and the relative value of WER reduced up to 61.72%. Also, our preservation strategies ensure that the performance on both existed and new languages remains intact.
XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model
Authors: Edresson Casanova; Kelly Davis; Eren GÃ¶lge; GÃ¶rkem GÃ¶knar; Iulian Gulea; Logan Hart; Aya Aljafari; Joshua Meyer; Reuben Morais; Samuel Olayemi; Julian Weber
Venue: Interspeech 2024 (main)

Abstract: Most Zero-shot Multi-speaker TTS (ZS-TTS) systems support only a single language. Although models like YourTTS, VALL-E X, Mega-TTS 2, and Voicebox explored  Multilingual ZS-TTS they are limited to just a few high/medium resource languages, limiting the applications of these models in most of the low/medium resource languages. In this paper, we aim to alleviate this issue by proposing and making publicly available the XTTS system. Our method builds upon the Tortoise model and adds several novel modifications to enable multilingual training, improve voice cloning, and enable faster training and inference. XTTS was trained in 16 languages and achieved state-of-the-art (SOTA) results in most of them.