GenAI4ELab/papercli-papers huggingface dataset:
HybridVC: Efficient Voice Style Conversion with Text and Audio Prompts
Authors: Xinlei Niu; Jing Zhang; Charles Patrick Martin
Venue: Interspeech 2024 (main)

Abstract: We introduce HybridVC, a voice conversion (VC) framework built upon a pre-trained conditional variational autoencoder (CVAE) that combines the strengths of a latent model with contrastive learning. HybridVC supports text and audio prompts, enabling more flexible voice style conversion. HybridVC models a latent distribution conditioned on speaker embeddings acquired by a pretrained speaker encoder and optimises style text embeddings to align with the speaker style information through contrastive learning in parallel. Therefore, HybridVC can be efficiently trained under limited computational resources. Our experiments demonstrate HybridVC's superior training efficiency and its capability for advanced multi-modal voice style conversion. This underscores its potential for widespread applications such as user-defined personalised voice in various social media platforms. A comprehensive ablation study further validates the effectiveness of our method.
DreamVoice: Text-Guided Voice Conversion
Authors: Jiarui Hai; Karan Thakkar; Helin Wang; Zengyi Qin; Mounya Elhilali
Venue: Interspeech 2024 (main)

Abstract: Generative voice technologies are rapidly evolving, offering opportunities for more personalized and inclusive experiences. Traditional one-shot voice conversion (VC) requires a target recording during inference, limiting ease of usage in generating desired voice timbres. Text-guided generation offers an intuitive solution to convert voices to desired "DreamVoices" according to the users' needs. Our paper presents two major contributions to VC technology: (1) DreamVoiceDB, a robust dataset of voice timbre annotations for 900 speakers from VCTK and LibriTTS. (2) Two text-guided VC methods: DreamVC, an end-to-end diffusion-based text-guided VC model; and DreamVG, a versatile text-to-voice generation plugin that can be combined with any one-shot VC models. The experimental results demonstrate that our proposed methods trained on the DreamVoiceDB dataset generate voice timbres accurately aligned with the text prompt and achieve high-quality VC.
Hear Your Face: Face-based voice conversion with F0 estimation
Authors: Jaejun Lee; Yoori Oh; Injune Hwang; Kyogu Lee
Venue: Interspeech 2024 (main)

Abstract: This paper delves into the emerging field of face-based voice conversion, leveraging the unique relationship between an individual's facial features and their vocal characteristics. We present a novel face-based voice conversion framework that particularly utilizes the average fundamental frequency of the target speaker, derived solely from their facial images. Through extensive analysis, our framework demonstrates superior speech generation quality and the ability to align facial features with voice characteristics, including tracking of the target speaker's fundamental frequency.
Accent Conversion with Articulatory Representations
Authors: Yashish M. Siriwardena; Nathan Swedlow; Audrey Howard; Evan Gitterman; Dan Darcy; Carol Espy-Wilson; Andrea Fanelli
Venue: Interspeech 2024 (main)

Abstract: Conversion of non-native accented speech to native (American) English has a wide range of applications such as improving intelligibility of non-native speech. Previous work on this domain has used phonetic posteriograms as the target speech representation to train an acoustic model which is then used to extract a compact representation of input speech for accent conversion. In this work, we introduce the idea of using an effective articulatory speech representation, extracted from an acoustic-to-articulatory speech inversion system, to improve the acoustic model used in accent conversion. The idea to incorporate articulatory representations originates from their ability to well characterize accents in speech. To incorporate articulatory representations with conventional phonetic posteriograms, a multi-task learning based acoustic model is proposed. Objective and subjective evaluations show that the use of articulatory representations can improve the effectiveness of accent conversion.
USD-AC: Unsupervised Speech Disentanglement for Accent Conversion
Authors: Jen-Hung Huang; Wei-Tsung Lee; Chung-Hsien Wu
Venue: Interspeech 2024 (main)

Abstract: This study proposes USD-AC, an innovative Unsupervised Speech Disentanglement Accent Conversion that does not require parallel data and text transcription for training, solving challenges such as limited labeled data and generalizability issues. USD-AC, grounded in speech decomposition, aims to separate accent features from linguistic content, enhancing its adaptability across various accent conversion tasks. It utilizes a pre-trained ASR model to extract linguistic content and incorporates accent embedding for accent representation. Adversarial training effectively disentangles accent information from other attributes, boosting conversion performance. USD-AC achieves remarkable outcomes for known speakers and accents and exhibits exceptional generalization to unseen speakers, accents, and content. Through experimental comparison, USDAC based on unsupervised learning has shown superiority and generalization ability compared to supervised learning methods.
Knowledge Distillation from Self-Supervised Representation Learning Model with Discrete Speech Units for Any-to-Any Streaming Voice Conversion
Authors: Hiroki Kanagawa; Yusuke Ijima
Venue: Interspeech 2024 (main)

Abstract: SSL models like HuBERT and WavLM serve as effective content encoders for non-parallel voice conversion (VC), but their large size and design for offline operation make streaming use a challenge. Thus, we derive novel lightweight streaming VC using knowledge distillation (KD) from the SSL model. A promising SSL model and its vector quantizer are used as the teacher content encoder. The student content encoder predicts discrete content from the teacher, ensuring consistency within the KD framework. To stabilize the converted speech's prosody, a prosody predictor using content and speaker information is employed. A HiFi-GAN-like decoder generates waveforms from speaker, content, and prosody inputs. Our student VC leverages the SSL model's robust content encoding without relying on it for inferencing, enabling streaming operation. Evaluations on any-to-any VC tasks show our approach achieved naturalness comparable to modern offline VCs and the teacher with SSL model while being streamable.
SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models
Authors: Dongchao Yang; Dingdong Wang; Haohan Guo; Xueyuan Chen; Xixin Wu; Helen Meng
Venue: Interspeech 2024 (main)

Abstract: In this study, we propose a simple and efficient Non-
Autoregressive (NAR) text-to-speech (TTS) system based on
diffusion, named SimpleSpeech. Its simpleness shows in three
aspects: (1) It can be trained on the speech-only dataset, without
any alignment information; (2) It directly takes plain text as
input and generates speech through an NAR way; (3) It tries to
model speech in a finite and compact latent space, which alleviates 
the modeling difficulty of diffusion. More specifically, we
propose a novel speech codec model (SQ-Codec) with scalar
quantization, SQ-Codec effectively maps the complex speech
 signal into a finite and compact latent space, named scalar latent
space. Benefits from SQ-Codec, we apply a novel transformer 
diffusion model in the scalar latent space of SQ-Codec.
We train SimpleSpeech on 4k hours of a speech-only dataset,
it shows natural prosody and voice cloning ability. Compared
 with previous large-scale TTS models, it presents significant
 speech quality and generation speed improvement. Demos are
 released.
Sample-Efficient Diffusion for Text-To-Speech Synthesis
Authors: Justin Lovelace; Soham Ray; Kwangyoun Kim; Kilian Q. Weinberger; Felix Wu
Venue: Interspeech 2024 (main)

Abstract: This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion. It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder. Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech â far less than current state-of-the-art systems. In fact, it synthesizes more intelligible speech than thestate-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data. Our implementation is available at https://github.com/justinlovelace/SESD.
Exploring the Robustness of Text-to-Speech Synthesis Based on Diffusion Probabilistic Models to Heavily Noisy Transcriptions
Authors: Jingyi Feng; Yusuke Yasuda; Tomoki Toda
Venue: Interspeech 2024 (main)

Abstract: Large data volumes can benefit text-to-speech (TTS), but speech data with high-quality annotation is limited. Automatic transcription enables the transcription of found speech data to enhance the data volume for TTS, but TTS training suffers from transcription errors. In this paper, we investigate the robustness of typical TTS models against heavily noisy transcripts, including diffusion, flow, and autoregressive-based TTS models, in terms of objective intelligibility and subjective naturalness. Our experimental results show that diffusion-based TTS is extremely robust to heavily noisy transcriptions, mitigating about 30% of the word error rate compared to autoregressive and flow-based models. We also show that iterative inference with a long diffusion time is key to the robustness of diffusionbased TTS based on likelihood analysis.
VoiceTailor: Lightweight Plug-In Adapter for Diffusion-Based Personalized Text-to-Speech
Authors: Heeseung Kim; Sang-gil Lee; Jiheum Yeom; Che Hyun Lee; Sungwon Kim; Sungroh Yoon
Venue: Interspeech 2024 (main)

Abstract: We propose VoiceTailor, a parameter-efficient speaker-adaptive text-to-speech (TTS) system, by equipping a pre-trained diffusion-based TTS model with a personalized adapter. VoiceTailor identifies pivotal modules that benefit from the adapter based on a weight change ratio analysis. We utilize Low-Rank Adaptation (LoRA) as a parameter-efficient adaptation method and incorporate the adapter into pivotal modules of the pre-trained diffusion decoder. To achieve powerful adaptation performance with few parameters, we explore various guidance techniques for speaker adaptation and investigate the best strategies to strengthen speaker information. VoiceTailor demonstrates comparable speaker adaptation performance to existing adaptive TTS models by fine-tuning only 0.25% of the total parameters. VoiceTailor shows strong robustness when adapting to a wide range of real-world speakers, as shown in the demo.
PitchFlow: adding pitch control to a Flow-matching based TTS model
Authors: Tasnima Sadekova; Mikhail Kudinov; Vadim Popov; Assel Yermekova; Artem Khrapov
Venue: Interspeech 2024 (main)

Abstract: In recent years, there have been various attempts to improve denoising diffusion probabilistic models and make them more suitable for real-world applications. One of the recent advances in this research direction is a flow-matching models framework which has already shown good results in image and speech generation tasks. Despite high quality and generation speed, flow-matching text-to-speech models still have problems with stability and control. To mitigate this issue, we propose two techniques: speaker scoring and pitch guidance allowing to control timbre and pitch contour of the generated speech. We show that the optimal choice of the prior leads to considerable improvement of similarity and a specific design of classifier guidance allows for fine-grained pitch control with high naturalness. Moreover, these techniques may be used to implement a voice conversion system of a competitive quality.
DualSpeech: Enhancing Speaker-Fidelity and Text-Intelligibility Through Dual Classifier-Free Guidance
Authors: Jinhyeok Yang; Junhyeok Lee; Hyeong-Seok Choi; Seunghoon Ji; Hyeongju Kim; Juheon Lee
Venue: Interspeech 2024 (main)

Abstract: Text-to-Speech (TTS) models have advanced significantly, aiming to accurately replicate human speech's diversity, including unique speaker identities and linguistic nuances. Despite these advancements, achieving an optimal balance between speaker-fidelity and text-intelligibility remains a challenge, particularly when diverse control demands are considered. Addressing this, we introduce DualSpeech, a TTS model that integrates phoneme-level latent diffusion with dual classifier-free guidance. This approach enables exceptional control over speaker-fidelity and text-intelligibility. Experimental results demonstrate that by utilizing the sophisticated control, DualSpeech surpasses existing state-of-the-art TTS models in performance. Demos are available at https://bit.ly/48Ewoib.
Generating Speakers by Prompting Listener Impressions for Pre-trained Multi-Speaker Text-to-Speech Systems
Authors: Zhengyang Chen; Xuechen Liu; Erica Cooper; Junichi Yamagishi; Yanmin Qian
Venue: Interspeech 2024 (main)

Abstract: This paper proposes a speech synthesis system that allows users to specify and control the acoustic characteristics of a speaker by means of prompts describing the speaker's traits of synthesized speech. Unlike previous approaches, our method utilizes listener impressions to construct prompts, which are easier to collect and align more naturally with everyday descriptions of speaker traits. We adopt the Lowrank Adaptation (LoRA) technique to swiftly tailor a pretrained language model to our needs, facilitating the extraction of speakerrelated traits from the prompt text. Besides, different from other promptdriven texttospeech (TTS) systems, we separate the prompttospeaker module from the multispeaker TTS system, enhancing system flexibility and compatibility with various pretrained multispeaker TTS systems. Moreover, for the prompt-to-speaker characteristic module, we also compared the discriminative method and flow-matching based generative method and we found that combining both methods can help the system simultaneously capture speaker-related information from prompts better and generate speech with higher fidelity.
TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizers
Authors: Yakun Song; Zhuo Chen; Xiaofei Wang; Ziyang Ma; Guanrou Yang; Xie Chen
Venue: Interspeech 2024 (main)

Abstract: Neural codec language model (LM) has demonstrated strong capability in zero-shot text-to-speech (TTS) synthesis. However, the codec LM often suffers from limitations in inference speed and stability, due to its auto-regressive nature and implicit alignment between text and audio. In this work, to handle these challenges, we introduce a new variant of neural codec LM, namely TacoLM. Specifically, TacoLM introduces a gated attention mechanism to improve the training and inference efficiency and reduce the model size. Meanwhile, an additional gated cross-attention layer is included for each decoder layer, which improves the efficiency and content accuracy of the synthesized speech. In the evaluation of the Librispeech corpus, the proposed TacoLM achieves a better word error rate, speaker similarity, and mean opinion score, with 90% fewer parameters and 5.2 times speed up, compared with VALL-E.  Demo and code is available at https://ereboas.github.io/TacoLM/.
Anonymising Elderly and Pathological Speech: Voice Conversion Using DDSP and Query-by-Example
Authors: Suhita Ghosh; Melanie Jouaiti; Arnab Das; Yamini Sinha; Tim Polzehl; Ingo Siegert; Sebastian Stober
Venue: Interspeech 2024 (main)

Abstract: Speech anonymisation aims to protect speaker identity by changing personal identifiers in speech while retaining linguistic content. Current methods fail to retain prosody and unique speech patterns found in elderly and pathological speech domains, which is essential for remote health monitoring. To address this gap, we propose a voice conversion-based method (DDSP-QbE) using differentiable digital signal processing and query-by-example. The proposed method, trained with novel losses, aids in disentangling linguistic, prosodic, and domain representations, enabling the model to adapt to uncommon speech patterns. Objective and subjective evaluations show that DDSP-QbE significantly outperforms the voice conversion state-of-the-art concerning intelligibility, prosody, and domain preservation across diverse datasets, pathologies, and speakers while maintaining quality and speaker anonymity. Experts validate domain preservation by analysing twelve clinically pertinent domain attributes.
Asynchronous Voice Anonymization Using Adversarial Perturbation On Speaker Embedding
Authors: Rui Wang; Liping Chen; Kong Aik Lee; Zhen-Hua Ling
Venue: Interspeech 2024 (main)

Abstract: Voice anonymization has been developed as a technique for preserving privacy by replacing the speaker's voice in a speech signal with that of a pseudo-speaker, thereby obscuring the original voice attributes from machine recognition and human perception. In this paper, we focus on altering the voice attributes against machine recognition while retaining human perception. We referred to this as the asynchronous voice anonymization. To this end, a speech generation framework incorporating a speaker disentanglement mechanism is employed to generate the anonymized speech. The speaker attributes are altered through adversarial perturbation applied on the speaker embedding, while human perception is preserved by controlling the intensity of perturbation. Experiments conducted on the LibriSpeech dataset showed that the speaker attributes were obscured with their human perception preserved for 60.71% of the processed utterances. Audio samples can be found in .
Probing the Feasibility of Multilingual Speaker Anonymization
Authors: Sarina Meyer; Florian Lux; Ngoc Thang Vu
Venue: Interspeech 2024 (main)

Abstract: In speaker anonymization, speech recordings are modified in a way that the identity of the speaker remains hidden. While this technology could help to protect the privacy of individuals around the globe, current research restricts this by focusing almost exclusively on English data. In this study, we extend a state-of-the-art anonymization system to nine languages by transforming language-dependent components to their multilingual counterparts. Experiments testing the robustness of the anonymized speech against privacy attacks and speech deterioration show an overall success of this system for all languages. The results suggest that speaker embeddings trained on English data can be applied across languages, and that the anonymization performance for a language is mainly affected by the quality of the speech synthesis component used for it.
DiffVC+: Improving Diffusion-based Voice Conversion for Speaker Anonymization
Authors: Fan Huang; Kun Zeng; Wei Zhu
Venue: Interspeech 2024 (main)

Abstract: The increasing risks of speech data leakage prompt growing concerns about voice privacy. This paper proposes DiffVC+, a speaker anonymization model designed to preserve speech privacy. It operates as a diffusion-based voice conversion model that suppresses identity information by converting the speaker's voice through flexible approaches. DiffVC+ comprises a self-supervised learning (SSL) content encoder that effectively extracts the source speech content, a speaker encoder and an embedding generator that both supply the target speaker embedding, and a diffusion-based decoder generating the converted speech. Furthermore, we propose DiffVC+ light and DiffVC+ decoupled for edge-side and server-side deployments, respectively. Experimental results demonstrate that our models significantly outperform the baseline in terms of the intelligibility and naturalness of the converted speech, while achieving competitive anonymization performance.
Learning from Back Chunks: Acquiring More Future Knowledge for Streaming ASR Models via Self Distillation
Authors: Yuting Yang; Guodong Ma; Yuke Li; Binbin Du; Haoqi Zhu; Liang Ruan
Venue: Interspeech 2024 (main)

Abstract: The performance of streaming automatic speech recognition (ASR) is often inferior to that of non-streaming speech recognition due to the absence of complete contextual information. However, we cannot optimize the model by merely accessing more future frames, as this would lead to considerable latency. In this paper, we propose Future-aware Transformer (FaT) that models long-distance future contextual dependencies by transferring information from later chunks to former chunks through look-ahead windows. Specifically, the chunk-based context is used to encode audio sequence features. On this basis, the look-ahead window provides more context information for each chunk and acts as a bridge to progressively transfer long-distance future information from later chunks to earlier ones via a future-aware distillation mechanism. Experiments on AISHELL-1 and AISHELL-2 demonstrate that the proposed method achieves superior accuracy and better streaming latency than several strong baselines.
Decoder-only Architecture for Streaming End-to-end Speech Recognition
Authors: Emiru Tsunoo; Hayato Futami; Yosuke Kashiwagi; Siddhant Arora; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: Decoder-only language models (LMs) have been successfully adopted for speech-processing tasks including automatic speech recognition (ASR). The LMs have ample expressiveness and perform efficiently. This efficiency is a suitable characteristic for streaming applications of ASR. In this work, we propose to use a decoder-only architecture for blockwise streaming ASR. In our approach, speech features are compressed using CTC output and context embedding using blockwise speech subnetwork, and are sequentially provided as prompts to the decoder. The decoder estimates the output tokens promptly at each block. To this end, we also propose a novel training scheme using random-length prefix prompts to make the model robust to the truncated prompts caused by blockwise processing. An experimental comparison shows that our proposal decoder-only streaming ASR achieves 8% relative word error rate reduction in the LibriSpeech test-other set while being twice as fast as the baseline model.
Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech Units: A Pilot Study
Authors: Peikun Chen; Sining Sun; Changhao Shan; Qing Yang; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: Unified speech-text models like SpeechGPT, VioLA, and AudioPaLM have shown impressive performance across various speech-related tasks, especially in Automatic Speech Recognition (ASR). These models typically adopt a unified method to model discrete speech and text tokens, followed by training a decoder-only transformer. However, they are all designed for non-streaming ASR tasks, where the entire speech utterance is needed during decoding. Hence, we introduce a decoder-only model exclusively designed for streaming recognition, incorporating a dedicated boundary token to facilitate streaming recognition and employing causal attention masking during the training phase. Furthermore, we introduce right-chunk attention and various data augmentation techniques to improve the modelâs contextual modeling abilities. While achieving streaming speech recognition, experiments on the AISHELL-1 and-2 datasets demonstrate the competitive performance of our streaming approach with non-streaming decoder-only counter-parts. The code we used for this work can be found here.