Guided-TTS:Text-to-Speech with Untranscribed Speech

1 INTRODUCTION . Neural text-to-speech ( TTS ) models have been achieved to generate high-quality human-like speech given text ( van den Oord et al . ( 2016 ) ; Shen et al . ( 2018 ) ) . In general , these TTS models are conditional generative models that encode text into the hidden representation and generate speech from the encoded representation . Early TTS models are autoregressive generative models which generate high-quality speech but suffer from slow synthesis speed due to the sequential sampling procedure ( Shen et al . ( 2018 ) ; Li et al . ( 2019 ) ) . Owing to the development of non-autoregressive generative models , recent TTS models are capable of generating high-quality speech with faster inference speed ( Ren et al . ( 2019 ) ; Ren et al . ( 2021 ) ; Kim et al . ( 2020 ) ; Popov et al . ( 2021 ) ) . Recently , high-quality end-to-end TTS models have been proposed that generate raw waveform from the text at once ( Kim et al . ( 2021 ) ; Weiss et al . ( 2021 ) ; Chen et al . ( 2021b ) ) . Despite the high-quality and fast inference speed of speech synthesis , most TTS models can be trained only if the transcribed data of the desired speaker is given . While long-form untranscribed data , such as audiobooks or podcasts , is available on various websites , it is challenging to use these unpaired speech data to train existing TTS models . To utilize these untranscribed data , long-form untranscribed speech data has to be segmented into sentence-level , and then each segmented speech should be transcribed accurately . Since the existing TTS models must directly model the conditional distribution of speech given text , the direct usage of untranscribed data remains challenging to solve . In this work , we propose Guided-TTS , an unconditional diffusion-based generative model trained on untranscribed data that leverages a phoneme classifier for text-to-speech synthesis . Trained on untranscribed speech data for the desired speaker , our unconditional diffusion probabilistic model learns to generate mel-spectrograms of the speaker without any context . As training data does not have to be aligned with text sequence for unconditional speech modeling , we simply use random chunks of untranscribed speech to train our unconditional generative model . This allows us to build training data without extra effort in modeling the speech of speakers for which only long-form untranscribed speech data is available . To guide the unconditional DDPM for TTS , we train a framewise phoneme classifier on transcribed data and use the gradients of the classifier during sampling . Although our unconditional generative model is trained without any transcript , Guided-TTS effectively generates mel-spectrograms given the transcript by guiding the generative process of unconditional DDPM with the phoneme classifier . We demonstrate that the proposed method , TTS by guiding the unconditional DDPM , matches the performance of the existing conditional TTS models on LJSpeech . We further show that by training the phoneme classifier on multi-speaker paired dataset , Guided-TTS also shows comparable performance without seeing any transcript of LJSpeech , which shows the possibility to build a high-quality text-to-speech model without a transcript for the desired speaker . We encourage the readers to listen to samples of Guided-TTS trained on various untranscribed datasets on our demo page . 1 2 BACKGROUND . 2.1 DENOISING DIFFUSION PROBABLISTIC MODELS ( DDPM ) AND ITS VARIANT . DDPM ( Ho et al . ( 2020 ) ) , which is recently proposed as a kind of probabilistic generative model , has been applied to various domains such as image ( Dhariwal & Nichol ( 2021 ) ) and audio ( Chen et al . ( 2021a ) ; Popov et al . ( 2021 ) ) . DDPM first defines a forward process that gradually corrupts dataX0 to a random noiseXT across T timesteps . The model learns the reverse process that follows the reverse trajectory of the predefined forward process to generate data from random noise . Recently , there have been approaches to formulate the trajectory between data and noise as a continuous stochastic differential equation ( SDE ) instead of a discrete-time Markov process ( Song et al . ( 2021b ) ) . Grad-TTS ( Popov et al . ( 2021 ) ) introduces SDE formulation to TTS , which we have followed and used . According to the formulation of Grad-TTS , the forward process that corrupts data X0 into the standard Gaussian noise XT is as follows : dXt = − 1 2 Xtβtdt+ √ βtdWt , ( 1 ) where βt is a predefined noise schedule , βt = β0+ ( βT−β0 ) t , andWt is a Wiener process . Anderson ( 1982 ) showed that the reverse process , which represents the trajectory from noise XT to X0 , can be formulated in SDE , which is defined as follows : dXt = ( − 1 2 Xt −∇Xt log pt ( Xt ) ) βtdt+ √ βtdW̃t , ( 2 ) where W̃t is a reverse time Wiener process . Given the score , the gradient of log density with respect to data ( i.e. , ∇Xt log pt ( Xt ) ) , for t ∈ [ 0 , T ] , we can sample data X0 from random noise XT by solving Eq . ( 2 ) . To generate data , the DDPM learns to estimate the score with the neural network sθ parameterized by θ . To estimate the score , Xt is sampled from the distribution derived from Eq . ( 1 ) given dataX0 , which is as follows : Xt|X0 ∼ N ( ρ ( X0 , t ) , λ ( t ) ) , ( 3 ) where ρ ( X0 , t ) = e− 1 2 ∫ t 0 βsdsX0 , and λ ( t ) = I−e− ∫ t 0 βsds . Then , the score can be derived from Eq . ( 3 ) ; ∇Xt log pt ( Xt|X0 ) = −λ ( t ) −1 t , given X0 ( Popov et al . ( 2021 ) ) . To train the model sθ ( Xt , t ) for ∀t ∈ [ 0 , T ] , the following loss is used : L ( θ ) = EtEX0E t [ ∥∥sθ ( Xt , t ) + λ ( t ) −1 t∥∥22 ] ( 4 ) which is L2 loss as in previous works ( Ho et al . ( 2020 ) , Song et al . ( 2021b ) ) . Using the model sθ ( Xt , t ) , we can generate sample X0 from noise by solving Eq . ( 2 ) . Grad-TTS generates data X0 from XT by setting T = 1 and using a fixed discretization strategy ( Song et al . ( 2021b ) ) : Xt− 1N = Xt + βt N ( 1 2 Xt +∇Xt log pt ( Xt ) ) + √ βt N zt ( 5 ) whereN is the number of steps to solve SDE , t ∈ { 1N , 2 N , ... , 1 } and zt is a standard Gaussian noise . 1Demo : https : //bit.ly/3oWhVJg 2.2 CLASSIFIER GUIDANCE . DDPM can be guided to generate samples with the desired condition without fine-tuning with the introduction of a classifier . Song et al . ( 2021b ) used the unconditional DDPM to generate classconditional images using a separately trained image classifier . Not only unconditional DDPM but also conditional DDPM can be guided using the classifier , which contributes to achieving state-ofthe-art performance for class-conditional image generation ( Dhariwal & Nichol ( 2021 ) ) . For conditional generation , the classifier pt ( y|Xt ) is trained to classify the noisy data Xt as the condition y . If Eq . ( 5 ) is modified , discretized SDE for the conditional generation can be obtained . Xt− 1N = Xt + βt N ( 1 2 Xt +∇Xt log pt ( Xt|y ) ) + √ βt N zt ( 6 ) ∇Xt log pt ( Xt|y ) = ∇Xt log pt ( Xt ) +∇Xt log pt ( y|Xt ) ( 7 ) If the unconditional score and the classifier for the condition are given , the sample X0 with the condition y can be generated using Eq . ( 6 ) . 3 GUIDED-TTS . In this section , we present Guided-TTS , which aims to build a high-quality text-to-speech model without the transcript of the target speaker . While other TTS models directly learn to generate speech from text , Guided-TTS models the unconditional distribution of speech to utilize speech-only data and guides the unconditional model to generate speech with a given text . To the best of our knowledge , Guided-TTS is the first TTS model that generates speech with the unconditional generative model . Both unconditional speech modeling and controllable generation with the unconditional generative model are well known to be challenging . To tackle these challenges , we adopt a diffusion-based generative model for unconditional speech generation , which has the advantages of modeling complex distributions and easy controllability . Additionally , we introduce a phoneme classifier to guide the unconditional DDPM for TTS , which requires transcribed data for training . In Guided-TTS , the generative model and the phoneme classifier are trained separately so that we can utilize different datasets to train each module . For instance , assume only the untranscribed speech data of the target speaker for TTS is available . With Guided-TTS , the phoneme classifier can still be trained on other transcribed datasets that contain rich , large-scale multi-speaker data . The phoneme classifier trained in such a manner effectively guides the generative model , trained only using the untranscribed speech data of the target speaker . By doing so , with the two modules independent of each other , we can achieve the text-to-speech model without any transcript of the target speaker . Guided-TTS consists of 4 modules : the unconditional DDPM , the phoneme classifier , the duration predictor , and the speaker encoder , as shown in Fig . 1 . Unconditional DDPM is a module that learns to generate mel-spectrogram unconditionally , and the remaining three modules are for TTS synthesis by guidance . We will explain the unconditional DDPM in Section 3.1 , followed by the method of guiding the unconditional model for TTS in Section 3.2 . 3.1 UNCONDITIONAL DDPM . Our unconditional DDPM models the unconditional distribution of speech PX without any transcript . We assume that the training data for the diffusion-based model has tens of hours of untranscribed speech data from the target speaker S for TTS . As our generative model learns only with speech data , training samples do not need to be aligned with the text . Thus , we use random chunks of untranscribed speech as training data to reduce the burden of not only speech transcription but also segmentation when only the long-form speech data is available for the target speaker . Given a mel-spectrogram X = X0 , we define a forward process as in Eq . ( 1 ) , which gradually corrupts data into noise , and approximate the reverse process in Eq . ( 2 ) by estimating the unconditional score ∇Xt log p ( Xt ) for each timestep t. At each iteration , Xt , t ∈ [ 0 , 1 ] is sampled from the mel-spectrogram X0 as in Eq . ( 3 ) , and score is estimated with the neural network sθ ( Xt , t ) parameterized by θ . The training objective of our unconditional model is in Eq . ( 4 ) . Similar to Grad-TTS ( Popov et al . ( 2021 ) ) , we regard mel-spectrogram as a 2D image with a single channel and use the U-Net architecture ( Ronneberger et al . ( 2015 ) ) as sθ . We use the same size of the architecture used to model 32× 32 sized images in Ho et al . ( 2020 ) to capture long-term dependencies without any text information , while Grad-TTS uses smaller architecture for the conditional distribution modeling .

Summary: This paper borrows the backbone from Grad-TTS [1], building an unconditional diffusion based model on speech input. With a phone classifier and duration predictor trained on transcribed data, and a speaker encoder trained on labeled speaker dataset, the model was able to synthesize speech on given text. Experimental results shows it's comparable with Grad-TTS [1] trained on transcribed data (by conditioning on the text during training). Ablation shows it can generalize to a diversified dataset.    [1] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8599–8608. PMLR, 2021a.