Guided-TTS:Text-to-Speech with Untranscribed Speech
Authors: Heeseung Kim
Sungwon Kim
Sungroh Yoon
Venue: ICLR 2022 Submitted (2022)
Keywords: Text-to-Speech
Speech Synthesis
DDPM
TTS
Untranscribed speech

Abstract: Most neural text-to-speech (TTS) models require $\langle$speech, transcript$\rangle$ paired data from the desired speaker for high-quality speech synthesis, which limits the usage of large amounts of untranscribed data for training. In this work, we present Guided-TTS, a high-quality TTS model that learns to generate speech from untranscribed speech data. Guided-TTS combines an unconditional diffusion probabilistic model with a separately trained phoneme classifier for text-to-speech. By modeling the unconditional distribution for speech, our model can utilize the untranscribed data for training. For text-to-speech synthesis, we guide the generative process of the unconditional DDPM via phoneme classification to produce mel-spectrograms from the conditional distribution given transcript. We show that Guided-TTS achieves comparable performance with the existing methods without any transcript for LJSpeech. Our results further show that a single speaker-dependent phoneme classifier trained on multispeaker large-scale data can guide unconditional DDPMs for various speakers to perform TTS.

Under review as a conference paper at ICLR 2022
GUIDED-TTS: TEXT-TO-SPEECH WITH UNTRAN- SCRIBED SPEECH
Anonymous authors Paper under double-blind review
ABSTRACT
Most neural text-to-speech (TTS) models require (speech, transcript) paired data from the desired speaker for high-quality speech synthesis, which limits the us- age of large amounts of untranscribed data for training. In this work, we present Guided-TTS, a high-quality TTS model that learns to generate speech from un- transcribed speech data. Guided-TT'S combines an unconditional diffusion prob- abilistic model with separately trained phoneme classifier for text-to -pteth. By modeling the unconditional distribution for speech, our model can utilize the untranscribed data for training. For texe-to-speech synthesis, we guide the gener- ative process of the unconditional DDPM via phoneme classification to produce mnelsppectrogrmss from the conditional distribution given transcript. We show that Guided-TT's achieves comparable performance with the existing methods with- out any transcript for LJSpeech. Our results further show that single speaker- dependent phoneme classifier trained on multispeaker large-scale data can guide unconditional DDPMs for various speakers to perform TTS.
INTRODUCTION
Neural text-to-speech (TTS) models have been achieved to generate high-quality human-like speech given text (van den Oord et al. (2016); [Shen et al. (2018)) In general, these TTS models are condi- tional generative models that encode text into the hidden representation and generate speech from the encoded representation. Early TTS models are autoregressive generative models which generate high-quality speech but suffer from slow synthesis speed due to the sequential sampling procedure (Shen et al. (2018); Li et al. (2019)). Owing to the development of non-autoregreesive generative models, recent TTS models are capable of generating high-quality speech with faster inference speed (Ren et al. 2019); ; Ren et al. (2021): 1) Kim et al. (2020); Popov al. (202T)) Recently, high-quality
end-to-end TTS models have been proposed that generate raw waveform from the text at once (Kim)
al. (2021) Weiss al. (202 Chen et al. (202
Despite the high-quality and fast inference speed of speech synthesis, most TTS models can be trained only if the transcribed data of the desired speaker is given. While long-form untranscribed data, such as audiobooks or podcasts, is available on various websites, it is challenging to use these unpaired speech data to train existing TTS models. To utilize these untranscribed data, long-form untranscribed speech data has to be segmented into sentence-level, and then each segmented speech should be transcribed accurately. Since the existing TTS models must directly model the conditional distribution of speech given text, the direct usage of untranscribed data remains challenging to solve.
In this work, we propose Guided-TTS, an unconditional biffusiin-bbssd generative model trained on untranscribed data that leverages phoneme classifier for text-to-speech synthesis. Trained on untranscribed speech data for the desired speaker, our unconditional diffusion probabilistic model learns to generate mel-speccrograms of the speaker without any context. As training data does not have to be aligned with text sequence for unconditional speech modeling, we simply use random chunks of untranscribed speech to train our unconditional generative model. This allows to build training data without extra effort in modeling the speech of speakers for which only long-form untranseribed speech data is available.
To guide the unconditional DDPM for TTS, we train a framewise phoneme classifier on transcribed data and use the gradients of the classifier during sampling. Although unconditional generative

---

Under review as a conference paper at ICLR 2022
model is trained without any transcript, Guided-TTS effectively generates mel-spectrograms given the transcript by guiding the generative process of unconditional DDPM with the phoneme classifier.
We demonstrate that the proposed method, TTS by guiding the unconditional DDPM, matches the performance of the existing conditional TTS models on LJSpeech. We further show that by training the phoneme classifier mance without seeing any transcript of LJSpeech, which shows the possibility to build a high-quality text-to-speech model without transcript for the desired speaker. We encourage the readers listen to samples of Guided-TTT TTS trained on various untranscribed datasets on our demo page. [
multi-speaker paired dataset, Guided-TTS also shows comparable perfor-
2 BACKGROUND
2.1 DENOISING DIFFUSION PROBABLISTIC MODELS (DDPM) AND ITS VARIANT
DDPM (Ho et al. 2020 which is recently proposed as a kind of probabilistic generative model, has been applied to various domains such as image (Dhariwal & Nichol (2021)) and audio (Chen) et al. (2021a); la): Popov et al. (202T)) DDPM first defines forward process that gradually corrupts data Xo to random noise XT across T timesteps. The model learns the reverse process that follows the reverse trajectory of the predefined forward process generate data from random noise.
Recently, there have been approaches to formulate the trajectory between data and noise as a contin- uous stochastic differential equation (SDE) instead of a discrete-time Markov process (Song et al. (2021b). Grad-TTS (Popov et al. (202T) introduces SDE formulation to TTS, which we have fol- lowed and used. According to the formulation of GTa,-TTS, the forward process that corrupts data Xa into the standard Gaussian noise X7 is as follows:
1
dX, - X + BidW,
(1)
where 9r is a predefined noise schedule, 3r (0T B. and W. is Wiener process. Anderson 1982) showed that the reverse process, which represents the trajectory from noise Xr to Xo. can be formulated in SDE, which is defined follows:
1 (- dX. =
- pxg v B.dW..
where W; is reverse time Wiener process. Given the score, the gradient of log density with respect to data (i.e., Vx, log (X.)), for i E [0, we can sample data X, from random noise Xr by solving Eq. (2. To generate data, the DDPM learns to estimate the score with the neural network sæ parameterized by 0.
To estimate the score, X, is sampled from the distribution derived from Eq. 1 given data Xo. which is as follows:
X1 (N(p(Xo, ((,
where p(Xo, t) 2 9, Xo. and A(t) e Sú 8,ds Then, the score can be derived from Eq. 3h: V x, log (X) - -X(t)= ce given No (Popov et al. (2011). To train the model so(X, (X,, t)
for Vt E [0, T), the following loss used:
L( = E,Ex, E.,, (X,,t) + A(t) [
(4)
which is L2 loss as in previous works (Ho et al. (020)). Song et al. (0021b)
Using the model se (X,,)), we can generate sample Xo from noise by solving Eq. 2 Grad-TTS
generates data Xo from XT by setting T = 1 and using a fixed discretization strategy (022b)):
Song et al.
3e 1
Be N ct
- = X, +
( 5
X. + Vx, log log):(X.)) +
(5)
where N is the number of steps to solve SDE 1 E ( 2 N. ..., 1) and 2, is standard Gaussian noise.
Demo: https://bit.ly/3oWh///
2

---

Under review as a conference paper at ICLR 2022
2.2 CLASSIFIER GUIDANCE
DDPM can be guided to generate samples with the desired condition without fine-tuning with the introduction of classifier. [Song et al. (2021b) used the unconditional DDPM generate class- conditional images using a separately trained image classifier. Not only unconditional DDPM but also conditional DDPM can be guided using the classifier, which contributes to achieving state-of- the-art performance for class-conditional image generation (Dhariwal & Nichol (202I).
For conditional generation, the classifier Pt (x) is trained classify the noisy data X, as the condition If Eq. (5) is modified, discretized SDE for the conditional generation can be obtained.
1 x X N 2 X, Vx, logp((X.lv)) +
B
(6)
Vx, log Pe (X,ly) = Vx, (X,) + Vx, log
(7)
If the unconditional score and the classifier for the condition given, the sample Xu with the condition y can be generated using Eq. (6).
3 GUIDED-TTS
In this section, we present Guided-TTS, which aims to build high-quality text-to-speech model without the transcript of the target speaker. While other TTS models directly learn to generate speech from text, Guided-TTS models the unconditional distribution of speech to utilize speech-only data and guides the unconditional model to generate speech with given text. To the best of our knowl- edge, Guided- -TSS is the first TTS model that generates speech with the unconditional generative model.
Both unconditional speech modeling and controllable generation with the unconditional generative model are well known be challenging. To tackle these challenges, adopt diffusion-based gen- erative model for unconditional speech generation, which has the advantages of modeling complex distributions and easy controllability. Additionally, we introduce a phoneme classifier to guide the unconditional DDPM for TTS, which requires transcribed data for training.
In Guided-TTS, the generative model and the phoneme classifier are trained separately so that can utilize different datasets to train each module. For instance, assume only the untranscribed speech data of the target speaker for TTS is available. With Guided-TTS. the phoneme classifier can still be trained on other transcribed datasets that contain rich, large-scale multi-speaker data. The phoneme classifier trained in such a manner effectively guides the generative model, trained only using the untranscribed speech data of the target speaker. By doing with the two modules independent of each other, we can achieve the text-to-speech model without any transcript the target speaker.
Guided-TTS consists of 4 modules: the unconditional DDPM, the phoneme classifier, the duration predictor, and the speaker encoder, as shown Fig. T Unconditional DDPM is a module that learns to generate mel-spectrogram unconditionally, and the remaining three modules are for TTS synthesis by guidance. We will explain the unconditional DDPM in Section 3.1, followed by the method of guiding the unconditional model for TTS in Section 3.2.
3.1 UNCONDITIONAL DDPM
Our unconditional DDPM models the unconditional distribution of speech Px without any tran- script. We assume that the training data for the diffusion-based model has tens of hours of untran- scribed speech data from the target speaker S for TTS. As our generative model learns only with speech data, training samples do not need be aligned with the text. Thus, we use random chunks of untranscribed speech as training data to reduce the burden of not only speech transcription but also segmentation when only the long-form speech data is available for the target speaker.
Given a mel-spectrooramm X -oo. define forward process in Eq. ( which gradually corrupts data into noise, and approximate the reverse process in Eq. 2 by estimating the uncon- ditional score Vx, p(X.) for each timestep t. At each iteration, X,, the mel-spectrogram X, as in Eq. (3). and score is estimated with the neural network sy (X,, t) parameterized by 0. The training objective of unconditional model is in Eq.
[0, is sampled from
3

---

We describe each module required for Under review as a conference paper at ICLR 2022
a b
Vx, log Po (X))
logp& (1) - 9
[1.8 2.3 0.71
Es
Unconditional DDPM
Xxt
Phoneme Classifier
Duration Predictor
Speaker Encoder
e-. -
t
Xt
Xt
y
[Xo x
The overall architecture of Guided-TTS. The unconditional DDPM learns to generate speech X, without the transcript. The other modules, the phoneme classifier, duration predictor, and speaker encoder are for guiding the unconditional DDPM to generate conditional samples given y. The speaker embedding eg is only used for training speaker-dependent modules.
Figure 1:
Similar to Grad-TTS (Popov al. 2222I), regard mel- spectrocto as 2D image with single channel and use the U-Net architecture (Ronneberger et al. (2015) ) as 8g. We use the same size of the architecture used to model 32 x 32 sized images in Ho al. (2020) to capture long-term depen- dencies without any text information, while Grad-TTS uses smaller architecture for the conditional distribution modeling.
3.2 TEXT-TO-SPEEET VIA CLASSIFIER GUIDANCE
For TTS synthesis, shown in Fig. [ in order to generate mel-spectrogram given text, our duration predictor outputs the duration for each text token and expands the transcript y frame-level phoneme label ğ. Then, sample a random noise X7 of the same length as ý from the standard normal distribution, and we can generate conditional samples by replacing the unconditional score Eq. (5) with the conditional score. As in Eq. (8), we can estimate the conditional score on the left side by adding the two terms on the right side: the first term is obtained from the unconditional DDPM, and the second term can be computed with the phoneme classifier. That is, we achieve to build a rext-to-speech model with the unconditional generative model for the speech by adding the gradient of the phoneme classifier during the generative process.
framewise phoneme classifier guide the unconditional DDPM. As
x, lolloop(XXll Xelú, spk S) Vx, log po (Xelspk: 5) V Vx, x, Po
( spk S)
(8)
If the transcribed speech dataset of the target speaker S is available, train all the unconditional DDPM, the phoneme classifier, and the duration predictor on the dataset the speaker S for TTS. Otherwise, if we only have the untranscribed speech data for the target speaker S, first train unconditional DDPM on the untranscribed speech data and leverage a large-scale multi-speaker transcribed speech dataset train the phoneme classifier and the duration predictor. In this case, to guide unconditional DDPM for the target speaker S, our phoneme classifier and duration predictor are designed to be speaker-dependee ing training. We provide the speaker embedding extracted from the pre-trained speaker verification network as condition both modules, as described Fig. 1 guidance below.
modules and generalize well the unseen speaker S dur-
Phoneme Classifier The phoneme classifier is a network trained on transcribed data that recog- nizes the phoneme corresponding to each frame of the input mel-spectrogram. To train the frame- wise phoneme classifier, we align transcript and speech using a forced alignment tool, Montreal Forced Aligner (MFA) (McAuliffe al. (2017)), and extract the frame-level phoneme label ÿ. The phoneme classifier is trained to classify the corrupted mpe-steetrogram X, sampled from Eq. (3) as the frame-level phoneme label ý. The training objective of the phoneme classifier is to maximize the
4

---

Under review as a conference paper at ICLR 2022
expectation of cross- between the phoneme label 9 and the output probability with respect to t [0, [].
phoneme classifier, and time embedding e₁. which extracted in the same way as in Popov et al. (202I), is used as global condition in WaveNet to provide information about the noise level of the corrupted input X, at timestep Additionally, for speaker-dependent classification, we use the speaker embedding es from the speaker encoder as the global condition.
We use WaveNet-like architecture (van den Oord et al. (0166)
Duration predictor is a module that predicts the duration of each text token for given text sequence y. We extract the duration label of each text token using MFA for the same data on which the phoneme classifier is trained. The duration predictor is trained to minimize L2 loss between the duration label and the estimated duration in the log-domain, and we round up the estimated duration during inference. The architecture of the duration predictor is the same as that of Glow-TTS (Kim al. (2020)) with the text encoder. We concatenate the text embedding and the speaker embedding es for training speaker-dependent module.
Duration Predictor
speaker information from the input mel- spectrogram and outputs the speaker embedding es. Similar to Jia et al. (2018), we train a speaker encoder with GF2E loss (Wan al. (20T8) on the speaker verífication dataset and use speaker en- coder condition speaker-dependent modules. To train speaker-dependent modules, we speaker embedding eş extracted from the clean mel-spectrogram Xo for each training data, For guidance, average and normalize the speaker embeddings of untranscribed speech for the desired speaker to extract ég.
Speaker Encoder Speaker encoder encodes
3.2. NORM-BASED GUIDANCE
Algorithm Norm-based Guidance
J: framewise phoneme label, s: gradient scale, 7: temperature
0: parameter of unconditional DDPM; d: parameter of phoneme classifier
X1 ~ [0, -1)
for i N to 1 do
t + N
0, f
V x, log po (X.)II//// (f)x)(
2, \O, T 11)
X, X₁ + de ( + Vx, +
log.o. ax + / #
end for return Xo
Initially, we scaled classifier gradient V log ( (J) Xx, spk = S) in Eq. using gradient scale which is the scaled version of Song et al. (2021b). However, when guiding the unconditional DDPM found that the norm of the unconditional score suddenly
with the framewise phoneme classifier, increases near the data. That is, the closer to the data, the phoneme classifier has little effect on log
the generative process of DDPM. The norm of unconditional score || log po (X.)] according to timestep i. is shown in Fig- 4 of Appendix A.4 Here, we propose norm-based guidance to guide the unconditional DDPM better in terms of generating speech conditioned
frame-level phoneme label ý. Norm- based guidance is a method of scaling the norm of the classifier gradient in proportion the norm of the score in order to prevent the effect of the gradient from being insignificant as the score rises steeply. The ratio between the norm of the scaled gradient and the norm of the score is defined as the gradient scale s. By adjusting s, we can determine how much the classifier gradient contributes to the guidance of unconditional DDPM. We also introduced the temperature parameter t when guiding the DDPM. We observed that tuning т to a value greater than helps generate nigh-quality melsspectrograms.
4 EXPERIMENTS
Datasets For experiments using a transcribed single dataset, we train our unconditional DDPM and phoneme classifier respectively speech-only IJSpeecch (Ito (2017)) and speech-transeript paired
5

---

Under review as a conference paper at ICLR 2022
LJSpeech to match the settings of baseline TTS models. In addition, since method makes use of a set of separately trained unconditional DDPM and phoneme classifier, they can also be trained with different datasets respectively. Here, by training the unconditional DDPM with speech-only target speaker dataset and the phoneme classifier with a transcribed large-scale multi-speaker dataset, we can build high-quality TTS model from any specific speaker dataset without any transcript. For the corresponding experiments, we use only the speech data from LJSpeech, Hi-Fi TTS (Bakhturina) al.] (. and Blizzard 2013 (King & Karaiskos (2013) for training unconditional DDPMs, and TIS (Zen et al. (2019)), a large-scale multi-speaker dataset with approximately 585 hours of speech uttered by 2456 speakers with corresponding texts, for training the speaker-dependent phoneme classifier to guide the various unconditional generative models. To extract the speaker embedding es from each utterance, we train speaker encoder on VoxCeleb2 (Chung et al. (2018). which is a speaker verification dataset that contains more than 1M utterances for 6112 speakers.
LJSpeech is a 24-hour single female speaker dataset consisting of 13,100 audio clips. Dataset is randomly split; 12,500 samples for the training set, 100 samples for the validation set, and 500 samples for the test set. Hi-Fi TTS is a multi-speaker dataset with 6 females and 4 males, and each speaker's 's data consists of at least 17 hours of speech. We select three of them (two males and one female) and use them for training three unconditional DDPMs, respectively. Blizzard 2013 is an 147 hours segmented and unsegmented audiobook data read by a single female speaker. We only use 5-second randomly clipped audio of unsegmented data to show that our model does not require text labeling or aligning for untranscribed data. Only audio files are used train unconditional DDPM.
Training Details We convert text into International Phonetic Alphabet (IPA) phoneme sequences us- ing open-source software (Bernard (2021) ). To extract the mel-spectrogram, we use the same hyper- parameters as Glow-TTS (Kim et al. )2020)). All modules are trained using Adam optimizer with a learning rate of 0. 0.00011. For the unconditional model and the phoneme classifier, Pg 0.05, $1 = 20 are used for beta schedule. For unconditional model, the base model has the same architecture and hyperparameters as DDPM (Ho et al. (2020) used for 32 32 sized image modeling, and for ab- lation, we use the same architecture and hyperparameters for the small model as Grad-TTS (Popov) ctt al. (02II). Other details and hyperparameters are described in the Appendix A..
Evaluation To compare the performance of models, we use pre-trained models and the official implementations of Glow-TTS and Grad-TTS For Glow-TTS, we use a pre-trained model with blank tokens between phonemes and use 1.5. We the same hyperparameters implementation, т = 1.5, and the number of reverse steps N = 50 for Grad-TTS. For our model, set 1.5, gradient scale s 0.3, and the number of reverse steps N 50. We use official implementation and pre-trained models of HiFi-GAN as a vocoder?
the official
To show whether Guided-TTS with the norm-based guidance generates the sentences of the given text accurately, we measure the character error rate (CER) and word error rate (WER) for each model, which are metrics commonly used in automatic speech recognition (ASR). We use the pre- trained CTC-based conformer (Graves et al. (2006), Gulati et al. (2020)) model provided by NEMO toolkit (Kuchaiev et al. (2019)) compute the metrics.
5 RESULTS
5.1 MODEL COMPARISON
We compare the performance of audio samples by measuring the 5-scale mean opinion score (MOS) on LJSpeech using Amazon Mechanical Turk. In addition, through CER and WER, we check whether the generated sample of Guided-TTS faithfully reflects the text. To calculate CER and WER, we first synthesize the speech of a given text for each model and provide it to the ASR model to extract the text corresponding to the generated sample. We then measure the CER and WER between the ground truth text and the text obtained from the ASR model. For evaluation, we ran- domly select 50 samples drawn from the test set of I.ISpeech, and the results are shown in Table In Table Guided-TTS-T T (T.Transcribbd) is specified for the case of training the unconditional
"Glow-TTS: https://bit.ly333333335
Grad-TTS: https:/bii.ll//3TTCmmll
'HiFi-GAN: https://ii...y/33XB55..
*Demo: https///ii.ly333WWWigg

---

Under review as a conference paper at ICLR 2022
DDPM and the other modules the speech data of the target speaker. The quality of Guided-TTS-- on LJSpeech shows that our model achieves TTS performance comparative to the baseline models. Furthermore, the ASR metrics of Guided-TTS. indicate that model generates speech samples according to given text stably like the existing TTS models do. Unlike Grad-TTS, which leverages the conditional DDPM for TTS, ing long-term dependency of speech without text. The qualitative results of Guided-TTS- with the small unconditional model (Guided-TTS-T (small)), whose architecture the same as the decoder of Grad-TTS, demonstrate the importance of modeling capability of unconditional DDPM for the naturalness of the samples and accurate pronunciation.
unconditional DDPM requires large model size for model-
Guided-TTI TTS-U (U:Untranscribed) is the case that we train unconditional DDPM using only the tar- get speaker's text-free speech and guide it using a speaker-dependert phoneme classifier separately trained on LibriTTS. Guided-TTS-U shows performance and ASR metrics comparable to other TTS models, which require transcribed data of the desired speaker for training. This demonstrates that Guided-TTS enables building a high-quality TTS model using the untranscribed speech of the target speaker. Samples of all models are available on the demo page.
Table 1: Mean Opinion Score (MOS) with 95% confidence intervals and Automated Speech Recog- nition (ASR) metrics of TTS models for LJSpeech. The ASR metric indicates character error rate (CER) and word error rate (WER) measured by the pre-trained ASR model. The unconditional DDPM in Guided-TTS uses only untranscribed speech of LJSpeech for training. Guided-TTS-1 is model whose phoneme classifier and duration predictor are trained using the transcript of LJSpeech. Guided-TTS-- uses a multi-speaker dataset instead of the transcript of LJSpeech to train phoneme classifier and duration predictor.
Method
Transcript of I.Sppeech
5-scale MOS
CER/WER(SA )
Ground Truth
4.46 0.05
0.79
Ground Truth (Mel + HiFi-GAN)
4.25 + 0.07
1,05 / 4.08
Glow-TTS
4.10 =
5.00
Grad-TTS
4.25 0.09
/
Muided-TTS-T
4.18 0.07
1.20 4.71
Guided-TTI -TS-eT------T----- (small)
4.01 = 0.07
1.65 (65/5..7
-TS-U
4.18 0.07
1.26 4.61
Table 2: Mean Opinion Score (MOS) of Guided-TTS-U with various untranscribed datasets with 95% confidence intervals. In Quided-TTS-LL U, the phoneme classifier and the duration predictor are trained on LibriTTS dataset, and the speaker encoder trained on VoxCeleb2 dataset.
Method
Untranscribed Data
5-scale MOS
Total duration (hrs)
Gender
LJSpeech
4.18 + 0.07
24
Female
Hi-Fi TTS ID: 92
3.97 + 0.08
27.3
Female
Guided-TTS-U
Hi-Fi TTS ID: 6097
3.82 + 0.08
30.1
Male
Hi-Fi TTS ID: 9017
3.72 + 0.08
58.0
Male
Blizzard 2013
3.85 + 0.08
149.4
Female
5.2 GENERALIZATION DIVERSE DATASETS
We show that our model can synthesize high-quality speech of LJSpeech using untranscribed speech in the previous section. Since our model trains the unconditional model and the classifier separately, TTS for various untranscribed datasets is possible with single classifier trained with a large-scale paired multi-speaker dataset. The performance of our multiple TTS models using the untranscribed speech of target speaker is presented in Table 2 For evaluation,
used the same 50 randomly cho- sen sentences from the test set of LJSpeech. Table 2 we show that if we have the phoneme clas- sifier, duration predictor, and speaker encoder trained on large-scale multi-speaker dataset,

---

Under review as a conference paper at ICLR 2022
build TTS model using untranscribed speech data of any speaker without additional burdens such as transcription and segmentation. In particular, Guided-TTS-U synthesizes high-quality samples when trained randomly cropped unsegmented Blizzard 2013 dataset. Our method enables TTS for diverse untranscribed datasets with various characteristics (e.g., gender, American/Brinn accent, prosody). There samples on the demo page for diverse datasets.
5.3 ANALYSIS ON NORM-BASED GUIDANCE
We also compare the proposed norm- based classifier guidance with the classifier guidance used in previous works (Song al. (2021b); Dhariwal & Nichol (202T)). A model that performs the conditional generation task with classifier guidance sometimes generates samples of conditions other than the target condition (Song et (2021b)). Similarly, observe that Guided-TTS with the previous guidance method produces mispronounced samples given text. To show the effect of norm-based guidance and adjustment of gradient scale, measure CER of Guided-TTS-T and Guided-TTH TTS- for LJSpeech according to the gradient scale We explore the gradient scale within 00.5, 1.0, ... 4.5] for the scaled version of the previous method (Song et al. (202Ib)), and [0.1, 0.2, ... 1.0] for the norm-based guidance.
14
14
Gulded-TTS-T
Gulded-TTS-T
12
Guided-TTS-U
12
Guided-TTS-U
10
Glow-TTS
10
Glow-TTS
CER (%)
Grad-TTS
CER (%)
Grad-TTS
0.2
0.4
1.0
Gradient scale
Gradient scale
(a) Classifier guidance in Song et al. 2021b
(b) Norm-based classifier guidance
Figure 2: CER according to gradient scales, (a) CER of Guided-TTS with the classifier guidance (scaled version of Song et al. (2021b)). (b) CER of Guided-TTS with the norm-based guidance
Fig. 2 the CER of the Guided-TTYs with the classifier guidance in the previous works and the proposed norm-based guidance. As shown in Fig. [2a] the sample generated by the existing guidance method shows far worse CER than the existing TTS models, which indicates that it is not suitable TTS. the other hand, Fig. 22 shows that the proposed guidance method with the appropriate gradient scale helps generate samples given text sentences accurately like the existing TTS models.
the gradient scale is too small, the effect of the classifier gradient is negligible, and generated samples do not reflect the given text. On the other hand, observed that guidance with gradient scale too large deteriorates the sample quality. For the proposed norm-based guidance, we set the default gradient scale to 0.3, which generates high-quality samples that exactly match the given text. Samples for multiple gradient scales with each guidance method are on the demo page.
RELATED WORK
Unconditional Speech Generation In general, the unconditional speech generative model (van den Oord et al. (2016): Vasquez & Lewis (2019)), which models audio without any information, is more challenging than conditional generative model that synthesizes speech using text or mel- spectrograms. Several works attempt unconditionally generate raw waveforms (van den Oord et al.) (2016); Donahue et al. (2019) or to model the unconditional distribution of Iatent code or mel-spectrogram of audio (van den Oord et al. (2017;; Vasquez & Lewis (209)) instead of directly modeling raw waveforms. Most existing unconditional models have only been used for uncondi- tional audio modeling and no other purpose. To the best of our knowledge, this is the first applica-
8

---

Under review as a conference paper at ICLR 2022
tion of unconditional model for TTS with appropriate guidance to enable speech synthesis using untranscribed data of the target speaker.
Text-to-Speech Models Most text-to-speech (TTS) models are composed of two parts:
model
that generates intermediate features (e.g., mel-spectrogram) from text (Shen et al.] (2018) and the
vocoder, which synthesizes raw waveforms from intermediate features (van den Oord et al.
(016
The autoregressive model is used for the text-to- tit-ttoointeeediite feature model (Wang et al.
(2017
Shen et al. (2018); Ping et al. (2018); Li al. (2019) and vocoder (van den Oord al. Kalchbrenner et al. (2188) to perform high-quality TTS. Since the autoregressive models generate samples sequential manner, their inference speed is slow. Thus, various parallel TTS mod-
(016);
els have been proposed to improve the sampling speed. Flow-based generative models (Kingma & Dhariwal (2018) and feed-forward models have been proposed for text-to-mel--ppeetrgraaa models (Ren et al. (2019) Ren et al. y0 Kim et al. 2020);; (Shih et al. (021)) and vocoders (Oord et al.
2018 : Prenger et al. ) Kim et al. (2019)). Also, variational autoencoder (Kingma & Welling
2014 based models (Lee et al. (2020);
; Liu et al. (2021). diffusion (Ho et al. (2020) based models
Chen et al. (202 Kong et al. (2021)) Popov et et (202T), Jeong et al. (2021)) GAN (Goodfellow
ct al. (2014)) based models (Kumar al. (2019) Binkowski al. (2019); Kong et al. (2020) have
been proposed as high-quality speech synthesis models with paralel sampling schemes.
Recently, end-to-end TTS models have been proposed, such as Ren et al. (2021). Donahue et al.
(202T) Weiss et al.] 2222)) Kim et al. (202T), and Chen et al.] (202 Most previous TTS models perform conditional generation tasks using transcribed speech data of the target speaker. We propose a new TTS model that learns to generate speech of the target speaker using untranscribed data and guides through phoneme classifier trained with other transcribed datasets. By guiding the uncon- ditional model with the independently trained classifier, we achieve high-performance TTS model using untranscribed data of the desired speaker without sentence- segmentation or transcription.
Diffusion-based Generative Models DDPM (Ho et al. (2020)) has undergone several theoretical developments (Song et al.] (202Tb)) and produces high quality samples in many domains (Ho et al.
(2000) Dhariwal & Nichol (2021); Chen et al.) (2021a): Poopvv et al. (2021) Luo & Hu] (2021 )
Many theoretical and practical breakthroughs (Nichol & Dhariwal (202 3 Lam et al. (2021); Chen
et al. (202 Song et al. (2021a); Watson al. (2021); Kong & Ping (202 Molicoeur-Martinee
ct al.
2021) have been proposed, and a continuous version DDPM, an SDE-based model (Song et al. b Popoy et al. (2021)) is also presented. DDPM has also shown strong performance in
speech synthesis (Chen et al. (2021a); Kong et al. (2021) Popoy et al. (2021); Deong et al.] (2021))
A pre-trained unconditional DDPM can be used for various tasks such imputation (Song et al. (2021b)) and controllable generation (Song et al. (20215)). In particular, the controllable genera- tion allows Dhariwal & Nichol (2021) to achieve state-of-the-art performance in class-conditions image generation by guiding the DDPM using a gradient from the classifier trained on the same dataset DDPM. We introduce the classifier guidance method of unconditional DDPM to text-to- speech synthesis. Our unconditional DDPM and the phoneme classifier can be trained using different datasets, making it possible to build a TTS model with the target speaker's untranscribed speech.
CONCLUSION
In this work, we present Guided-TTS, a new type of TTS model that generates speech given tran- script by guiding the unconditional diffusion-based model for speech. As Guided-TTS models un- conditional distribution for speech, the proposed model can construct a TTS model using the desired speaker's untranseribed data. Thanks the properties of fiffusion-based generative models, our unconditional generative model can generate speech when a transcript is given by introducing the separately trained phoneme classifier. To the best of our knowledge, Guided-TTS is the first TTS model to leverage the unconditional generative model for speech. We showed that Guided- TTS matches the performance of the previous TTS models on LJSpeech. Moreover, Guided-TTS without the transcript of the target speaker generates samples comparable to existing models using tran- scripts by training its phoneme classifier on a large-scale multi-speaker dataset. We also showed that the single well- well-performed phoneme classifier can guide various unconditional DDPMs to generate high-quality sample. We believe that Guided-TTe can reduce the burden of constructing training datasets for high-quality TTS.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Brian D O Anderson. Reverse-time diffusion equation models. 313-326, May 1982.
Stochastic Process. Appl., 12(3):
Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg, and Yang Zhang. Hi-fi multi english tts dataset. arXiv preprint arXiv:2104.1197, 2021.
Mathieu Bernard. Phonemizer.
2021.
Mikolaj Binkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan. High Fidelity Speech Synthesis with Adver- sarial Networks. In International Conference on Learning Representations, 2019.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- Grad: Estimating Gradients for Waveform Generation. In International Conference on Learning Representations, 2021a.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. WaveGrad 2: Iterative Refinement for Text-to-Speedt Speech Synthesis. In Proc. Interspeech 2021, pp. 3765-3769, 2021b. doi: 01144///neerpeech2222111155
S. Chung, A. Nagrani, and A. Zisserman. SPEECH, 2018.
Voxceleb2:
Deep speaker recognition. In INTER-
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.00 05233, 055.3333,
2021.
C. Donahue, McAuley, and M. Puckette. Adversarial audio synthesis. International Conference on Learning Representations (ICLR), 2019.
Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End-to- end Adversarial Sexech- In International Conference Learning Representations, 2021.
net URL httthttps///ppeeeee/// n forum? reddrsfllz -aifflz-sjj8
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Pro- ceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution Convolution-augmentte augmented Transformer for Speech Recognition. In Proc. Interspeech 2020, pp. 5036-5040) 2020. doi: 10.2 10.21437/nterspecch22222 2000-33005.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising Diffusion Probabilistic Models. In Ad- vances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 6-12, 2020, virtual, volume 33. Curran As- sociates, Inc., 2020.
2017. Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff- TTS: A Denoising Diffusion Model for Text-to-Speechh In Proc. Interspeech 2021, pp. 3605- 3609, 2021. doi: 10221457/mnerppecch22211111
Keith Ito. The speech datasct.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, and Yonghui Wu. Transfer learning from speaker verifica- tion to multispeaker text-to-see sokt-to-ppecch synthesis. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bias Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 31. Curran Associates, Inc., 2018. URL htt http://tttttp:atteeeeeeeeeeeeeee neuriin..............o.m..
sc/paper/2018/ ooc.aa/2228111114222200000003133
10

---

Under review as a conference paper at ICLR 2022
Alexia DolicoeurMaartinen Ke Li, Rémi Piché-Taillefer Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 2021.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pp. 2410-2419. PMLR, 2018.
Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-TTS: A Generative Flow for Texe-h--Speech via Monotonic Alignment Search. Advances in Neural Information Processing Systems, 33, 2020.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. arXiv preprint arXiv:2 106.00103, 2021.
Sungwon Kim, Sang- Lang-Gil Lee, Jongyoon Song, Jachyeon Kim, and Sungroh Yoon. Flowavenet: A generative flow for raw audio. In International Conference on Machine Learning, pp. 3370-3378, 2019.
Simon J. King and Vasilis Karaiskos. The blizzard challenge 2013. In Blizzard Challenge Work- shop, 2013.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances Neural Information Processing Systems, pp. 10236-10245, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations (ICLR), 2014.
Jungil Kong, Jachyeon Kim, and Jackyoung Bac. HiFi-GAN: Generative Adversarial networks for Efficient and High Fidelity Speech Synthesis. Advances in Neural Information Processing Systems, 33, 2020.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021. URL /
net/ forum? dacpuummmd-ggiii agddd-ggjjooofo. 4cd0frAP
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. In International Conference on Learning Representations, 2021.
Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, et al. Nemo: toolkit for building ai applications using neural modules. arXiv preprint
(0t::1909.0955. 2019.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In Advances in Neural Information Processing Systems 32, pp. 14910-14921, 2019.
Max WY Lam, Jun Wang, Rongjie Huang, Dan Su, and Dong Yu. Bilateral denoising diffusion models. arXiv preprint arXiv:2 108./ //544, 2021.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. Jnttt://///aan
lecun. cow. wttp://w//////tcccetcttoo wexdb/mnissite
Yoonhyung Lee, Joongbo Shin, and Kyomin Jung. Bidirectional variational inference for non-
autoregressive text-to-speech. In International Conference on Learning Representations, 2020.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with trans- former network. In Proceedings the the AAAI Conference on Artificial Intelligence, volume 33, pp- 6706-6713 13,66611116611 2019.
Peng Liu, Yuewen Cao, Songxiang Liu, Na Hu, Guangzhi Li, Chao Weng, and Dan Su. Vara-tts: Non-autoregreesivee text-to tox-tto-seeech synthesis based on very deep vae with residual attention. arXiv preprint ar 2rxiv 06431, 2021.
11

---

Under review as a conference paper at ICLR 2022
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2837-2845, June 2021.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In INTERSPEECH, 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic mod- els. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Confer- ence Machine Learning, volume 139 of Proceedings of Machine Learming Research, pp. 8162 8171. PMLR, 18 24 Jul 2021. URL hatthhttss//// httass///occeedngg..oo ptt.s//////....e////33////
ni nichol2la. html
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference on machine learning, pp. 3918-3926. PMLR, 2018
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan Ömer Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: Scaling text-to-speech with convolutional se- quence learning. In International Conference Learning Representations, 2018.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8599 8608. PMLR, 2021.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621. IEEE, 2019.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech: Fast, Robust and Controllable Text to Speech. volume 32, pp. 3171-3180, 2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. FastSpeech 2: Fast and Representations, 2021. URL http::tttpppsereeeeeeeeeee
End-to-End Text to Speech. In International Conference on Learning
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779-4783. IEEE, 2018.
Kevin Shih, Rafael Valle, Rohan Badlani, Adrian I Lancucki, Wei Ping, and Bryan Catanzaro. Rad- its: Parallel flow-hased tts with robust alignment learning and diverse synthesis. In ICML Work- shop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a. URL htt.htttpss///eeeeeeewwww net.
forum? d= 1 arCHIP
Yang Song, Jascha Sick-Dickst,in, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- lional Conference on Learning Representations, 2021b. URL. htttps://spenreee
net/
forum? waaimnid==PxTTe PIPPP 12RRX 2RRHS
Aäron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609.0349, 2016.
12

---

Under review as a conference paper at ICLR 2022
Aaron den Oord, Oriol Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 1306-6315, 2017.
Sean Vasquez and Mike Lewis. Melnet: A generative model for audio in the frequency domain.
arXiv preprini arXiv: 0000000003, 2019.
Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4879 4883, 2018. doi: 10.1109//ASSSP20088 8.4466665....
Yuxuan Wang. R.J. Skerry-Rye Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, Quoe Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. Tacotron: Towards End-to-End Speech Synthesis. In Proc. Inter- speech 2017, pp. 4006 2017. doi: 10.21437/mersseechh.07.. 2ttp
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning efficiently sam- ple from diffusion probabilistic models. arXiv preprint arXiv:2106.03802 2021.
Ron J Weiss, RJ Skerry-Ryan, Eric Battenberg, Soroosh Mariooryad, and Diederik P Kingma. Wave- tacotron: Spectrogran-free end-to-end text-to-speech synthesis. ICASSP 2021-202) IEEE ternational Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5679-5683.
IEEE, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. In Proc. Interspeech 2019, pp. 1526-1530, 2019. doi: 21244447/merspecch2009992-
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 INPAINTING
We perform the inpainting task to show how well the unconditional DDPM learns the dependen- cies in mel-spectrogram. The pre-trained unconditional DDPM fills out the masked part of the mel-spectrogram. We use samples from three speakers; one female speaker (ID: 92) and one male speaker (ID: 6097) from Hi-Fi TTS and female speaker from LJSpeech. Two cross- coos--happd masks (LJSpeech, Hi-Fi TTS male) and one binarized MNIST (LeCun & Cortes] (0100) mask (Hi-Fi TTS female) used for masking. We set 1000 the number of reverse steps N and 1.5 for inpainting. The method of inpainting is the same as Song et al. (2021b). and the algorithm is as follows:
Algorithm Inpainting Mel-spectrogram
Binary Mask: M, Original spessppctrogrmm: Xo 0: parameter of unconditional DDPM
X1 [00, T-)
for i N to 1 do + N
t
,( 4- e Po
A(1) 5 1i 3,do
X, ~ (()(Xx,,)) (t))
X, + X, M + Xx @ (1 M)
2 N
X, - X, + (
Vx, (X.)) + V/22
end for return Xo M + Xo 3 -1-M) M)
Hi-Fi TTS (ID: 6097)
Hi-F TTS (ID: 92)
2
IJ Speech
(a) GroundTTui Truth
(b) Masked Input:
(c) Impainting
Figure 3: Mel-spectrogram inpainting results of unconditional DDPM trained on LJSpeech, and two speakers (Speaker ID:: 92, 6097) from Hi-Fi TTS.
The inpainting results are shown in Fig. 3 where (a) is the original mel-spectrogram, (b) is the masked mel-spectrogram, and (c) is the result of inpainting on the masked part. As shown in Fig. 3 we show that our unconditional DDPM learns the adjacent frequency and temporal dependencies of the mel-spectrogram. Samples of inpainting results are provided the demo page.
A.2 TRAINING DETAILS AND HYPERPARAMTERS
In this section, modules are trained on LJSpeech. In Guided-TTi -TS-U, we use only untranscribed data of the various target speakers for training unconditional DDPMs and train phoneme classifier and duration
the training details and detailed hyperparameters. In Guided-TTS-T. all
14

---

Under review as a conference paper at ICLR 2022
predictor LibriTTS. Alignment labels are required for training the phoneme classifier and the duration predictor, and we train MFA to extract the alignment using the same dataset used to train the phoneme classifier and duration predictor.
For both Guided-TTS-T and Guided-TTS-UE the unconditional models are trained with batch size 16 for all datasets. The duration predictor is trained for 20 epochs with batch size 64. The phoneme classifier Guided-TTS-T uses a WaveNet-like structure with 256 residual channels, residual blocks stacks of dilated convolution layers, and is trained for 1000 epochs with batch size 64. For Guided-TTS TTS-U, we use a WaveNet-like structure with 512 residual channels, 3 residual blocks stacks of dilated convolution layers for the phoneme classifier, and it trained for 140 epochs with batch size 64. The speaker encoder is two-layer LSTM with 768 channels followed by a linear projection layer to extract 356-dimensional speaker embedding es, and trained for 300K iterations.
For sampling, we use the last checkpoint for the unconditional DDPM and the speaker encoder. We use the checkpoint of the epoch with the best metric for the phoneme classifier (validation accuracy) and the duration predictor (validation loss).
A.3 HARDWARE AND SAMPLING SPEED
We conduct all experiments and evaluations using NVIDIA's RTX 8000 with 48GB memory. Al- though model is not focused on fast inference, Guided-TTS can perform real-time speech syn- thesis on GPU for N = 50, which is the number of reverse steps we use for evaluation. We measure the sampling speed of Guided-TTS using real-time factor (RTF). We also measure how much time it takes to compute the unconditional score (V X, log pe
(e(X)) (X²)) and gradient of the classifier xVx, log P& X,)). TuS-ed-TTS-T achieves an RTF of 0.56, of which 0.39 is used to calculate the score and O. 16 is used for classifier gradient calculation. Similarly, Guided-TTS: U achieves an RTF of 0.63, 0.39 for computing the score and 0.23 for computing the gradient of the classifier.
A.4 NORM OF THE UNCONDITIONAL SCORE AND CLASSIFIER GRADIENT
||Vxllogp&(X>]]
(|Vx)
2000
120
1750
100
1500
1250
80
1000
60
750
500
40
250
20
0.0
0.2
0.4
0.8
1.0
0.4
0.6
1.0
Time t
Time t
(a) The norm of the unconditional score
(b) The gradient norm of the classifier
Figure& 4: The norm of the unconditional score and the classifier gradient for each timestep
The norm of the unconditional score and the gradient norm of the classifier for each timestep are 3 shown in Fig. 4 We sample X; at a total of 1000 timesteps (t E (2000 2000 -.... 2000 )) using the Eq. (3) for all 500 samples from the test set of the I.ISpeech. We then obtain the norm of the unconditional score and the gradient of the classifier using the sampled X, with the modules of Guided-TTS-T. Each norm is averaged over 500 samples for each timestep. As shown in Fig. 4al the norm of the unconditional score rises steeply around 0. This is about 70 times larger than the norm of the classifier gradient near = 0 (Fig. 4b)), which significantly reduces the effect of the classifier guidance. To alleviate this problem, propose the norm-based guidance in Section 3.2.1, which helps prevent both the gradient of the classifier from being ignored and the issue of synthesized speech not matching the text.
1999
15


Reviews:
[{"review_id": "xjNAarzNPIx", "reviewer": "Reviewer_MW7v", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) I really like the direction, doing TTS by decoupling audio modeling, text (phoneme) modeling and speaker modeling. This could potentially  utilize large amount data and respect to each task's data requirement, for example, audio modeling could use much more noisy data.\n(2) The way it adding gradients from phoneme classifier to guide unconditional model is new to speech application. The paper also propose practical trick to make it works better as norm based guidance.\n\nWeakness:\n\n(1) The paper doesn't really prove having additional (more diversified dataset) is better. From Table 1, using LibriTTS and LJSpeech looks about the same. To train the phoenme classifier it still need the whole LJSpeech dataset. Table 2 trying to emphasis the model can decouple generative model training and the classifier. However, it doesn't compare with a proper baseline, for example, you can also train a vocoder and mel-predictor plus speaker emb on different dataset.\n\n(2) The paper doesn't answer how stable of such a model. One potential issue of decoupled model is the TTS cannot always follow the given text. It's better to report the ASR metric, so it would make sure the MOS wasn't misleading, for example. it could generate human sound, but with lots of pronunciation error and still get good MOS score.\n\n(3) Following on (1), it's unclear to me for table 2 what data used to train classifier and what data for DPM.\n\n(4) In the related section, it's better to compare with non e2e model, given the fact this model use phoneme classifier, aligner (to get duration) and a vocoder like DPM. As a comparison, traditional tts (before tacotron) model will first train a model to get linguistic features and duration and then use a vocoder on top of it. \n\n\nOther comments:\n(1) wihtout -> without in introduction. \n(2) Table 1 and 2 was not very clear to me. Can you explain \"In addition, we show that we can generate high quality samples using untranscribed speech of speaker\", does it mea your phoneme classifier is trained on different data? It's better to mark it in the table.", "summary_of_the_paper": "This paper borrows the backbone from Grad-TTS [1], building an unconditional diffusion based model on speech input. With a phone classifier and duration predictor trained on transcribed data, and a speaker encoder trained on labeled speaker dataset, the model was able to synthesize speech on given text. Experimental results shows it's comparable with Grad-TTS [1] trained on transcribed data (by conditioning on the text during training). Ablation shows it can generalize to a diversified dataset. \n\n\n[1] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS:\nA Diffusion Probabilistic Model for Text-to-Speech. In Proceedings of the 38th International\nConference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of\nProceedings of Machine Learning Research, pp. 8599–8608. PMLR, 2021a.", "summary_of_the_review": "The biggest advantage of the proposed framework is it was able to use large amount speech data so it potentially could modeling wider range of prosody or other aspect of speech. However, the paper's experimental results cannot well-support it. I would happy to change the score if the author can improve follows:\n\n(1) Adding evidence such guidance framework is relative stable, e.g. faithfully reflect the text it been given. \n(2) Demonstrate this model is better than a simpler baseline: a text-to-mel model + speaker verification network + vocoder (trained on diversified mel). My bet is if here is a win, DPM model might capture better prosody. If showing prosody win is hard, at least demonstrate this model give better results on Blizzard which i assume the training data is more noisy?\n(3) The readability of Table 1 and 2 can be improved. E.g. adding what data been trained for the classifier, which speaker been used etc.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MVu4VFKH_k", "reviewer": "Reviewer_6P3J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The application of the method from Song et al to TTS is novel.\n- The demonstrated method allows the use of lots of untranscribed data for TTS, which is very useful.\n\nWeaknesses:\n\n- The novelty is limited. It is just a direct application of a previously proposed method (Song et al) on TTS.\n- The experiments and analysis are weak.\n- Even with the use of more data, the method is not able to outperform existing models.\n- No published code.\n\nUnconditional DDPM can be trained unsupervisedly, yes.\nBut not the phoneme classifier?\n\nFor training framewise phoneme classifier, we align transcript and speech using a forced alignment tool, Montreal Forced Aligner (MFA).\nPretrained MFA models? So implicitly making use of additional data? This should be made very clear. A statement like \"Guided-TTS only makes use of LJSpeech for this experiment\" is factually incorrect.\n\n\"the norm of the unconditional score\"\nIs this the norm of the gradient or of the log prob, or what exactly?\n\ntts -> TTS. Other English grammar errors (missing articles etc).\n\nHiFi-GAN vocoder, where is it from? pretrained? public?\n\nAs I understand, Table 1 uses only LJSpeech (fully transcribed) for all models, including Guided-TTS, except the last row of Guided-TTS, which uses LibriTTS for the phoneme classifier?\nGrad-TTS performs better (Table 1).\n\nTable 2 uses only the specified other corpora (e.g. HiFi TTS) for training but not LJSpeech in that case?\nWhy does it perform worse, even with more data? Because of mismatched conditions?\nWhat would happen when LJSpeech + HiFi-TTS is combined, or other combinations? It should get better overall, also over the LJSpeech-only case, right? This should be verified.\n\nOnly those two tables with experiments?\nThis is way too little. A lot of further things should be analyzed. E.g.:\n\n- What is the effect of the norm-based guidance and the gradient scale s?\n- What is the effect of different types of classifiers, e.g. making the classifier stronger or weaker?\n- Can we actually improve by the use of more (untranscribed) data? Isn't this the main motivation behind this? That was my understanding. But this is actually not shown, or not even tested. What is the point then in being able to use more data?\n- Etc\n\nWhere is the code?\n", "summary_of_the_paper": "Combines an unconditional denoising diffusion probabilistic model (DDPM) with a separately trained phoneme classifier to be able to guide the generation of mel spectogram towards a given aligned phoneme sequence, thus effectively having a conditional generative model. \n\nThis is based on the same principles which were applied before on images (Song et al. (2021b)).", "summary_of_the_review": "While the novelty is limited, this is still a very interesting direction to investigate. However, in its current form, it provides way too little experiments and analysis.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Fj0obcxEfA5", "reviewer": "Reviewer_XRxC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed Guided-TTS model utilizes both diffusion probabilistic model with the gradient signals from phoneme classifier, duration predictor and speaker encoder. The overall architecture is more like pretraining + finetuning or multitask learning (if jointly trained), which is not exactly 'generate speech from untranscribed speech data', since both untranscribed and transcribed data are used in the process.\n\nBesides that, one of the main benefits of using untranscribed speech data, which is to reduce the amount of annotated transcribed data required, is not showed in the paper. In the experiment results, the pattern of data annotation reduction is not illustrated and discussed.", "summary_of_the_paper": "This paper presents Guided-TTS which is a TTS model that learns to generate speech from untranscribed speech data. Guided-TTS combines an unconditional diffusion probabilistic model with a separately trained phoneme classifier to guide the generative\nprocess of mel-spectrograms from the conditional distribution given transcript.", "summary_of_the_review": "Overall, this paper proposes a new architecture of utilizing both untranscribed and transcribed speech data to generate speech with good quality.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CfnTyMFg-pr", "reviewer": "Reviewer_T6xj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the Paper:\n- Proposes a solution to one of the biggest bottlenecks in TTS.\n- The idea of using feedback from a separate model as conditioning signal during sampling has been done for image generation, however the application of a phoneme classifier for this in the speech domain is a very elegant solution.\n- Experiments of transferring the knowledge of the phoneme-guidance module to an unseen corpus show promising results for training on speech completely without transcription.\n- In general, convincing experiments and results\n\nWeaknesses of the Paper:\n- As stated in section 5.1, the model needs to be quite large. This probably also means expensive computation during both training and inference. There are no mentions of training time, training hardware or real-time-factor and hardware during inference. The latter of which is quite important for TTS.\n- The phoneme recognizer still needs paired data in order to be trained. And while it seems to work well cross-speaker and even cross-accent and cross-gender as the demo page shows, I assume it would work much less well across e.g. languages or highly expressive domains. So paired data would still be needed for those applications, which are the ones that suffer the most from low-resource.\n\nComments:\n- The weakness of paired data being required for the phoneme classifier in challenging domains could potentially be fixed by auto-alignment frameworks, that learn alignments self- or semi-supervised. Those tend to deliver less accurate results however.  Are the accurate alignments of an aligner such as the MFA used in the paper required? I would be interested in the quality-drop-off given less accurate alignments for training the phoneme recognizer, since the guided sampling seems quite complex. In other words, how precise does the guidance of the phoneme classifier have to be?\n- I find the level of control that the use of a duration predictor offers intriguing. I think it would be interesting to see how well the unconditional model can handle unnatural conditions, such as holding the same phoneme for multiple seconds. ", "summary_of_the_paper": "The authors propose a denoising diffusion probabilistic model (DDPM) to learn to produce natural spectrograms from noise without a condition. This enables them to train a generative model on unlabelled speech data. They show the effectiveness of their approach by inpainting masked out parts of a spectrogram and by showing off audio samples of unconditioned spectrogram babble, that has been vocoded to a waveform. They further propose a phoneme classification module that serves as conditioning signal for the DDPM during sampling in order to generate spectrograms that match a given phoneme sequence, turning the unconditional DDPM into a text-to-speech model, which the authors call Guided-TTS. This allows for all components of the model to be trained individually on different datasets, alleviating the need for large labelled datasets for TTS.", "summary_of_the_review": "This paper presents a novel method for TTS with convincing experiments and results. Although it has some weaknesses, I tend to see it marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes using unlabelled speech data for TTS by decoupling parts of the model.
However, all reveiwers agree that the technique is already known and the experimental results are not strong enough to make advantage of training on more data.
A reject.
Decision: Reject