High Fidelity Speech Synthesis with Adversarial Networks
Authors: Mikołaj Bińkowski
Jeff Donahue
Sander Dieleman
Aidan Clark
Erich Elsen
Norman Casagrande
Luis C. Cobo
Karen Simonyan
Venue:  (2020)
Keywords: texttospeech
speechsynthesis
audiosynthesis
gans
generativeadversarialnetworks
implicitgenerativemodels

Abstract: Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images. However, their application in the audio domain has received limited attention,
and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech. To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech.
Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes. The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced.  To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS - Mean Opinion Score), as well as novel quantitative metrics (Fréchet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS. We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator. Listen to GAN-TTS reading this abstract at https://storage.googleapis.com/deepmind-media/research/abstract.wav

Published as a conference paper at ICLR 2020
HIGH FIDELITY SPEECH SYNTHESIS WITH ADVERSARIAL NETWORKS
Mikolaj Birikowski*
Department of Mathematics
Imperial College London mi mikbinkowsk gma i com
Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, Karen Simonyan DeepMind
(jeffdonahin sedie em, a diddanclln danclark, eri che, ncasagrande, luisca, simonyan (roonyyan@google.coo
ABSTRACT
Generative adversarial networks have seen rapid development recent years and have led to remarkable improvements in generative modelling of images. How- ever, their application autoregressive models, such as WaveNet, remain the state of the art in genera- tive modelling of audio signals such as human speech. To address this paucity, we introduce GAN-TTS, Generative Adversarial Network for Text-to-Speech. Our architecture is composed of conditional feed-forward generator produc- ing speech audio, and an ensemble of discriminators which operate ran- dom windows of different sizes. The discriminators analyse the audio both in terms of general realism, well how well the audio corresponds to the ut- terance that should be pronounced. To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS Mean Opinion Score), as well as novel quantitative metrics (Fréchet DeepSpeech Distance and Ker- nel DeepSpeech Distance), which we find be well correlated with MOS. We show that GAN-TTS is capable of generating high-fidelity speech with natural- ness comparable to the state-of-the-an models, and unlike autoregressive models, it is highly parallelisable thanks to efficient feed-forward generator. Listen GAN-TTS reading this abstract at /ttp:////t/ttssssttttsstee www.::geo..gggepp....o.o
the audio domain has received limited attention, and
INTRODUCTION
The Sexteto-Speech (TTS) task consists the conversion text into speech audio. In recent years, the TTS field has seen remarkable progress, sparked by the development of neural autoregressive models for raw audio waveforms such as WaveNet (van den Oord al., 2016), SampleRNN (Mehri al.) 2017) and WaveRNN (Kalchbrenner et al.) 2018). A notable limitation of these models is that they are difficult to parallelise over time: they predict each time step of an audio signal in sequence, which is computationally expensive and often impractical. A lot of recent research neural models for TTS has focused on improving parallelism by predicting multiple time steps in parallel, e.g. using flow-based models (van den Oord et al.. 2018 Ping et al. 2019 Prenger al., 2019 Kim et al. 2019). Such highly parallelisable models are more suitable to run efficiently on modern hardware.
An alternative approach for parallel waveform generation would be to use Generative Adversar- ial Networks (GANs, Goodfellow 2014). GANs currently constitute one of the dominant paradigms for generative modelling of images, and they are able to produce high-fidelity samples
Work done DeepMind.

---

Published as a conference paper at ICLR 2020
that almost indistinguishable from real data. However, their application to audio generation tasks has seen relatively limited success so far. In this paper, we explore raw waveform generation with GANs, and demonstrate that adversarially trained fecd-forward generators are indeed able to synthesise high-fidelity speech audio. Our contributions are as follows:
We introduce GAN-TTS, a Generative Adversarial Network for text-conditional high- fidelity speech synthesis. Its feed-forward generator is a convolutional neural network, coupled with an ensemble of multiple discriminators which evaluate the generated (and real) audio based on multi-frequency random windows. Notably, some discriminators take the linguistic conditioning into account (so they can measure how well the generated au- dio corresponds to the input utterance), while others ignore the conditioning, and can only assess the general realism of the audio.
We propose a family of quantitative metries for speech generation based on Fréchet Incep- tion Distance (FID, Heusel et al., 2017) and Kernel Inception Distance (KID, Binkowski et al., 2018), where we replace the Inception image recognition network with the Deep- Speech audio recognition network. The code for our metrics is publicly available online")
We present quantitative and subjective evaluation of TTS-GAN and its ablations, demon- strating the importance of our architectural choices. Our best-performing model achieves MOS of 4.2, which is comparable to the state-of-the-arr WaveNet MOS of 4.4, and estab- lishes GANs viable option for efficient TTS.
RELATED WORK
2.1 AUDIO GENERATION
Most neural models for audio generation are likelihood-based: they represent an explicit probability distribution and the likelihood of the observed data is maximised under this distribution. Autore- gressive models achieve this by factorising the joint distribution into a product of conditional distri- butions (van den Oord et al. 2016 Mchri et al. 2017] Kalchbrenner et al., 2018 Arik et al.) 2017). Another strategy is to use invertible ferw-forward neural network model the joint density di- rectly (Prenger et al. 2019: Kim et al. 2019). Alternatively, an invertible feed-forward model can be trained by distilling an autoregressive model using probability density distillation (van den Oord eet al. 2018: Ping et al. 2019), which enables it to focus on particular modes. Such mode-seeking behaviour often desirable in conditional generation settings: we want the generated speech sig- nals to sound realistic and correspond to the given text, but we are not interested in modelling every possible variation that occurs in the data. This reduces model capacity requirements, because parts of the data distribution may be ignored. Note that adversarial models exhibit similar behaviour, but without the distillation and invertibility requirements.
Many audio generation models, including all of those discussed so far, operate in the waveform domain: they directly model the amplitude of the waveform as it evolves over time. This is in stark contrast to most audio models designed for discriminative tasks (e.g. audio classification): such models tend to operate on fineufrey representations of audio (spectrograms), which encode certain inductive biases with respect to the human perception of sound, and usually discard all phase information in the signal. While phase information often inconsequential for discriminative tasks, generated audio signals must have realistic phase component, because fidelity as judged by humans is severely affected otherwise, Because no special treatment for the phase component of the signal required when generating directly in the waveform domain, this is usually more practical.
Tacotron (Wang et al. 2017) and MelNet /Vasquez & Lewis 2019) constitute notable exceptions, and they use the Griffin-Lim algorithm (Griffin & Lim| 1984) to reconstruct missing phase informa- tion. which the models themselves do not generate. Models like Deep Voice 2 & 3 (Gibiansky et al.) 2017 Ping al. 2018) and Tacotron 2 (Shen et al. 2018) achieve a compromise by first generating a spectral representation, and then using a separate autoregressive model to turn it into a waveform and fill any missing spectral information. Because the generated spectrograms are imperfect, the waveform model has the additional task of correcting any mistakes. Char2wav (Sotelo et al. 2017) uses intermediate vocoder features in a similar fashion.
https::/ www.ttttth.... http::///g..hu....ooooii/iiiiiia nkwwkk http://www.ooooeeeeeeeeeeeecccae stances
2

---

Published as a conference paper at ICLR 2020
2.2 GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Networks (GANs, Goodfellow et al., 2014) form a subclass of implicit gen- erative models that relies on adversarial training of two networks: the generator, which attempts to produce samples that mimic the reference distribution, and the discriminator, which tries to dif- ferentiate between real and generated samples and, in doing so, provides useful gradient signal to the generator. Following rapid development, GANs have achieved state-of-the- -hrt results in im- age (Zhang et al. 2019 Brock al. 2019 Karras et al., 2019) and video generation (Clark et al. 2019), and have been successfully applied for unsupervised feature learning (Donahue et al.) 2017
Dumoulin et al., 2017a 7a Donahue Simonyan 2019), among many other applications.
Despite achieving impressive results in these domains, limited work has so far shown good perfor- mance of GANs in audio generation. Two notable exceptions include WaveGAN (Donahue et al.) 2019) and GANSymth (Engel et al. 2019). which both successfully applied GANs to simple datasets of audio data. The former is the most similar to this work in the sense that it uses GANs to generate raw audio; results were obtained for dataset of spoken commands of digits from zero to nine. The latter provides state-of-the-an results for a dataset of single note recordings from various musical in- struments (NSynth, |Engel et al. (017)) by training GANs to generate magnitude-phane spectrograms of the notes (which can easily be converted to waveforms, up to 64000 samples in length). Neekhaaa al. (2019) propose an adversarial vocoder model that is able to synthesise magnitude spectrograms from mel-spectrograms generated by Tacotron 2, and combine this with phase estimation using the Local Weighted Sums technique (Le Roux et al.] 2010).. (aato al. (2018) propose to enhance a parametric speech synthesiser with a GAN to avoid oversmoothing of the generated speech param- eters. Yamamoto et al. (2019) replace the probabilistic distillation loss Parallel WaveNet with an adversarial version. Tanaka et al. (2018) use variant of CycleGAN directly on raw waveforms for voice conversion.
GANs have seen limited application at large scale in non-visual domains so far. Two seconds of audio at 2 has a dimensionality of 48000, which is comparable to RGB images at 128 128 resolution. Until recently, high-quality GAN- -engraee images at such higher resolution were uncommon (Zhang et al. 2019) Karras et al. 2019), and it was not clear that training GANs at scale would lead to extensive improvements (Brock al. 2019).
Multiple diseriminators have been used in GANs for different purposes. For images, Denton et al. 2015); Zhang et al.) (2017) Karras et al. (2018) proposed to use separate discriminators for differ- ent resolutions. Similar approaches have also been used in image-to-image transfer (Huang et al.) 2018) and video synthesis (Saito & Saito 2018). Clark al. (2019), on the other hand, combine a di-discriminator that scores the video at lower resolution and a 2D-frame discriminator which looks individual frames. In adversarial feature learning, Donahue Simonyan (2019) combine outputs from three discriminators to differentiate between joint distributions of images and latents. Discriminators operating on windows of the input have been used in adversarial texture synthesis (Li & Wandd 2016) and image translation (Isola et al.) 2017
Zhu et al. 2017).
In parallel with our work, MelGAN (Kumar et al. 2019) used GANs for mel-spectrogram inver- sion, with multiple discriminators operating at different frequencies. MelGAN enforces the map- ping between input spectrogram conditioning and generated audio via an In reconstruction loss in discriminator feature space, in contrast with our work, which enforces the mapping adversarially using conditional discriminators. When combined with a Text2mel model, their generator achieves MOS of 3.7.
3 GAN-TTS
3.1 DATASET
Our text-to-speech models are trained on a dataset which contains high-fidelity audio of human speech with the corresponding linguistic features and pitch information. The linguistic features encode phonetic and duration information, while the pitch is represented by the logarithmic funda- mental frequency log Fu. In total, there are 567 features. We do not use ground-truth duration and
22 is commonly used frequency for speech, because the absence of frequencies above 12kHz does mcaningfully affect fidelity.
3

---

Published as a conference paper at ICLR 2020
pitch for subjective evaluation; instead use duration and pitch predicted by separate models. The dataset is formed of variable-length audio clips containing single sequences, spoken by profes- sional voice actor in North American English. For training, we sample 2 second windows (filtering out shorter examples) together with corresponding linguistic features. The total length of the filtered dataset is 44 hours. The sampling frequency of the audio is 24kH7, while the linguistic features and pitch are computed for 5ms windows (at 200Hz). This means that the generator network needs to learn how convert the linguistic features and pitch into raw audio, while upsampling the signal by a factor of 120. We apply a fa-law transform to account for the logarithmic perception of volume (see Appendix C))
3.2 GENERATOR
A summary generator G's architecture is presented in Table 2 in Appendix[AA2 The input to G a sequence of linguistic and pitch features at 200Hz, and its output is the raw waveform at 24kHz. The generator is composed of seven blocks (GBlocks, Figure[Taj, each of which is a stack of two residual blocks (He et al. 2016). As the generator is producing raw audio (e.g. 2s training clip corresponds 48000 samples), we use dilated convolutions (Yu & Koltun 2016) to ensure that the receptive field of G is large enough to capture long-term dependencies. The four kernel size-3 convolutions in each GBlock have increasing dilation factors: 1, 4, 8. Convolutions are preceded by Conditional Batch Normalisation (Dumoulin et al. 2017b). conditioned on the linear embeddings of the noise term 2 ~ (O, 1128) in the single-speaker case, or the concatenation of 2 and a one- hot representation of the speaker IDD in the multi-speaker case. The embeddings each BatchNorm instance. A GBlock contains two skip connections, the first of which performs upsampling if the output frequency is higher than the input, and also contains size-1 convolution if the number of output channels is different from the input. GBlocks 3-7 gradually upsample the temporal dimension of hidden representations by factors of 2, 2, 3,5, 5, while the number of channels is reduced by GBlocks 3, 6 and 7 (by a factor of 2 each). The final convolutional layer with Tanh activation produces a single-channel audio waveform.
to a sequence
different for
3.3 ENSEMBLE OF RANDOM WINDOW DISCRIMINATORS
Instead of a single discriminator, we use an ensemble of Random Window Diseriminators (RWDs) which operate on randomly sub-sampled fragments of the real generated samples. The ensem- ble allows for the evaluation of audio in different complementary ways, and is obtained by taking a Cartesian product of two parameter spaces: (i) the size of the random windows fed into the dis- criminator; (ii) whether a discriminator is conditioned on linguistic and pitch features. For example, in our best-performing model, we consider five window sizes (240, 480, 960, 1920, 3600 samples), which yields 10 discriminators in total. Notably, the number of discriminators only affects the train- ing computation requirements, as inference time only the generator network is used, while the discriminators are discarded. However, thanks to the use of relatively short random windows, the proposed ensemble leads to faster training than conventional discriminators.
Using random windows of different size, rather than the full generated sample, has a data augmen- tation effect and also reduces the computational complexity of RWDs, as explained next. In the first layer of each discriminator, we reshape (downsample) the input raw waveform to a constant temporal dimension w 240 by moving consecutive blocks of samples into the channel dimension, i.e. from wwk, 1] to [w, ktl. where l is the downsampling factor (e.g. k = 8 for input window size 1920). This way, all the RWDs have the same architecture and similar computational complexity despite different window sizes. We confirm these design choices experimentally in Section S
The conditional discriminators have access to linguistic and pitch features, and can measure whether the generated audio matches the input conditioning. This means that random windows in conditional discriminators need to be aligned with the conditioning frequency to preserve the correspondence between the waveform and linguistic features within the sampled window. This limits the valid sam- pling to that of the frequency of the conditioning signal (200Hz, or every 5ms). The unconditional discriminators, on the contrary, only evaluate whether the generated audio sounds realistic regardless of the conditioning. The random windows for these discriminators are sampled without constraints at full 24kHz frequency, which further increases the amount of training data.
4

---

Published as a conference paper at ICLR 2020
More formally, let À 120 denote the frequency ratio between waveform and linguistic features and let (+ *; 0) and D (+; 8*) be conditional and unconditional discriminator networks parametrized
by and f', respectively, which downsample the waveform input * by a factor k We define conditional and unconditional RWDs stochastic functions:
cRWDR (z, c; = D (Fjj+wks G/X:Gtwk)/AAO), j~
wk})
(1)
uRWD k, (x;
D (Tig+wkk 8'),
((0,, N wk]),
(2)
where a and c respectively the waveform and linguistic features and di: = (at, 01+1.-..,, ",,1)))
denotes vector slice.
The final ensemble discriminator combines different RWD's:
(x,
Σ
(R, + uR; (x;
0" = U(e (8%)
k5/11244881553
(3)
Algorithm iin Appendix D shows pseudocode for computation of RWD*. In Section we describe other combinations of RWDs as well as a full, deterministic discriminator which we used in our ablation study.
3.4 DISCRIMINATOR ARCHITECTURE
The full discriminator architecture shown Figure 2 The discriminators consists of blocks (DBlocks) that are similar to the GBlocks used in the generator, but without batch normalisation. The architectures of standard and conditional DBlocks are shown in Figures Tb and respectivvll The only difference between the two DBlocks is that in the conditional DBlock, the embedding of the linguistic features is added after the first convolution. The first and the last two DBlocks do not downsample (i.e. keep the temporal dimension fixed). Apart from that, we add at least two downsampling blocks in the middle, with downsample factors depending on k, so as to match the frequency of the linguistic features (see Appendix [A.2 for details). Unconditional RWDs are com- posed entirely of DBlocks. In conditional RWDs, the input waveform is gradually downsampled by DBlocks, until the temporal dimension of the activation is equal to that of the conditioning, which point conditional DBlock is used. This joint information is then passed to the remaining DBlocks, whose final output average-pooled to obtain scalar. The dilation factors in the DBlocks' convolu- tions follow the pattern 1, 2, 1, 2.... unlike the generator, the discriminator operates on a relatively small window, and we did not observe any benefit from using larger dilation factors.
4 EVALUATION
We provide subjective human evaluation of our model using Mean Opinion Scores (MOS), well as quantitative metrics.
4.1 MOS
We evaluate our model on a set of 1000 sentences, using human evaluators. Each evaluator was asked mark the subjective naturalness a sentence on a 1-5 Likert scale, comparing to the scores reported by van den Oord et al. (2018) for WaveNet and Parallel WaveNet.
Although our model was trained generate second audio clips with the starting point not neces- sarily aligned with the beginning of a sentence, we are able to generate samples of arbitrary length. This is feasible due to fully convolutional nature of the generator and carried out using a con- volutional masking trick, detailed in Appendix A.I Human evaluators scored full sentences with length of up to 15 seconds.
'Here we only require that * E ok and E I (wkk for window size Since we consider fully-convolutinss architectures with average pooling after the top layer, DÊ and D do not depend the value of We descrihe architecture details of D and D for l E (1,2, 4, 8,8, 15) in Appendix A.2 A.2
5

---

Published as a conference paper at ICLR 2020
Downsample
Conv
Conv Mer
ReLU
Conv
Conv Aneec 2et,
ReLU
Conv Hareed is Kung lie
BatchNorm
Linear
Conv
ReLU
ReLU
Downsample
BatchNorm
Linear
(b) Conditional DBlock
Conv
Upsample
Downsample
Conv
Conv
ReLU
Conv
BatchNorm
Linear
Conv
ReLU
Haeeii Ai
Conv
Upsample
ReLU
ReLU
BatchNorm
Linear
Downsample
(a) GBleck
(c) DBlock
Figure 1: Residual blocks used in the model. Convolutional layers have the same number of input and output channels and no dilation unless stated otherwise. h hidden layer representation, / linguistic features, z noise vector, m channel multiplier, m = for downsampling blocks (i.e. their downsample factor is greater than 1) and m = 1 otherwise, M- G's input channels, M = 2N in blocks 3, 6, and M = N otherwise; size refers to kernel size.
4.2 SPEECH DISTANCES
We introduce family of quantitative metrics for generative models of speech, which include the unconditional and conditional Frécher DeepSpeech Distance (FDSD, cFDSD) and Kernel Deep- Speech Distance (KDSD, cKDSD). These metries follow common metries used in evaluation of GANs for images, Fréchei Inception Distance (FID, Heusel et al. 2017) and Kernel Inception Dis- lance (KID, Binkkwwki al. 2018).
FID and KID compute the Fréchet distance and the Maximum Mean Discrepancy (MMD, Gret- al., 2012) respectively between representations of reference and generated distributions ex- tracted from pre-trained Inception network (Szegedy et al. 2016). To obtain analogous metrics for speech, we extract the features from an open-source implementation of an accurate speech recogni- tion model, DeepSpeech2 (Amodei et al., 2016). Specifically, we use the implementation available the NVIDIA OpenSeq2Seq library (Kuchaiev et al.) 2018) and extract features from the last layer, whose output is used in the CTC loss during training. We use representations in the resulting feature space to compute the Fréchet distance and MMD (See Appendix [B.I for details).
We note that Kilgour et al. (2019) proposed similar metric, Fréchet Audio Distance. This metric uses network trained for audio event classification on the AudioSet dataset (Gemmeke et al. 2017) feature extractor, whereas we network that trained for speech recognition.
As conditioning plays a crucial role in our task, we compute two variants of these metrics, con- ditional (cFDSD, cKDSD) and unconditional (FDSD, KDSD). Both Fréchet and Kernel distance provide scores with respect to a reference real sample and require both the real sample and the generated to be independent and identically distributed. Assume that variables real and G

---

Published as a conference paper at ICLR 2020
uRWD2
cRWD
15
Linear
Avg. pool
Linear
DBlock x2
Avg. pool
Cond. DBlock
DBlock x2
DBlock
DBlock
16
800M
256
60
400Kz
256
downsomple
downsomple:2
DBlock
DBlock
48
128
120
128
downsomple.
dhampppple-
DBlock
DBlock
Reshape
Reshape
240 72440
240 Z600HY
15
24182
3600
24042
Sraguisticfeaturre
arbtraar'
campain aligned
samphig
Figure 2: Multiple Random Window Discriminator architecture. The discriminator combines out- puts from 5 unconditional (uRWDs, left) and 5 conditional (cRWDs, right) discriminators; one of each group is detailed the diagram. The number of downsampling blocks is fixed for uRWDs and depends the input window size wk for cRWDs, see Table 3 Next to each block present the dimensionality and frequency of its outputs; ch. number of output channels.
are drawn from the real and and generated distributions, while e drawn from the distribution of linguistic features. In the conditional case, cFDSD and cKDSD compute distances between condi- tional distributions pxx0pp
G |c) and p(xreal c). In the unconditional case, FDSD and KDSD compare p(x() and p(x
Both metrics are estimated using 10,000 generated and reference samples, drawn independently with the same (in the conditional case), independent (in the unconditional case) linguistic features. This procedure is detailed in Appendix B.3
The main reason for using both Fréchet and Kernel distances is the popularity of FID in the image domain, despite the issue of its biased estimator, as shown by Binkowski et al. (2188). Thanks to the availability of unbiased estimator of MMD, this issue does not apply to kernel-based distances. For instance, they yield zero values for real data, which allows comparison in the conditional case. We give more details on these distances Appendix B.2
EXPERIMENTS
In this section we discuss the experiments, comparing GAN-TTS with WaveNet and carrying out ablations that validate our architectural choices.
As mentioned in Section 3, the main architectural choices made in our model include the use of multiple RWDs, conditional and unconditional, with number of different downsampling factors. We thus consider the following ablations of our best-performing model:
1. full-input discriminator FullD Df.
2. single conditional RWD: cRWD,,

---

Published as a conference paper at ICLR 2020
3. multiple conditional RWDs: cRWD (1,2,4,8.16) ExE(1.2.18.33) cRWDk:
4. single conditional and single unconditional RWD: cRWD, + uRWD),
5. five independent cRWDs and uRWDs:
(cRWD, jRWD1) x5 (z, c) := > (RWD1(I, c, 0.) + uRWD (z; 8(),
6. 10 RWDs without downsampling but with different window sizes:
110.2.200 = (1.A.....)))) (cRWD1, uRWD1,2404)
7. 10 RWDs with longer window: RWD
All other parameters of these models were the same as in the proposed onc. In Appendix we present details of the hyperparameters used during training well as pseudocode for training GAN- TTS.
5.1 RESULTS
KDSD x10
cKDSD X x10
model
MOS
FDSD
cFDSD
natural speech
4.55 + 0.075
0.161
N/A
0
WaveNet, van den Oord 2016
4.41 = 0.069
Parallel WaveNer, van den Oord 20188
4.41 + 0.078
FullD
1.889 - 0.057
4.51
4.46
785
782
cRWD
3.394 = 0.058
0.362
0.217
35.2
30.9
cRWD (1.2.4.8.15)
3,498 = 0.059
0.398
0.284
42.1
37.9
cRWD uRWD;
3.502 = 0.057
0.259
0.144
16.6
12.3
(cRWD, + j K5
3.526 + 0.054
0.194
0.073
5.59
1.34
RWD 1,20x (1,2,4..155)
4.154 + 0.050
0.184
0.061
3.73
0.54
RWD 480
4.195 = 0.045
0.193
0.069
5.28
0.98
GAN-TTS (RWD')
4.213 = 0.046
0.184
0.060
3.84
0.37
Table I: Results from prior work, the ablation study and the proposed model. Mean opinion scores for natural speech, WaveNet and Parallel WaveNet are taken from van den Oord et al. (2018) and are not directly comparable due to dataset differences. For natural speech present estimated FDSD non-zero due to the bias of the estimator and theoretical values KDSD and cKDSD. cFDSD unavailable; see Appendix B.2
Table I presents quantitative evaluations of the proposed model, together with benchmarks and other variants of GAN-TTS that we considered in this work.
Our best model achieves worse yet comparable scores to the strong baselines, WaveNet and Parallel WaveNet. This performance, however, has not yet been achieved using adversarial techniques and still very good, especially when compared to parametric text-to-speech models. These results however not directly comparable due to dataset differences; for instance WaveNet and Parallel WaveNet were trained on 65 hours of data, a bit more than GAN-TTS.
Our ablation study confirms the importance of combining multiple RWDs. The deterministic full discriminator achieved the worst scores. All multiple-RWD models achieved better results than a single cRWD;; all models that used unconditional RWDs were better than those that did not. Com- paring 10-discriminator models, it is clear that combinations of different window sizes were bene- ficial, as a simple ensemble of 10 fixed-size windows was significantly worse. All three 10-RWD models with varying discriminator sizes achieved similar mean opinion scores, with the downsam- pling model with base window size 240 performing best.
We also observe noticeable correlation between human evaluation scores (MOS) and the proposed metries, which demonstrates that these metrics are well-suited for the evaluation of neural audio synthesis models.
5.2 DISCUSSTON
Random window discriminators. Although it is difficult to say why RWDs work much better than the full discriminator, conjecture that this is because of the relative simplicity of the dis-
8

---

Published as a conference paper at ICLR 2020
tributions that the former must discriminate between, and the number of different samples we draw from these distributions. For example, the largest window discriminators used in our best model discriminate between distributions supported on IR 3600 and there respectively 371 and 44,401 different windows that can be sub-sampled from a 2s clip (real or generated) by conditional and unconditional RWDs of effective window size 3600. The full discriminator, the other hand, always sees full real or generated examples sampled from a distribution supported on R 48000
Computational efficiency. Our Generator has a larger receptive field (590ms, i.e. 118 steps at the frequency of the linguistic features) and three times fewer FLOOPs (0. 60.64 MFLOP/sample) than Parallel WaveNet (receptive field size: 320ms, 1.97 MFLOP/sample). However, the discriminators used in our ensemble compare windows of shorter sizes, from 10ms to 150ms. Since these windows are much shorter than the entire generated clips. training with ensembles of such RWDs is faster than with FulID. In terms of depth, our generator has 30 layers, which is half of Parallel WaveNet's ss while the depths of the discriminators vary between 11 and 17 layers, as discussed in Appendix A.2
Stability. The proposed model enjoyed very stable training, with gradual improvement of subjec- tive sample quality and decreasing values of the proposed metrics. Despite training for as many as million steps, we have not experienced model collapses often reported in GAN literature and studied in detail by Brock et al. (2019).
CONCLUSION
We have introduced GAN-TTS, a GAN for raw audio text-to-speech generation. Unlike state-of- the-art text-to-speech models, GAN-TTS is adversarially trained and the resulting generator a feed-forward convolutional network. This allows for very efficient audio generation, which is im- portant in practical applications. Our architectural exploration lead to the development of a model with an ensemble of unconditional and conditional Random Window Discriminators operating at different window sizes, which respectively assess the realism of the generated speech and its cor- respondence with the input text. We showed in ablation study that each of these components is instrumental to achieving good performance. We have also proposed a family of quantitative metrics for generative models of speech: (conditional) Fréchet DeepSpeech Distance and (conditional) Ker- nel DeepSpeech Distance, and demonstrated experimentally that these metrics rank models in line with Mean Opinion Scores obtained through human evaluation. The metries publicly available for machine learning community, as is the DeepSpeech recognition model they are based on. Our quantitative results well as subjective evaluation of the generated samples showcase the feasibility of text-to-speech generation with GANs.
ACKNOWLEDGMENNT
We would like to thank Aäron van den Oord, Andrew Brock and the rest of the DeepMind team.
REFERENCES
Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jing- dong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep Speech 2: End-to-end speech recognition in English and Mandarin. ICML., 2016.
Sercan Ô Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al. Deep Voice: Real-time neural text-to-speech. In ICML, 2017.
Mikolaj Binkowski, Dougal Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In ICLR, 2018

---

Published as a conference paper at ICLR 2020
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. In JCLR, 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Efficient video generation on complex datasets. 1rX.v:1907.0657, 2019.
Emily L Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a Laplacian pyramid of adversarial networks. In NewrlPS, 2015.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In ICLR, 2019.
Jeff Donahue and Karen Simonyan.
Large scale adversarial representation learning.
arXiv:1907.02544, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. ICLR, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. In ICLR, 2017a.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In ICLR, 2017b.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dicleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with WaveNet autoencoders. In ICML, 2017.
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Guirajani, Chris Donahue, and Adam Roberts. GANSynth: Adversarial neural audio synthesis. In ICLR, 2019.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. IEEE ICASSP 2017, New Orleans, LA, 2017.
Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep Voice 2: Multi-speaker neural text-to-speech. In NeurIPS, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
A. Gretton, K. Borgwardt, Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. JMLR, 2012.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time Fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1984.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. In NeurlPS, 2017.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML,, 2015.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. sange-to-image translation with conditional adversarial networks. In CVPR, 2017.
10

---

Published as a conference paper at ICLR 2020
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In ICML, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICI.R, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet audio distance: A metric for evaluating music enhancement algorithms. In Interspeech, 2019.
Sungwon Kim, Sang-Gil Lee, Jongyoon Song, Jachyeon Kim, and Sungroh Yoon. FloWaveNet: A generative flow for raw audio. In ICML, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Oleksii Kuchaiev, Boris Ginshurg, Igor Gitman, Vitaly Lavrukhin, Carl Case, and Paulius Micikevi- cius. OpenSeq2Seq: Extensible toolkit for distributed and mixed precision training of sequence- to-sequence models. In NLP-OSS, 2018.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, and Aaron Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In NeurlPS, 2019.
Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono, and Shigeki Sagayama. Fast signal recon- struction from magnitude STFT spectrogram based on spectrogram consistency. In DAFx, 2010.
Chuan Li and Michael Wand. Precomputed real-time texture synthesis with Markovian generative adversarial networks. In ECCV, 2016.
Jae Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv: 2017.
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. SampleRNN: An unconditional end-to-end neural audio generation model. In ICLR, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Paarth Neekhara, Chris Donahue, Miller Puckette, Shlomo Dubnov, and Julian McAuley, Expediting TTS synthesis with adversarial vocoding. In Interspeech, 2019.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep Voice 3: 2000-speaker neural text-to-speech. In ICLR, 2018.
Wei Ping, Kainan Peng, and Jitong Chen. ClariNet: Parallel wave generation in end-to-end text-to- speech. In ICLR, 2019.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. WaveGlow: A flow-based generative network for speech synthesis. In ICASSP, 2019.
Masaki Saito and Shunta Saito. TGANv2: Efficient training of large models for video generation with multiple subsampling layers. 1.1.025. 2018.
Y. Saito, S. Takamichi, and H. Saruwatari. Statistical parametric speech synthesis incorporating generative adversarial networks. JEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 26(1):84-96 Jan 2018.
Andrew Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR, 2014.

---

Published as a conference paper at ICLR 2020
Jonathan Shen, Ruoming Pang. Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan, et al. Natural TTS synthesis by condi- tioning WaveNet on Mel spectrogram predictions. In ICASSP. 2018.
Jose Sotelo, Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio. Char2Wav: End-to-end speech synthesis. In ICLR, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception architecture for computer vision. In CVPR, 2016.
Kou Tanaka, Takuhiro Kaneko, Nobukatsu Hojo, and Hirokazu Kameoka. Synthetic-to-nar -o-tttra- speech waveform conversion using cycle-consistent adversarial networks. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 632-639. IEEE, 2018.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv: (0499, 2016.
Aäron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg. Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet:
Fast high-fidelity speech synthesis. In ICML, 2018.
Sean Vasquez and Mike Lewis. MelNet: A generative model for audio in the frequency domain. 10X::0990601083 2019.
Yuxuan Wang. RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Probability density distillation with gen- crative adversarial networks for high-quality parallel waveform generation. arXiv preprint 1oXi::190..044777 2019.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adver- sarial networks. In ICCV, 2017.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, 2019.
Jun Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
12

---

Published as a conference paper at ICLR 2020
non-zero padding
batch
1st conv, k
2nd conv,
affected non-padding
time
1st conv, ks=3
n-th conv, ks=3
n+ 1-th conv
input mask
input mask
Figure 3: Masking scheme for sampling different-length samples. Top: processing a batch of sam- ples of different lengths padded with zeros leads to interference between padding and non-padding after the second convolution, not seen during training. Bottom: masking after each convolution ensures that the meaningful input seen by each layer is padded with zeros only.
A ARCHITECTURE DETAILS
A.1 MASKING CONVOLUTIONS TO GENERATE LONGER SAMPLES
Since our generator is a fully-convolutional network, theory it is capable of generating samples of arbitrary length. However, since deep learning frameworks usually require processing fixed-size samples in batches for efficiency reasons, our inputs different lengths need to be zero-padded to fit in a fixed-size tensor. Convolutional layers, including the ones used in our model, often pad their inputs to create outputs the desired dimensionalityy, hence we only need to ensure that the padded part of the imput tensors to all layers is always zero. As shown in Figure 3 this would not normally be the case after the second convolutional layer, since convolutions (with kernel sizes greater than one) would propagate non-zero values outside the border between meaningful input and padding. A simple way to address this issue is masking, i.e. multiplying the input by a zero-one mask tensor, directly before each convolutional layer. This enables batched sampling of utterances of different length, which is efficient on many hardware platforms, optimised for batching.
A.2 GENERATOR AND DISCRIMINATOR DETAILS
layer/input
freq.
ch
linguistic features,
400
200Hz
567
conv, kernel size
400
200Hz
768
GBlock
400
200Hz
768
GBlock
400
200Hz
768
GBlock, upsample x2
800
400Hz
384
GBlock, upsample x2
1600
800Hz
384
GBlock, upsample x2
3200
1600HZ
384
GBlock, upsample x3
9600
4800HZ
192
GBlock, upsample x5
48000
24kHz
96
conv, kernel size
48000
24kHz
Tanh
Table Architecture of GAN-TTS's Generator. denotes the temporal dimension, while ch denotes the number of channels. The rightmost three columns describe dimensions of the output of the corresponding layer.
In Table 2 we present the details of Generator architecture. Overall, the generator has 30 layers, most of which parts of dilated residual blocks.
the numbers of residual DBlocks and downsample factors these blocks for different initial downsample factors of RWDs.
13

---

Published as a conference paper at ICLR 2020
D
D
factors
num. blocks
depth
factors
num. blocks
depth
1, 2,1
17
1,5, 3,1,
5
11
1, 3,
15
1,1,3,11
5
1, 1.5.3,2.1.1
6
13
1,5,3.1,1
5
11
1,5,3,1,1
11
1,5,3,1,1
5
15
1,2,2,2,111
6
13
1,2,2,1,1
5
11
Table 3: Downsample factors discriminators for different initial stride values k.
All conditional discriminators eventually add the representations of the waveform and the linguistic features. This happens once the temporal dimension of the main residual stack is downsampled to the dimension of the linguistic features, i.e. by a factor of 120. Downsampling carried out via an initial reshape operation (by factor &: varying per RWD) and then in residual blocks, whose downsample factors are prime divisors of 120/k; /k, in decreasing order. For unconditional discriminators, we use only the first two largest prime divisors of 120/k.
Algorithm in Appendix D presents pseudocode for forward pass through RWDensemble.
B DEEPSPEECH DISTANCES DETAILS
B.1 DEEPSPEECH2
Our evaluation metries extract high-level features from raw audio using the pre-trained DeepSpeech2 model from the NVIDIA OpenSeq2Seq library (Kuchaiev et al. 2018). Let wo 480 be 20ms window of raw audio at 24kHz, and let f R - 1o00 be function that maps such win- dow through the DeepSpeech2 network up to the 1600-dimensional output of the layer labeled ForwardP ass ds2 www.dennc//eeeeeeeeee Reshape
We use default values for all settings of the DeepSpeech2 model; f also includes the model's preprocessing layers.
For audio clip a e R 100w we define
198
Da 199
Σ
/2tm///5//2tt/ 1600
where a;:j = (at, ai+1...., a,+1...., 11 a vector slice.
The function DS therefore computes 1600 features for each 20ms window, sampled evenly with 10ms overlap, and then takes the average of the features along the temporal dimension.
B.2 METRICS IN DISTRIBUTION SPACE
Given samples X E Rad and Raxd, where the representation dimension, the Fréchet distance and MMD can be computed using the following estimators:
Fréchet (X, Y) =I|ux - px 112 Tr
Σx + Ey 2x 2/22
(5)
MMD (X, YYY) m(m - 1) Im
Σ
x,,, + n(n 1)
Σ A(Y,Y,))
151,55
$$1
14)
m
k(X, Yj),
where μxx fx. fy and Ex, Σy are the means and covariance matrices of X and Y respectively, while k: Rd x E2 - IR is positive definite kernel function. Following Birikowski et al. (2018) use the polynomial kernel
k(x,,) v) (3=² y 1)²
14

---

Published as a conference paper at ICLR 2020
Estimator (5) has been found be biased (Binkowski et al., 2018 For this reason, FID estimates for real data (i.e. when X and Y are both drawn independently from the same distribution) KID, however, does not suffer from this issue thanks to the use of the unbiased estimator (6). These properties also apply the proposed SpepSpeech metrics.
for large sample sizes.
positive,
though the thcoretical value of such metric is zero.
The lack bias an estimator is particularly important for establishing scores on real data for conditional distances. In our conditional text-to-speech setting, we cannot sample independent real samples with the same conditioning, and for this reason we cannot estimate the value of cFDSD for real data, which would be positive due bias of estimator (5). For cKDSD, however, we know that such an estimator would have given values very close to zero, if we had been able to evaluate it on two real i. samples with the same conditioning.
B.3 DISTANCE ESTIMATION
Let G and DS represent the generator function and function that maps audio to DeepSpeech2 features as defined in Eq. Let
X- {DS (G(c,, 2))) Xeal (D5(
X
(D
p(xreal, ,,,, - I,.., 2N are jointly sampled real examples and linguistic fea- tures, and 2; itd N(O, 1). In the conditional case, we use the same conditioning in the reference and generated samples, comparing conditional distributions p(x[(c) and p(x"eal(c))
where (Ii, c;) sid
cFDSD (p(x G [c), p(x (ex))))l)))) Fréchet (x Xxrr)),
(9)
cKDSD (p(x 9 (c), p(x = MMD (X x!xo)),
(10)
where Fréchet and MMD are estimators of the Fréchet distance and MMD defined in Eq. and 6 respectively.
In the unconditional case, we compare p(x²) and p(xreat):
FDSD (p(x)), ((xx)))(x"eo))) ((xx)a))) = Fréchet (x,xx)l))
(11)
KDSD (p(x p(x m(a)) MMD (x5, reeal)
(12)
C μ--LAW PREPROCESSING
Many generative models of audio use the p-law transform to account for the logarithmic perception of volume. Although p-laww is typically used in the context of non-uniform quantisation, the transform without the quantisation step as our model operates in the continuous domain:
In(1 + plz²)
F(x:) sgn
(13)
In + p))
where [-1,1] and μ = 2 - 255 for 8-bit encoding or μ 2 65, 535 for 16-bit encoding.
Our early experiments showed better performance of models generating y-law transformed audio than non-transformed waveforms. We used the 16-bit transformation.
D TRAINING DETAILS
We train all models with single discriminator step per generator step, but with doubled learning rate: 10 for the former, compared to 5 x 10-5 for the latter. We the hinge loss (Lim & Ye 2017 batch size of 1024 and the Adam optimizer (Kingma & Ba 2015) with hyperparameters 3 0, B2 0.999.
Following Brock et al. (2019), we use spectral normalisation (Myyato et al. 2018) and orthogonal initialisation (Saxe et al. 2014 in both the generator and discriminator(s), and apply off-diagonal
15

---

Published as a conference paper at ICLR 2020
Algorithm TTS-GAN
require: N waveform length, waveform-coonttionin frequency ratio, w base window size, Nateps number of training steps, Rhateh batch size,
nJ. 7/6 discriminator and generator learning rates.
1: procedure RWD* c,
Compute value of RWDs ensemble
2:
U. 4- 0
3:
for (1,2,4,8, 15) do j - 410, 22,, , wkk))
conditional RWDs D sample random index D assign the slice of waveform assign the corresponding slice of conditioning
4:
5:
6:
7:
v U- V D
8:
end for
9:
for
21 4, 15) do j + 41 ((0, 1,... N wk})
unconditional RWDs D sample random index D assign the slice of waveform
10:
3 Ijj+uu
V v- D (i; 0%)
13: end for
return
15: end procedure
1: procedure TTS-GANÇS (*)
Main training algorithm
2: for s := 1 to Nateps do
3: #discriminator step
lid 4: (z),,c), (Insatch (osatch) ~ p(x, c)
> sample batch of iid real waveforms and corresponding linguistic features sample batch of iid linguistic features
5:
d (g...., Saawech
sid p(c)
6:
(21...., Zmoatch
sid
N(0, 1128)
P> sample noise
7:
x' + G(d', 2; 2))
generate fake waveforms
8:
lo maatch Li=1
Hbatch
((1 RWD' (r, (*)++ (1 RWD" (1¹ ce (*))+)
9:
compute Ds' loss b update Ds
10:
0" + Adam( (o
11:
#generator step
12:
+ (c...... de Aowtth
Hid
p(c)
sample batch of iid linguistic features
13:
- (Z1,..., Znoateh x' G(d,22 22735) 16 + 1
sid N N( 1128)
b sample noise generate fake waveforms b compute G's loss
15:
RWD
Pbarch
1=1
16:
1/ + Adam( ulG, 77)
A update G
17:
end for return
18:
19:
end procedure
orthogonal regularisation (Brock et al. 2016 2019) and exponential moving averaging to the gen- erator weights with a decay rate of 0.9999 sampling. We also use cross-replica BatchNorm (Toffe & Szegedy (2155), which aggregates batch statistics from all devices across which the batch split and standing statistics during sampling. The latter means that we accumulate batch statisties from 100 forward passes through the generator before the actual sampling takes place, allowing for inference at arbitrary batch sizes.
In fact, accumulating standing statistics makes the BatchNorm layers in the generator independent of any characteristics of the samples produced during inference. This technique is thus vital for sampling audio of unspecified length: producing samples that are longer than those used during training typically requires using a smaller batch size, with partially padded samples (See Appendix A.T. These smaller batches would naturally have different statisties than the batches used during
16

---

Published as a conference paper at ICLR 2020
cFDSD
0.2
step
Figure 4: Learning curve for the GAN-TTS model in terms of cFDSD.
training. We trained models on Cloud TPU v3 Pods with data parallelism 128 replicas for million generator and discriminator updates, which usually took up to 48 hours.
In Algorithm present the pseudocode for training GAN-TTS.
Figure presents the stable and gradual decrease of cFDSD during training.
17


Reviews:
[{"review_id": "rklAPQJOKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I want thank the authors for solving this long-standing GAN challenge in raw waveform synthesis. With all due respect, previous GAN trials for audio synthesis are inspiring, but their audio qualities are far away from the state-of-the-art results. Although the speech fidelity of GAN-TTS is still worse than WaveNet and Parallel WaveNet from the posted sample, it has begun to close the significant performance gap that has existed between autoregressive models and GANs for raw audios. Overall, this is a very good paper with significant contributions to the filed.\n\nDetailed comment:\n\n1, In WaveNet, the conditional features (linguistic / mel-spectrogram) are added as bias terms in the convolutional layers. Did the authors tried this alternative architecture for the generator, which uses the white noisy z as network input (similar as flow-based models, e.g., Parallel WaveNet) and the conditional features as bias term in the convolutional layers? \n\n2, Could the authors comment the importance of serval architecture choices in this work? From Table 1, it seems to me that the ensemble of random window discriminators is the most important (perhaps the only important) contributing factor for the success. For example, the MOS score was boosted from 1.889 to 4.213 by replacing a single full discriminator to the ensemble of RWDs.\n\n3, The notations in Eq. (1) and (2) are messy. Although I can figure their meaning from the context, one may clarify certain notations if they appear at the first time. \n\n4, The stable training (NO model collapses) is pretty impressive. Could the authors shed some light on the potential reason? Does the ensemble of RWD regularizes the training? What's your experience for training FullD (does not have random window ) and cRWD_1 (only has one random window discriminator)? Are they still very stable? Also, could the authors comment on the importance of large batch size -- 1024 for stable training of GAN-TTS? \n\n5, Although there is a notable difference, one may properly mention previous work Yamamoto et al. (2019), which uses GAN as an auxiliary loss within ClariNet and obtains high-fidelity speech ( https://r9y9.github.io/demos/projects/interspeech2019/ ).   \n\nYamamoto et al. Probability Density Distillation with Generative Adversarial Networks for High-Quality Parallel Waveform Generation. 2019.\n\n\n=== update === \n\nThank you for the detailed response.  \n2,  Thanks for the elaboration.    \n4,  It would be very interesting to see an analysis of model stability with smaller batch sizes. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyg3Zr60FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper puts forth adversarial architectures for TTS. Currently, there aren't many examples (e.g. Donahue et al,  Engel et al. referenced in paper) of GANs being used successfully in TTS, so this papers in this area are significant. \n\nThe architectures proposed are convolutional (in the manner of Yu and Koltun), with increasing receptive field sizes taking into account the long term dependency structure inherent in speech signals. The input to the generator are linguistic and pitch signals - extracted externally, and noise. In that sense, we are working with a conditional GAN. \n\nI found the discriminator design very interesting. As the comment below notes, it is a sort of patch GAN discriminator (See pix2pix, and this comment from Philip Isola - https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/39) and that is could be quite significant in that it classifies at different scales. In the image world, having a single discriminator for the whole model would not take into account local structure of the images. Likewise, perhaps we can imagine something similar in the case of audio at varying scales - in fact, audio dependencies are even more long range. That might be one reason why the variable window sizes work here. \n\nThe paper also presents to image analogues for metrics based on FID and the KID, with the features being taken from DeepSpeech2. \n\nI found the speech sample presented very convincing. In general, the architectures are also presented quite clearly, so it seems that we might be able to reproduce these experiments in our own practice. It is also promising that producing good speech could be achieved by a non-autoregressive or attention based architecture.\n\nThe authors mention that they hardly encounter any issues with training stability and mode collapse. Is that because of the design of the multiple discriminator architecture?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlJ-YsaKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to enable GAN based TTS in the time domain with the careful designs of the (non-autoregressive) generator and discriminator. There have been various trials of GAN-TTS but not so many success and I'm glad to hear that the proposed method seems to enable GAN-TTS with fast inference thanks to the non-autoregressive property. The method also proposes new objective measures inspired by the image recognition network based on the high-level features generated by end-to-end ASR, which is also another important contribution of this paper. \n\nMy concern for this paper is reproducibility. Although I really appreciate the authors' efforts on providing implementational details in the appendix, the code and data do not seem to be publicly available, and I'm expecting that the implementation of this technique is relatively hard due to their complex designs of the generator and discriminator. Apart from that, the paper is well written overall by well describing the trend of GAN studies in the image processing and the application of such image processing oriented GAN techniques to TTS.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors design a GAN-based text-to-speech synthesis model that performs competitively with state-of-the-art synthesizers.  The reviewers and I agree that this appears to be the first really successful effort at GAN-based synthesis.  Additional positives are that the model is designed to be highly parallelisable, and that the authors also propose several automatic measures of performance in addition to reporting human mean opinion scores.  The automatic measures correlate well (though far from perfectly) with human judgments, and in any case are a nice contribution to the area of evaluation of generative models.  It would be even more convincing if the authors presented human A/B forced-choice test results (in addition to the mean opinion scores), which are often included in speech synthesis evaluation, but this is a minor quibble.
Decision: Accept (Talk)