High Fidelity Speech Synthesis with Adversarial Networks

1 INTRODUCTION . The Text-to-Speech ( TTS ) task consists in the conversion of text into speech audio . In recent years , the TTS field has seen remarkable progress , sparked by the development of neural autoregressive models for raw audio waveforms such as WaveNet ( van den Oord et al. , 2016 ) , SampleRNN ( Mehri et al. , 2017 ) and WaveRNN ( Kalchbrenner et al. , 2018 ) . A notable limitation of these models is that they are difficult to parallelise over time : they predict each time step of an audio signal in sequence , which is computationally expensive and often impractical . A lot of recent research on neural models for TTS has focused on improving parallelism by predicting multiple time steps in parallel , e.g . using flow-based models ( van den Oord et al. , 2018 ; Ping et al. , 2019 ; Prenger et al. , 2019 ; Kim et al. , 2019 ) . Such highly parallelisable models are more suitable to run efficiently on modern hardware . An alternative approach for parallel waveform generation would be to use Generative Adversarial Networks ( GANs , Goodfellow et al. , 2014 ) . GANs currently constitute one of the dominant paradigms for generative modelling of images , and they are able to produce high-fidelity samples ∗Work done at DeepMind . that are almost indistinguishable from real data . However , their application to audio generation tasks has seen relatively limited success so far . In this paper , we explore raw waveform generation with GANs , and demonstrate that adversarially trained feed-forward generators are indeed able to synthesise high-fidelity speech audio . Our contributions are as follows : • We introduce GAN-TTS , a Generative Adversarial Network for text-conditional highfidelity speech synthesis . Its feed-forward generator is a convolutional neural network , coupled with an ensemble of multiple discriminators which evaluate the generated ( and real ) audio based on multi-frequency random windows . Notably , some discriminators take the linguistic conditioning into account ( so they can measure how well the generated audio corresponds to the input utterance ) , while others ignore the conditioning , and can only assess the general realism of the audio . • We propose a family of quantitative metrics for speech generation based on Fréchet Inception Distance ( FID , Heusel et al. , 2017 ) and Kernel Inception Distance ( KID , Bińkowski et al. , 2018 ) , where we replace the Inception image recognition network with the DeepSpeech audio recognition network . The code for our metrics is publicly available online1 . • We present quantitative and subjective evaluation of TTS-GAN and its ablations , demonstrating the importance of our architectural choices . Our best-performing model achieves a MOS of 4.2 , which is comparable to the state-of-the-art WaveNet MOS of 4.4 , and establishes GANs as a viable option for efficient TTS . 2 RELATED WORK . 2.1 AUDIO GENERATION . Most neural models for audio generation are likelihood-based : they represent an explicit probability distribution and the likelihood of the observed data is maximised under this distribution . Autoregressive models achieve this by factorising the joint distribution into a product of conditional distributions ( van den Oord et al. , 2016 ; Mehri et al. , 2017 ; Kalchbrenner et al. , 2018 ; Arik et al. , 2017 ) . Another strategy is to use an invertible feed-forward neural network to model the joint density directly ( Prenger et al. , 2019 ; Kim et al. , 2019 ) . Alternatively , an invertible feed-forward model can be trained by distilling an autoregressive model using probability density distillation ( van den Oord et al. , 2018 ; Ping et al. , 2019 ) , which enables it to focus on particular modes . Such mode-seeking behaviour is often desirable in conditional generation settings : we want the generated speech signals to sound realistic and correspond to the given text , but we are not interested in modelling every possible variation that occurs in the data . This reduces model capacity requirements , because parts of the data distribution may be ignored . Note that adversarial models exhibit similar behaviour , but without the distillation and invertibility requirements . Many audio generation models , including all of those discussed so far , operate in the waveform domain : they directly model the amplitude of the waveform as it evolves over time . This is in stark contrast to most audio models designed for discriminative tasks ( e.g . audio classification ) : such models tend to operate on time-frequency representations of audio ( spectrograms ) , which encode certain inductive biases with respect to the human perception of sound , and usually discard all phase information in the signal . While phase information is often inconsequential for discriminative tasks , generated audio signals must have a realistic phase component , because fidelity as judged by humans is severely affected otherwise . Because no special treatment for the phase component of the signal is required when generating directly in the waveform domain , this is usually more practical . Tacotron ( Wang et al. , 2017 ) and MelNet ( Vasquez & Lewis , 2019 ) constitute notable exceptions , and they use the Griffin-Lim algorithm ( Griffin & Lim , 1984 ) to reconstruct missing phase information , which the models themselves do not generate . Models like Deep Voice 2 & 3 ( Gibiansky et al. , 2017 ; Ping et al. , 2018 ) and Tacotron 2 ( Shen et al. , 2018 ) achieve a compromise by first generating a spectral representation , and then using a separate autoregressive model to turn it into a waveform and fill in any missing spectral information . Because the generated spectrograms are imperfect , the waveform model has the additional task of correcting any mistakes . Char2wav ( Sotelo et al. , 2017 ) uses intermediate vocoder features in a similar fashion . 1https : //github.com/mbinkowski/DeepSpeechDistances 2.2 GENERATIVE ADVERSARIAL NETWORKS . Generative Adversarial Networks ( GANs , Goodfellow et al. , 2014 ) form a subclass of implicit generative models that relies on adversarial training of two networks : the generator , which attempts to produce samples that mimic the reference distribution , and the discriminator , which tries to differentiate between real and generated samples and , in doing so , provides a useful gradient signal to the generator . Following rapid development , GANs have achieved state-of-the-art results in image ( Zhang et al. , 2019 ; Brock et al. , 2019 ; Karras et al. , 2019 ) and video generation ( Clark et al. , 2019 ) , and have been successfully applied for unsupervised feature learning ( Donahue et al. , 2017 ; Dumoulin et al. , 2017a ; Donahue & Simonyan , 2019 ) , among many other applications . Despite achieving impressive results in these domains , limited work has so far shown good performance of GANs in audio generation . Two notable exceptions include WaveGAN ( Donahue et al. , 2019 ) and GANSynth ( Engel et al. , 2019 ) , which both successfully applied GANs to simple datasets of audio data . The former is the most similar to this work in the sense that it uses GANs to generate raw audio ; results were obtained for a dataset of spoken commands of digits from zero to nine . The latter provides state-of-the-art results for a dataset of single note recordings from various musical instruments ( NSynth , Engel et al. , 2017 ) by training GANs to generate magnitude-phase spectrograms of the notes ( which can easily be converted to waveforms , up to 64000 samples in length ) . Neekhara et al . ( 2019 ) propose an adversarial vocoder model that is able to synthesise magnitude spectrograms from mel-spectrograms generated by Tacotron 2 , and combine this with phase estimation using the Local Weighted Sums technique ( Le Roux et al. , 2010 ) . Saito et al . ( 2018 ) propose to enhance a parametric speech synthesiser with a GAN to avoid oversmoothing of the generated speech parameters . Yamamoto et al . ( 2019 ) replace the probabilistic distillation loss in Parallel WaveNet with an adversarial version . Tanaka et al . ( 2018 ) use a variant of CycleGAN directly on raw waveforms for voice conversion . GANs have seen limited application at large scale in non-visual domains so far . Two seconds of audio at 24kHz2 has a dimensionality of 48000 , which is comparable to RGB images at 128 × 128 resolution . Until recently , high-quality GAN-generated images at such or higher resolution were uncommon ( Zhang et al. , 2019 ; Karras et al. , 2019 ) , and it was not clear that training GANs at scale would lead to extensive improvements ( Brock et al. , 2019 ) . Multiple discriminators have been used in GANs for different purposes . For images , Denton et al . ( 2015 ) ; Zhang et al . ( 2017 ) ; Karras et al . ( 2018 ) proposed to use separate discriminators for different resolutions . Similar approaches have also been used in image-to-image transfer ( Huang et al. , 2018 ) and video synthesis ( Saito & Saito , 2018 ) . Clark et al . ( 2019 ) , on the other hand , combine a 3D-discriminator that scores the video at lower resolution and a 2D-frame discriminator which looks at individual frames . In adversarial feature learning , Donahue & Simonyan ( 2019 ) combine outputs from three discriminators to differentiate between joint distributions of images and latents . Discriminators operating on windows of the input have been used in adversarial texture synthesis ( Li & Wand , 2016 ) and image translation ( Isola et al. , 2017 ; Zhu et al. , 2017 ) . In parallel with our work , MelGAN ( Kumar et al. , 2019 ) used GANs for mel-spectrogram inversion , with multiple discriminators operating at different frequencies . MelGAN enforces the mapping between input spectrogram conditioning and generated audio via an L1 reconstruction loss in discriminator feature space , in contrast with our work , which enforces the mapping adversarially using conditional discriminators . When combined with a Text2mel model , their generator achieves a MOS of 3.7 . 3 GAN-TTS . 3.1 DATASET . Our text-to-speech models are trained on a dataset which contains high-fidelity audio of human speech with the corresponding linguistic features and pitch information . The linguistic features encode phonetic and duration information , while the pitch is represented by the logarithmic fundamental frequency logF0 . In total , there are 567 features . We do not use ground-truth duration and 224kHz is a commonly used frequency for speech , because the absence of frequencies above 12kHz does not meaningfully affect fidelity . pitch for subjective evaluation ; we instead use duration and pitch predicted by separate models . The dataset is formed of variable-length audio clips containing single sequences , spoken by a professional voice actor in North American English . For training , we sample 2 second windows ( filtering out shorter examples ) together with corresponding linguistic features . The total length of the filtered dataset is 44 hours . The sampling frequency of the audio is 24kHz , while the linguistic features and pitch are computed for 5ms windows ( at 200Hz ) . This means that the generator network needs to learn how to convert the linguistic features and pitch into raw audio , while upsampling the signal by a factor of 120 . We apply a µ-law transform to account for the logarithmic perception of volume ( see Appendix C ) . 3.2 GENERATOR . A summary of generator G ’ s architecture is presented in Table 2 in Appendix A.2 . The input to G is a sequence of linguistic and pitch features at 200Hz , and its output is the raw waveform at 24kHz . The generator is composed of seven blocks ( GBlocks , Figure 1a ) , each of which is a stack of two residual blocks ( He et al. , 2016 ) . As the generator is producing raw audio ( e.g . a 2s training clip corresponds to a sequence of 48000 samples ) , we use dilated convolutions ( Yu & Koltun , 2016 ) to ensure that the receptive field of G is large enough to capture long-term dependencies . The four kernel size-3 convolutions in each GBlock have increasing dilation factors : 1 , 2 , 4 , 8 . Convolutions are preceded by Conditional Batch Normalisation ( Dumoulin et al. , 2017b ) , conditioned on the linear embeddings of the noise term z ∼ N ( 0 , I128 ) in the single-speaker case , or the concatenation of z and a onehot representation of the speaker ID in the multi-speaker case . The embeddings are different for each BatchNorm instance . A GBlock contains two skip connections , the first of which performs upsampling if the output frequency is higher than the input , and it also contains a size-1 convolution if the number of output channels is different from the input . GBlocks 3–7 gradually upsample the temporal dimension of hidden representations by factors of 2 , 2 , 2 , 3 , 5 , while the number of channels is reduced by GBlocks 3 , 6 and 7 ( by a factor of 2 each ) . The final convolutional layer with Tanh activation produces a single-channel audio waveform .

Summary: This paper proposes to enable GAN based TTS in the time domain with the careful designs of the (non-autoregressive) generator and discriminator. There have been various trials of GAN-TTS but not so many success and I'm glad to hear that the proposed method seems to enable GAN-TTS with fast inference thanks to the non-autoregressive property. The method also proposes new objective measures inspired by the image recognition network based on the high-level features generated by end-to-end ASR, which is also another important contribution of this paper.