High Fidelity Speech Synthesis with Adversarial Networks

1 INTRODUCTION . The Text-to-Speech ( TTS ) task consists in the conversion of text into speech audio . In recent years , the TTS field has seen remarkable progress , sparked by the development of neural autoregressive models for raw audio waveforms such as WaveNet ( van den Oord et al. , 2016 ) , SampleRNN ( Mehri et al. , 2017 ) and WaveRNN ( Kalchbrenner et al. , 2018 ) . A notable limitation of these models is that they are difficult to parallelise over time : they predict each time step of an audio signal in sequence , which is computationally expensive and often impractical . A lot of recent research on neural models for TTS has focused on improving parallelism by predicting multiple time steps in parallel , e.g . using flow-based models ( van den Oord et al. , 2018 ; Ping et al. , 2019 ; Prenger et al. , 2019 ; Kim et al. , 2019 ) . Such highly parallelisable models are more suitable to run efficiently on modern hardware . An alternative approach for parallel waveform generation would be to use Generative Adversarial Networks ( GANs , Goodfellow et al. , 2014 ) . GANs currently constitute one of the dominant paradigms for generative modelling of images , and they are able to produce high-fidelity samples ∗Work done at DeepMind . that are almost indistinguishable from real data . However , their application to audio generation tasks has seen relatively limited success so far . In this paper , we explore raw waveform generation with GANs , and demonstrate that adversarially trained feed-forward generators are indeed able to synthesise high-fidelity speech audio . Our contributions are as follows : • We introduce GAN-TTS , a Generative Adversarial Network for text-conditional highfidelity speech synthesis . Its feed-forward generator is a convolutional neural network , coupled with an ensemble of multiple discriminators which evaluate the generated ( and real ) audio based on multi-frequency random windows . Notably , some discriminators take the linguistic conditioning into account ( so they can measure how well the generated audio corresponds to the input utterance ) , while others ignore the conditioning , and can only assess the general realism of the audio . • We propose a family of quantitative metrics for speech generation based on Fréchet Inception Distance ( FID , Heusel et al. , 2017 ) and Kernel Inception Distance ( KID , Bińkowski et al. , 2018 ) , where we replace the Inception image recognition network with the DeepSpeech audio recognition network . The code for our metrics is publicly available online1 . • We present quantitative and subjective evaluation of TTS-GAN and its ablations , demonstrating the importance of our architectural choices . Our best-performing model achieves a MOS of 4.2 , which is comparable to the state-of-the-art WaveNet MOS of 4.4 , and establishes GANs as a viable option for efficient TTS . 2 RELATED WORK . 2.1 AUDIO GENERATION . Most neural models for audio generation are likelihood-based : they represent an explicit probability distribution and the likelihood of the observed data is maximised under this distribution . Autoregressive models achieve this by factorising the joint distribution into a product of conditional distributions ( van den Oord et al. , 2016 ; Mehri et al. , 2017 ; Kalchbrenner et al. , 2018 ; Arik et al. , 2017 ) . Another strategy is to use an invertible feed-forward neural network to model the joint density directly ( Prenger et al. , 2019 ; Kim et al. , 2019 ) . Alternatively , an invertible feed-forward model can be trained by distilling an autoregressive model using probability density distillation ( van den Oord et al. , 2018 ; Ping et al. , 2019 ) , which enables it to focus on particular modes . Such mode-seeking behaviour is often desirable in conditional generation settings : we want the generated speech signals to sound realistic and correspond to the given text , but we are not interested in modelling every possible variation that occurs in the data . This reduces model capacity requirements , because parts of the data distribution may be ignored . Note that adversarial models exhibit similar behaviour , but without the distillation and invertibility requirements . Many audio generation models , including all of those discussed so far , operate in the waveform domain : they directly model the amplitude of the waveform as it evolves over time . This is in stark contrast to most audio models designed for discriminative tasks ( e.g . audio classification ) : such models tend to operate on time-frequency representations of audio ( spectrograms ) , which encode certain inductive biases with respect to the human perception of sound , and usually discard all phase information in the signal . While phase information is often inconsequential for discriminative tasks , generated audio signals must have a realistic phase component , because fidelity as judged by humans is severely affected otherwise . Because no special treatment for the phase component of the signal is required when generating directly in the waveform domain , this is usually more practical . Tacotron ( Wang et al. , 2017 ) and MelNet ( Vasquez & Lewis , 2019 ) constitute notable exceptions , and they use the Griffin-Lim algorithm ( Griffin & Lim , 1984 ) to reconstruct missing phase information , which the models themselves do not generate . Models like Deep Voice 2 & 3 ( Gibiansky et al. , 2017 ; Ping et al. , 2018 ) and Tacotron 2 ( Shen et al. , 2018 ) achieve a compromise by first generating a spectral representation , and then using a separate autoregressive model to turn it into a waveform and fill in any missing spectral information . Because the generated spectrograms are imperfect , the waveform model has the additional task of correcting any mistakes . Char2wav ( Sotelo et al. , 2017 ) uses intermediate vocoder features in a similar fashion . 1https : //github.com/mbinkowski/DeepSpeechDistances 2.2 GENERATIVE ADVERSARIAL NETWORKS . Generative Adversarial Networks ( GANs , Goodfellow et al. , 2014 ) form a subclass of implicit generative models that relies on adversarial training of two networks : the generator , which attempts to produce samples that mimic the reference distribution , and the discriminator , which tries to differentiate between real and generated samples and , in doing so , provides a useful gradient signal to the generator . Following rapid development , GANs have achieved state-of-the-art results in image ( Zhang et al. , 2019 ; Brock et al. , 2019 ; Karras et al. , 2019 ) and video generation ( Clark et al. , 2019 ) , and have been successfully applied for unsupervised feature learning ( Donahue et al. , 2017 ; Dumoulin et al. , 2017a ; Donahue & Simonyan , 2019 ) , among many other applications . Despite achieving impressive results in these domains , limited work has so far shown good performance of GANs in audio generation . Two notable exceptions include WaveGAN ( Donahue et al. , 2019 ) and GANSynth ( Engel et al. , 2019 ) , which both successfully applied GANs to simple datasets of audio data . The former is the most similar to this work in the sense that it uses GANs to generate raw audio ; results were obtained for a dataset of spoken commands of digits from zero to nine . The latter provides state-of-the-art results for a dataset of single note recordings from various musical instruments ( NSynth , Engel et al. , 2017 ) by training GANs to generate magnitude-phase spectrograms of the notes ( which can easily be converted to waveforms , up to 64000 samples in length ) . Neekhara et al . ( 2019 ) propose an adversarial vocoder model that is able to synthesise magnitude spectrograms from mel-spectrograms generated by Tacotron 2 , and combine this with phase estimation using the Local Weighted Sums technique ( Le Roux et al. , 2010 ) . Saito et al . ( 2018 ) propose to enhance a parametric speech synthesiser with a GAN to avoid oversmoothing of the generated speech parameters . Yamamoto et al . ( 2019 ) replace the probabilistic distillation loss in Parallel WaveNet with an adversarial version . Tanaka et al . ( 2018 ) use a variant of CycleGAN directly on raw waveforms for voice conversion . GANs have seen limited application at large scale in non-visual domains so far . Two seconds of audio at 24kHz2 has a dimensionality of 48000 , which is comparable to RGB images at 128 × 128 resolution . Until recently , high-quality GAN-generated images at such or higher resolution were uncommon ( Zhang et al. , 2019 ; Karras et al. , 2019 ) , and it was not clear that training GANs at scale would lead to extensive improvements ( Brock et al. , 2019 ) . Multiple discriminators have been used in GANs for different purposes . For images , Denton et al . ( 2015 ) ; Zhang et al . ( 2017 ) ; Karras et al . ( 2018 ) proposed to use separate discriminators for different resolutions . Similar approaches have also been used in image-to-image transfer ( Huang et al. , 2018 ) and video synthesis ( Saito & Saito , 2018 ) . Clark et al . ( 2019 ) , on the other hand , combine a 3D-discriminator that scores the video at lower resolution and a 2D-frame discriminator which looks at individual frames . In adversarial feature learning , Donahue & Simonyan ( 2019 ) combine outputs from three discriminators to differentiate between joint distributions of images and latents . Discriminators operating on windows of the input have been used in adversarial texture synthesis ( Li & Wand , 2016 ) and image translation ( Isola et al. , 2017 ; Zhu et al. , 2017 ) . In parallel with our work , MelGAN ( Kumar et al. , 2019 ) used GANs for mel-spectrogram inversion , with multiple discriminators operating at different frequencies . MelGAN enforces the mapping between input spectrogram conditioning and generated audio via an L1 reconstruction loss in discriminator feature space , in contrast with our work , which enforces the mapping adversarially using conditional discriminators . When combined with a Text2mel model , their generator achieves a MOS of 3.7 . 3 GAN-TTS . 3.1 DATASET . Our text-to-speech models are trained on a dataset which contains high-fidelity audio of human speech with the corresponding linguistic features and pitch information . The linguistic features encode phonetic and duration information , while the pitch is represented by the logarithmic fundamental frequency logF0 . In total , there are 567 features . We do not use ground-truth duration and 224kHz is a commonly used frequency for speech , because the absence of frequencies above 12kHz does not meaningfully affect fidelity . pitch for subjective evaluation ; we instead use duration and pitch predicted by separate models . The dataset is formed of variable-length audio clips containing single sequences , spoken by a professional voice actor in North American English . For training , we sample 2 second windows ( filtering out shorter examples ) together with corresponding linguistic features . The total length of the filtered dataset is 44 hours . The sampling frequency of the audio is 24kHz , while the linguistic features and pitch are computed for 5ms windows ( at 200Hz ) . This means that the generator network needs to learn how to convert the linguistic features and pitch into raw audio , while upsampling the signal by a factor of 120 . We apply a µ-law transform to account for the logarithmic perception of volume ( see Appendix C ) . 3.2 GENERATOR . A summary of generator G ’ s architecture is presented in Table 2 in Appendix A.2 . The input to G is a sequence of linguistic and pitch features at 200Hz , and its output is the raw waveform at 24kHz . The generator is composed of seven blocks ( GBlocks , Figure 1a ) , each of which is a stack of two residual blocks ( He et al. , 2016 ) . As the generator is producing raw audio ( e.g . a 2s training clip corresponds to a sequence of 48000 samples ) , we use dilated convolutions ( Yu & Koltun , 2016 ) to ensure that the receptive field of G is large enough to capture long-term dependencies . The four kernel size-3 convolutions in each GBlock have increasing dilation factors : 1 , 2 , 4 , 8 . Convolutions are preceded by Conditional Batch Normalisation ( Dumoulin et al. , 2017b ) , conditioned on the linear embeddings of the noise term z ∼ N ( 0 , I128 ) in the single-speaker case , or the concatenation of z and a onehot representation of the speaker ID in the multi-speaker case . The embeddings are different for each BatchNorm instance . A GBlock contains two skip connections , the first of which performs upsampling if the output frequency is higher than the input , and it also contains a size-1 convolution if the number of output channels is different from the input . GBlocks 3–7 gradually upsample the temporal dimension of hidden representations by factors of 2 , 2 , 2 , 3 , 5 , while the number of channels is reduced by GBlocks 3 , 6 and 7 ( by a factor of 2 each ) . The final convolutional layer with Tanh activation produces a single-channel audio waveform .

Summary: I want thank the authors for solving this long-standing GAN challenge in raw waveform synthesis. With all due respect, previous GAN trials for audio synthesis are inspiring, but their audio qualities are far away from the state-of-the-art results. Although the speech fidelity of GAN-TTS is still worse than WaveNet and Parallel WaveNet from the posted sample, it has begun to close the significant performance gap that has existed between autoregressive models and GANs for raw audios. Overall, this is a very good paper with significant contributions to the filed.