GANSynth: Adversarial Neural Audio Synthesis

1 INTRODUCTION . Neural audio synthesis , training generative models to efficiently produce audio with both highfidelity and global structure , is a challenging open problem as it requires modeling temporal scales over at least five orders of magnitude ( ∼0.1ms to ∼100s ) . Large advances in the state-of-the art have been pioneered almost exclusively by autoregressive models , such as WaveNet , which solve the scale problem by focusing on the finest scale possible ( a single audio sample ) and rely upon external conditioning signals for global structure ( van den Oord et al. , 2016 ) . This comes at the cost of slow sampling speed , since they rely on inefficient ancestral sampling to generate waveforms one audio sample at a time . Due to their high quality , a lot of research has gone into speeding up generation , but the methods introduce significant overhead such as training a secondary student network or writing highly customized low-level kernels ( van den Oord et al. , 2018 ; Paine et al. , 2016 ) . Furthermore , since these large models operate at a fine timescale , their autoencoder variants are restricted to only modeling local latent structure due to memory constraints ( Engel et al. , 2017 ) . On the other end of the spectrum , Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have seen great recent success at generating high resolution images ( Radford et al. , 2016 ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Berthelot et al. , 2017 ; Kodali et al. , 2017 ; Karras et al. , 2018a ; Miyato et al. , 2018 ) . Typical GANs achieve both efficient parallel sampling and global latent control by conditioning a stack of transposed convolutions on a latent vector , The potential for audio GANs extends further , as adversarial costs have unlocked intriguing domain transformations for images that could possibly have analogues in audio ( Isola et al. , 2017 ; Zhu et al. , 2017 ; Wolf et al. , 2017 ; Jin et al. , 2017 ) . However , attempts to adapt image GAN architectures to generate waveforms in a straightforward manner ( Donahue et al. , 2019 ) fail to reach the same level of perceptual fidelity as their image counterparts . 1Online resources : Colab Notebook : http : //goo.gl/magenta/gansynth-demo , Audio Examples : http : //goo.gl/magenta/gansynth-examples , Code : http : //goo.gl/magenta/gansynth-code 1.1 GENERATING INSTRUMENT TIMBRES . GAN researchers have made rapid progress in image modeling by evaluating models on focused datasets with limited degrees of freedom , and gradually stepping up to less constrained domains . For example , the popular CelebA dataset ( Liu et al. , 2015 ) is restricted to faces that have been centered and cropped , removing variance in posture and pose , and providing a common reference for qualitative improvements ( Radford et al. , 2016 ; Karras et al. , 2018a ) in generating realistic texture and fine-scale features . Later models then built on that foundation to generalize to broader domains ( Karras et al. , 2018b ; Brock et al. , 2019 ) . The NSynth dataset ( Engel et al. , 2017 ) 2 was introduced with similar motivation for audio . Rather than containing all types of audio , NSynth consists solely of individual notes from musical instruments across a range of pitches , timbres , and volumes . Similar to CelebA , all the data is aligned and cropped to reduce variance and focus on fine-scale details , which in audio corresponds to timbre and fidelity . Further , each note is also accompanied by an array of attribute labels to enable exploring conditional generation . The original NSynth paper introduced both autoregressive WaveNet autoencoders and bottleneck spectrogram autoencoders , but without the ability to unconditionally sample from a prior . Follow up work has explored diverse approaches including frame-based regression models ( Defossez et al. , 2018 ) , inverse scattering networks ( Andreux & Mallat , 2018 ) , VAEs with perceptual priors ( Esling et al. , 2018 ) , and adversarial regularization for domain transfer ( Mor et al. , 2019 ) . This work builds on these efforts by introducing adversarial training and exploring effective representations for noncausal convolutional generation as typical found in GANs . 2https : //magenta.tensorflow.org/datasets/nsynth 1.2 EFFECTIVE AUDIO REPRESENTATIONS FOR GANS . Unlike images , most audio waveforms—such as speech and music—are highly periodic . Convolutional filters trained for different tasks on this data commonly learn to form logarithmically-scaled frequency selective filter banks spanning the range of human hearing ( Dieleman & Schrauwen , 2014 ; Zhu et al. , 2016 ) . Human perception is also highly sensitive to discontinuities and irregularities in periodic waveforms , so maintaining the regularity of periodic signals over short to intermediate timescales ( 1ms - 100ms ) is crucial . Figure 1 shows that when the stride of the frames does not exactly equal a waveform ’ s periodicity , the alignment ( phase ) of the two precesses over time . This condition is assured as at any time there are typically many different frequencies in a given signal . This is a challenge for a synthesis network , as it must learn all the appropriate frequency and phase combinations and activate them in just the right combination to produce a coherent waveform . This phase precession is exactly the same phenomena observed with a short-time Fourier transform ( STFT ) , which is composed of strided filterbanks just like convolutional networks . Phase precession also occurs in situations where filterbanks overlap ( window or kernel size < stride ) . In the middle of Figure 1 , we diagram another approach to generating coherent waveforms loosely inspired by the phase vocoder ( Dolson , 1986 ) . A pure tone produces a phase that precesses . Unwrapping the phase , by adding 2π whenever it crosses a phase discontinuity , causes the precessing phase to grow linearly . We then observe that the derivative of the unwrapped phase with respect to time remains constant and is equal to the angular difference between the frame stride and signal periodicity . This is commonly referred to as the instantaneous angular frequency , and is a time varying measure of the true signal oscillation . With a slight abuse of terminology we will simply refer to it as the instantaneous frequency ( IF ) ( Boashash , 1992 ) . Note that for the spectra at the bottom of Figure 1 , the pure harmonic frequencies of a trumpet cause the wrapped phase spectra to oscillate at different rates while the unwrapped phase smoothly diverges and the IF spectra forms solid bands where the harmonic frequencies are present . 1.3 CONTRIBUTIONS . In this paper , we investigate the interplay of architecture and representation in synthesizing coherent audio with GANs . Our key findings include : • Generating log-magnitude spectrograms and phases directly with GANs can produce more coherent waveforms than directly generating waveforms with strided convolutions . • Estimating IF spectra leads to more coherent audio still than estimating phase . • It is important to keep harmonics from overlapping . Both increasing the STFT frame size and switching to mel frequency scale improve performance by creating more separation between the lower harmonic frequencies . Harmonic frequencies are multiples of the fundamental , so low pitches have tightly-spaced harmonics , which can cause blurring and overlap . • On the NSynth dataset , GANs can outperform a strong WaveNet baseline in automatic and human evaluations , and generate examples ∼54,000 times faster . • Global conditioning on latent and pitch vectors allow GANs to generate perceptually smooth interpolation in timbre , and consistent timbral identity across pitch . 2 EXPERIMENTAL DETAILS . 2.1 DATASET . We focus our study on the NSynth dataset , which contains 300,000 musical notes from 1,000 different instruments aligned and recorded in isolation . NSynth is a difficult dataset composed of highly diverse timbres and pitches , but it is also highly structured with labels for pitch , velocity , instrument , and acoustic qualities ( Liu et al. , 2015 ; Engel et al. , 2017 ) . Each sample is four seconds long , and sampled at 16kHz , giving 64,000 dimensions . As we wanted to included human evaluations on audio quality , we restricted ourselves to training on the subset of acoustic instruments and fundamental pitches ranging from MIDI 24-84 ( ∼32-1000Hz ) , as those timbres are most likely to sound natural to an average listener . This left us with 70,379 examples from instruments that are mostly strings , brass , woodwinds , and mallets . We created a new test/train 80/20 split from shuffled data , as the original split was divided along instrument type , which isn ’ t desirable for this task . 2.2 ARCHITECTURE AND REPRESENTATIONS . Taking inspiration from successes in image generation , we adapt the progressive training methods of Karras et al . ( 2018a ) to instead generate audio spectra 3 . While e search over a variety of hyperparameter configurations and learning rates , we direct readers to the original paper for an in-depth analysis ( Karras et al. , 2018a ) , and the appendix for complete details . Briefly , the model samples a random vector z from a spherical Gaussian , and runs it through a stack of transposed convolutions to upsample and generate output data x = G ( z ) , which is fed into a discriminator network of downsampling convolutions ( whose architecture mirrors the generator ’ s ) to estimate a divergence measure between the real and generated distributions ( Arjovsky et al. , 2017 ) . As in Karras et al . ( 2018a ) , we use a gradient penalty ( Gulrajani et al. , 2017 ) to promote Lipschitz continuity , and pixel normalization at each layer . We also try training both progressive and nonprogressive variants , and see comparable quality in both . While it is not essential for success , we do see slightly better convergence time and sample diversity for progressive training , so for the remainder of the paper , all models are compared with progressive training . Unlike Progressive GAN , our method involves conditioning on an additional source of information . Specifically , we append a one-hot representation of musical pitch to the latent vector , with the musically-desirable goal of achieving independent control of pitch and timbre . To encourage the generator to use the pitch information , we also add an auxiliary classification ( Odena et al. , 2017 ) loss to the discriminator that tries to predict the pitch label . For spectral representations , we compute STFT magnitudes and phase angles using TensorFlow ’ s built-in implementation . We use an STFT with 256 stride and 1024 frame size , resulting in 75 % frame overlap and 513 frequency bins . We trim the Nyquist frequency and pad in time to get an “ image ” of size ( 256 , 512 , 2 ) . The two channel dimension correspond to magnitude and phase . We take the log of the magnitude to better constrain the range and then scale the magnitudes to be between -1 and 1 to match the tanh output nonlinearity of the generator network . The phase angle is also scaled to between -1 and 1 and we refer to these variants as “ phase ” models . We optionally unwrap the phase angle and take the finite difference as in Figure 1 ; we call the resulting models “ instantaneous frequency ” ( “ IF ” ) models . We also find performance is sensitive to having sufficient frequency resolution at the lower frequency range . Maintaining 75 % overlap we are able to double the STFT frame size and stride , resulting in spectral images with size ( 128 , 1024 , 2 ) , which we refer to as high frequency resolution , “ + H ” , variants . Lastly , to provide even more separation of lower frequencies we transform both the log magnitudes and instantaneous frequencies to a mel frequency scale without dimensional compression ( 1024 bins ) , which we refer to as “ IF-Mel ” variants . To convert back to linear STFTs we just use the approximate inverse linear transformation , which , perhaps surprisingly does not harm audio quality significantly . It is important for us to compare against strong baselines , so we adapt WaveGAN ( Donahue et al. , 2019 ) , the current state of the art in waveform generation with GANs , to accept pitch conditioning and retrain it on our subset of the NSynth dataset . We also independently train our own waveform generating GANs off the progressive codebase and our best models achieve similar performance to WaveGAN without progressive training , so we opt to primarily show numbers from WaveGAN instead ( see appendix Table 2 for more details ) . Beyond GANs , WaveNet ( van den Oord et al. , 2016 ) is currently the state of the art in generative modeling of audio . Prior work on the NSynth dataset used an WaveNet autoencoder to interpolate between sounds ( Engel et al. , 2017 ) , but is not a generative model as it requires conditioning on the original audio . Thus , we create strong WaveNet baselines by adapting the architecture to accept the same one-hot pitch conditioning signal as the GANs . We train variants using both a categorical 8-bit mu law and 16-bit mixture of logistics for the output distributions , but find that the 8-bit model is more stable and outperforms the 16-bit model ( see appendix Table 2 for more details ) . 3Code : http : //goo.gl/magenta/gansynth-code

Summary: This paper proposes an approach that uses GAN framework to generate audio through modeling log magnitudes and instantaneous frequencies with sufficient frequency resolution in the spectral domain. Experiments on NSynth dataset show that it gives better results then WaveNet. The most successful deep generative models are WaveNET,  Parallel WaveNet and Tacotran that are applied to speech synthesis, the method should be tested for speech synthesis and compared with WaveNet, Parallel WaveNet as well as Tacotran.