Chunked Autoregressive GAN for Conditional Waveform Synthesis

Conditional waveform synthesis models learn a distribution of audio waveforms given conditioning such as text , mel-spectrograms , or MIDI . These systems employ deep generative models that model the waveform via either sequential ( autoregressive ) or parallel ( non-autoregressive ) sampling . Generative adversarial networks ( GANs ) have become a common choice for non-autoregressive waveform synthesis . However , state-of-the-art GAN-based models produce artifacts when performing mel-spectrogram inversion . In this paper , we demonstrate that these artifacts correspond with an inability for the generator to learn accurate pitch and periodicity . We show that simple pitch and periodicity conditioning is insufficient for reducing this error relative to using autoregression . We discuss the inductive bias that autoregression provides for learning the relationship between instantaneous frequency and phase , and show that this inductive bias holds even when autoregressively sampling large chunks of the waveform during each forward pass . Relative to prior state-of- the-art GAN-based models , our proposed model , Chunked Autoregressive GAN ( CARGAN ) reduces pitch error by 40-60 % , reduces training time by 58 % , maintains a fast generation speed suitable for realtime or interactive applications , and maintains or improves subjective quality . 1 INTRODUCTION . Conditional audio waveform generation has seen remarkable improvements in fidelity and computational cost in recent years , benefiting applications such as text-to-speech ( Kim et al. , 2021 ) , voice editing ( Qian et al. , 2020 ; Morrison et al. , 2020 ) , and controllable music generation ( Dhariwal et al. , 2020 ) . These advancements are driven by concurrent improvements in deep generative models ( Bond-Taylor et al. , 2021 ) , which model the distribution of training data with or without auxiliary conditioning information . Conditional audio generation models distributions of audio waveforms with conditioning such as text ( Kim et al. , 2021 ) , mel-spectrograms ( Kong et al. , 2020 ) , or MIDI ( Hawthorne et al. , 2018 ) . In this paper , we focus on tasks such as spectrogram-to-waveform inversion , where the temporal alignment between the conditioning and the waveform is known and the alignment does not have to be learned by the model ( Donahue et al. , 2020 ; Kim et al. , 2021 ) . Generative neural networks are the dominant method for audio generation tasks , strongly outperforming comparable methods that use only digital signal processing ( DSP ) techniques ( Griffin & Lim , 1984 ; Morise et al. , 2016 ) . Neural methods learn the waveform distribution directly from data . Let x = { x1 , . . . , xT } be an audio waveform and c = { c1 , . . . , cU } be an aligned—but not necessarily same length—conditioning signal ( e.g. , a mel-spectrogram ) . Conditional neural audio generation aims to learn the conditional distribution p ( x|c ) . Such models are commonly classified as either generating samples sequentially ( autoregressive ) or in parallel ( non-autoregressive ) . We next describe autoregressive and non-autoregressive methods for audio generation , as well as hybrid methods that aim to combine the benefits of both methods . Autoregressive methods Autoregressive models parameterize p ( x|c ) via the following factorization . p ( x|c ) = T∏ t=0 p ( xt|x1 , . . . , xt−1 , c ) ( 1 ) These models produce subjectively high-quality audio . However , autoregressive generation involves sequentially sampling each audio sample , where each sample requires a forward pass through a neural network . Thus , one major disadvantage of autoregressive models is that generation is slow . Examples of autoregressive models include WaveNet ( Oord et al. , 2016 ) , SampleRNN ( Mehri et al. , 2016 ) , WaveRNN ( Kalchbrenner et al. , 2018 ) , and Jukebox ( Dhariwal et al. , 2020 ) . WaveNet introduced the idea of autoregressive waveform generation , demonstrating high-quality generation of speech from aligned linguistic features ( e.g. , pitch and phonemes ) as well as unconditioned generation of speech and music . SampleRNN uses a hierarchy of gated recurrent units ( Cho et al. , 2014 ) that operate at increasing temporal resolution to reduce the computation needed to generate each sample . WaveRNN uses a smaller architecture and methods such as weight sparsification ( Narang et al. , 2017 ; Zhu & Gupta , 2017 ) to improve generation speed without sacrificing quality . The resulting model can perform real-time generation , but requires a complex engineering pipeline that includes sparse matrix operations and custom GPU kernels . Jukebox uses a sparse autoregressive transformer ( Child et al. , 2019 ) to generate discrete codes of a hierarchical VQVAE ( Razavi et al. , 2019 ) . Jukebox can generate coherent musical excerpts from a variety of genres with rich conditioning , but requires over eight hours on a V100 GPU to produce one minute of audio . Non-autoregressive methods Non-autoregressive models parameterize p ( x|c ) directly , using methods such as normalizing flows ( Dinh et al. , 2014 ) , denoising diffusion probabilistic models ( DDPMs ) ( Ho et al. , 2020 ) , source-filter models ( Fant , 1970 ) , and generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . Exemplar models for conditional audio generation include the flow-based WaveGlow ( Prenger et al. , 2019 ) , the DDPM-based WaveGrad ( Chen et al. , 2020 ) , the Neural Source-Filter ( NSF ) model ( Wang et al. , 2019 ) , and the GAN-based HiFi-GAN ( Kong et al. , 2020 ) . WaveGlow is composed of invertible operations with simple Jacobians such as 1x1 convolutions and affine transformations ( Kingma & Dhariwal , 2018 ) to model a sequence of functions of random variables drawn from increasingly complex distributions . While generation with WaveGlow is fast , it requires substantial training resources ( e.g. , a week of training on 8 V100s ) as well as memory resources due to the high parameter count . WaveGrad models the gradient of the log probability density ( i.e. , the score function ) via score matching ( Hyvärinen & Dayan , 2005 ) and draws samples from the score-based model using Langevin dynamics ( Song & Ermon , 2019 ) . Samples provided by the authors of WaveGrad indicate that the model does not accurately reproduce high-frequency content . NSF creates a deterministic waveform with the correct frequency and uses a neural network to transform this waveform into speech . Source-filter models require accurate pitch estimation and are restricted to monophonic speech signals . HiFi-GAN uses a convolutional generator to generate audio and multiple discriminators that evaluate the generated waveform at various resolutions and strides . HiFi-GAN permits fast generation on CPU and GPU , but requires two weeks of training on two V100 GPUs and regularly produces artifacts , which we discuss in Section 2 . Hybrid methods Hybrid methods combine the strengths of both autoregressive and nonautoregressive models by generating more than one sample on each forward pass . Existing hybrid methods for waveform synthesis parameterize p ( x|c ) by factoring into chunks of adjacent audio samples that are autoregressively generated . p ( x|c ) = T∏ t= { k,2k , ... , T } p ( xt−k , . . . , xt|x1 , . . . , xt−k−1 , c ) ( 2 ) Subscale WaveRNN ( Kalchbrenner et al. , 2018 ) is one such model , generating 16 samples at a time . WaveFlow ( Ping et al. , 2020 ) and Wave-Tacotron ( Weiss et al. , 2021 ) are both flow-based hybrid models . WaveFlow uses autoregression to handle short-range dependencies and a nonautoregressive architecture to model long-range correlations , while Wave-Tacotron uses autoregression for long-range dependencies and a non-autoregressive model for short-range dependencies . These hybrid models demonstrate a tradeoff between the high model capacity of autoregressive models and the low latency of non-autoregressive models . In this work , we present Chunked Autoregressive GAN ( CARGAN ) , a hybrid GAN-based model for conditional waveform synthesis . CARGAN features fast training , reduced pitch error , and equivalent or improved subjective quality relative to previous GAN-based models . Specifically , we make the following contributions . • We show that GAN-based models such as HiFi-GAN do not accurately preserve the pitch and periodicity of the audio signal , causing audible artifacts in the generated audio that persist across many recent works . • We demonstrate a close relationship between pitch , phase , and autoregression , and show that autoregressive models possess an inductive bias towards learning pitch and phase that is related to their ability to learn the cumulative sum operator . • We show that performing autoregression in large chunks allows us to maintain the inductive bias of autoregression towards modeling pitch and phase while maintaining a fast generation speed , and that the optimal chunk size is related to the causal receptive field of the generator . • We identify and document common artifacts in GAN-based waveform synthesis , providing examples and offering possible causes . We also make our code available under an opensource license1 . 2 PROBLEMS WITH NON-AUTOREGRESSIVE GANS . State-of-the-art GAN-based waveform synthesis models , such as HiFi-GAN ( Kong et al. , 2020 ) , demonstrate impressive subjective quality and generation speed , but exhibit pitch inaccuracy , audible artifacts caused by periodicity inaccuracy , and limitations induced by the objective function . Pitch error artifacts We trained HiFi-GAN ( V1 ) on the VCTK dataset ( Yamagishi et al. , 2019 ) and evaluated the pitch accuracy on 256 randomly selected sentences from a validation set containing speakers seen during training . We use the pitch representation described in Appendix A to extract pitch as well as a binary voiced/unvoiced classification for each frame , which indicates whether the frame exhibits the periodic structure of a pitched sound . We measure the pitch error in regions where both the original and generated speech are voiced . We measure the root-mean-square of the pitch error in cents , defined as 1200 log2 ( y/ŷ ) for pitch values y , ŷ in Hz . We find an average error of 51.2 cents—more than half a semi-tone . A side-by-side listening comparison of examples with high pitch error and the ground truth audio confirms that current GAN models make audible intonation errors ( see Figure 1 ) . Listening examples available on the companion website2 . Periodicity artifacts High pitch error indicates inaccurate reconstruction , but is often not perceptible in isolation as speech with slightly different pitch may still sound natural . We manually collect examples from HiFi-GAN that contain audible artifacts . Using our periodicity representation described in Appendix A , which uses the confidence of a pretrained neural pitch estimator to measure whether a frame of audio exhibits periodic structure between zero ( completely aperiodic ) and one ( completely periodic ) , we find that these artifacts are commonly regions of high periodicity error ( see Figure 1 ) . This indicates that existing GAN-based models make audible voiced/unvoiced errors that perceptually degrade the audio . On the companion website , we provide listening examples as well as examples of this artifact from multiple recent works to demonstrate its prevalence . We 1Code is available at url redacted for double-blind submission . 2Audio examples are available at https : //master.d37eli0fst0eyi.amplifyapp.com/ . strongly encourage readers to listen to these examples , as addressing this artifact is central to our work . Inherent limitations Current GAN-based models for waveform synthesis depend on a feature matching loss ( Larsen et al. , 2016 ; Kumar et al. , 2019 ) . This loss is computed by passing the generated and corresponding real audio through the discriminators , and taking the L1 distances between all activations . Penalizing the L1 distance between activations of early layers requires the generator to produce a waveform that is close to ground truth after only one convolution and non-linearity . This requires the generator to know the initial phase . Otherwise , the generator can incur a large loss due to phase rotation . As we will discuss in Section 3 , the initial phase is not known in the non-autoregressive setting .

Summary: This paper proposes a conditional waveform synthesis (CWS) model called Chunked Autoregressive GAN (CARGAN). By combining the advantages of the AR and non-AR generative models, CARGAN achieves better pitch accuracy with faster training speed and less memory usage without much decline in generation speed.  First, it shows that the previous GAN-based non-AR CWS models do not accurately preserve the pitch and periodicity of the audio signal. Then, this paper demonstrates that there is a close relationship between pitch and phase, and the inductive bias of AR model is good at learning the relationship. Especially, this paper supports the claim by conducting an experiment that compares the ability of  AR and non-AR models to learn the cumulative sum operation, because the instantaneous frequency and instantaneous phase have the cumulative sum relationship. Lastly, by comparing HIFI-GAN and CARGAN based on spectrogram-to-waveform conversion task, this paper shows that CARGAN reconstructs the original waveform more precisely. Furthermore, CARGAN shows faster training speed and less memory usage than those of the HIFI-GAN.