Progressive Upsampling Audio Synthesis via Effective Adversarial Training

1 INTRODUCTION . Synthesis of realistic sound is a long-studied research topic , with various real-world applications such as text-to-speech ( TTS ) ( Wang et al. , 2017 ; Ping et al. , 2018 ) , sound effect ( Raghuvanshi et al. , 2016 ) , and music generation ( Briot et al. , 2017 ; Dong et al. , 2018 ; Huang et al. , 2019 ) . Various techniques have been developed ranging from the sample-based to more computational ones such as the additive/subtractive synthesis , frequency modulation granular synthesis , and even a full physicsbased simulation ( Cook , 2002 ) . Human ’ s audible frequency range is up to 20 kHz , so the standard sampling rate for music and sound is 44.1 kHz . Thus , for interactive applications and live performances , the generation of the high temporal-resolution audio ( i.e. , 44.1 kHz ) in real-time has to meet the standard of human perceptual sensitivity to sound . However , the aforementioned methods often fail to do so , due to their heavy computational complexity with respect to the data size . Because of this , professional sound synthesizers usually have no choice but to rely on hardware implementations . ( Wessel & Wright , 2002 ) Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have emerged as a promising approach to the versatile ( e.g. , conditional generation from a low-dimensional latent vector ( Mirza & Osindero , 2014 ) ) and high-quality ( e.g. , super-resolution GAN ( Ledig et al. , 2017 ) ) image . One of the first GAN models for sound synthesis have been designed to first produce the spectrogram ( or some other similar intermediate representations ) ( Donahue et al. , 2019 ; Engel et al. , 2019 ; Marafioti et al. , 2019 ) . A spectrogram is a compact 2D representation of audio signals in terms of its frequency spectrum over time . The spectrogram can then be converted into the estimated time-domain waveform using the Griffin & Lim algorithm ( Griffin & Lim , 1984 ) . However , such a conversion process does not only introduces nontrivial errors but also runs slowly , preventing the approach from being applied at an interactive rate1 . WaveGAN ( Donahue et al. , 2019 ) was the first and state-of-the-art GAN model that can generate raw waveform audio from scratch . The first generations of sound-generating GANs , like the WaveGAN and its followers , have been influenced much by the enormously successful generative models for image synthesis . They can 1The interactive rate refers to the the maximum temporal threshold of around 10msec ( Wessel & Wright , 2002 ) over which humans would not be able to recognize the sound making event and the resultant sound as occuring at the same time . be divided into those that employ the single decoder architecture ( e.g. , DCGAN and StyleGAN ( Radford et al. , 2016 ; Karras et al. , 2019 ) ) and those that encode and decode the intermediate representations in several and progressive stages ( e.g. , StackGAN and progressive GAN ( Zhang et al. , 2017 ; Karras et al. , 2018 ) ) . WaveGAN is the direct descendant of DCGAN with modification for the 1D audio data , while GANSynth applied the concept of progressive generation of audio , but using the 2D spectrogram , treating the audio as a 2D image . No previous work in GAN based audio generation has attempted the direct and fast synthesis of 1D raw audio waveform employing the multiple and progressive encoder-decoder architecture . Therefore , in this paper , we propose PUGAN , modification and extension of WaveGAN architecture for efficiently synthesizing raw-waveform audio through progressive training . PUGAN generates low sampling rate audio using the first few layers of the original WaveGAN ( referred to as the lightweight WaveGAN module ) . The latter layers of WaveGAN are replaced with the bandwidth extension modules , each of which is composed of the neural upsampling layer and encoder/decoder . They progressively output ( progressively trained too ) the higher sampling rate audio . For the effective progressive training and generation , instead of the usual upsampling method such as the nearest neighbor used in image generation , PUGAN uses a new upsampling methods often employed in the digital signal processing ( DSP ) field in an attempt to preserve the frequency information of the original data ( Oppenheim , 1999 ) . This upsample process consists of the zero insertion and 1D convolution to function as an interpolation infinite impulse response ( IIR ) filter . On the discriminator side , we add the Sinc convolution ( Ravanelli & Bengio , 2018 ) before the first layer to replicate the function of the parameterized low pass Sinc filter , also a popular technique in the DSP area . We have also evaluated PUGAN in terms of both quantitative computational performance and qualitative metrics including the human perceptual evaluation . ( demo and code : https : //pugan-iclrdemo.herokuapp.com/ ) Overall , our contributions include the following : • propose PUGAN , with novel neural modules ( upsampling and bandwidth extension ) for the efficient generation of raw waveform audio , • apply the concept of resampling ( in the generator ) and sinc convolution layers ( in the discriminator ) suitable for handling sound generation instead of the conventional upsampling or convolution methods , and • demonstrate the effectiveness of the proposed approach by generating raw waveform audio with significantly less number of parameters in real-time with equivalent output quality as WaveGAN . 2 RELATED WORK . We first review related research in two areas , namely , the GAN-based sound generation and audioto-audio conversion . 2.1 GAN BASED AUDIO GENERATION . WaveGAN ( Donahue et al. , 2019 ) and GANSynth ( Engel et al. , 2019 ) are the two recent notable work that have applied the GAN technique to sound effects generation for the first time . WaveGAN modified the DCGAN and took the approach to operate for and generate one dimensional sound data ( raw-waveform ) fast and directly ( and distinguishing itself from the work like the SpecGAN ( Donahue et al. , 2019 ) which used the usual 2D/image-based processing and spectrogram output representation ) . WaveGAN also added the phase shuffle module to prevent the discriminator from learning the checkerboard artifact , and post-processing convolution layer with a relatively wide kernel size for noise reduction . GANSynth generated sound effects through the spectrogram-like representation , but its output quality was satisfactory for only pure tone instrumental sounds . TiF-GAN ( Marafioti et al. , 2019 ) made a marginal improvement by adding a provisional step for the phase information reconstruction . Note that in the generative setting , the 2D based approach ( using representations like spectrograms ) is considered problematic as spectrograms are not fully invertible to sound without a loss ( thus inexact ) and the inversion by , say , the most popular Griffin & Lim algorithm is time-consuming . A fast version of Griffin & Lim algorithm ( Perraudin et al. , 2013 ) and Deep Griffin & Lim iteration approaches ( Masuyama et al. , 2019 ) have been proposed to improve the inversion performance ; they have not reached the aforementioned level required for interactive applications . In addition , all previous GAN-based sound synthesizers were configured and experimented to output up to only 16 kHz audio . Such a sampling rate is sufficient for general TTS systems , but , as noted earlier , not for general sound effects or music . WaveNet ( Oord et al. , 2016 ) can generate 44.1 kHz sampling rate audio , but aside from the questionable quality of the output , its autoregressive nature make it difficult to achieve real-time performance through the GPU parallelization . In this regard , once trained , the generator part of GAN generally can operate faster . But for WaveGAN to generate 44.1 kHz audio , additional transposed convolutional layers ( to the current 16 kHz generator ) would make the architectural parameters prohibitively large ( two times higher ) and likewise the generation time . 2.2 AUDIO-TO-AUDIO CONVERSION . Audio-to-audio conversion refers to the task of taking an input audio sample and converting it into another with different characteristics . Most deep learning based audio conversion models have been influenced by similar image-to-image translation research . For instance , CycleGAN-VC ( Kaneko & Kameoka , 2018 ) , StarGAN-VC ( Kameoka et al. , 2018 ) , and WaveCycleGAN ( Tanaka et al. , 2018 ) all looked into the problem of voice conversion , and were based on the previous works of CycleGAN ( Zhu et al. , 2017 ) and StarGAN ( Choi et al. , 2018 ) . The task of denoising ( Pascual et al. , 2017 ) or generation of super-resolution signal ( Eskimez & Koishida , 2019 ) can also be regarded as a form of signal ( or audio ) conversion . Recently , few attempts have been made to apply GAN to the task of bandwidth extensions such as the SSRGAN ( Eskimez & Koishida , 2019 ; Li et al. , 2019 ) . Note that our objective is the generation of high-resolution audio and sound effects rather than just conversion . 3 DATA CHARACTERISTICS : AUDIO VERSUS IMAGE . In this section , we discuss potential reasons of why conventional GAN architectures have been successful in generating 2D images ( Zhu et al. , 2017 ; Karras et al. , 2018 ; 2019 ) , but less so for 1D sound waves with respect to their data characteristics . This analysis can give us hints on how to newly configure the GAN architecture to generate the sound signal faster with higher quality . Image and sound , both as signals , contain information across the frequency domain . Sound has the added dimension of time . Humans are highly sensitive to variation over time of the sound content over all the frequency range , which makes its quality depend on reproduction of all frequency components . In other words , in sound , the different frequency range may represent a particular characteristic ( e.g. , low bass male sound vs. high pitch female sound ) ( Klevans & Rodman , 1997 ) . In contrast , in images , high resolution components often correspond to details or even noises , and as such static image recognition and understanding may depend less on them ( Heittola et al. , 2009 ) . Upsampling of the data are important parts of the GAN architecture ( especially with respect to the conversion process ) . In image generation , for the reason mentioned above , the upsampling by standard interpolation , such as the nearest neighbor or linear interpolation , may suffice . Fig . 1 compares the application of the simple nearest neighbor based upsampling and linear interpolation to the sinc function based upsampling ( or resampling as better known in the DSP area ) . Fig . 1 shows the upsampling results using the nearest neighbor , linear interpolation and Kaiser resampling methods . The examples illustrate the occurrences of high frequency sidebands ( noise ) when the first two simpler upsampling methods are used . This may be particularly problematic if the high-resolution output is required . Another possibly effective method for dealing with signals of multi-frequency components is the resolution-wise progressive generation ( and training ) technique , as was demonstrated by the work of Progressive GAN ( Karras et al. , 2018 ) . While the original Progressive GAN was applied for 2D images , and similarly to spectrogram generation , we have applied the same idea to the 1D audio signal . However , the preliminary pilot result was not satisfactory ; the reconstructed results were unnaturally smooth in the high frequency range . This is attributed to the similar reason , the stride-1 transposed convolution layer effectively acting as a simple moving averaging method . On the other hand , for an audio generation as WaveGAN has implemented , the upsampling based on the transposed convolution is more proper than the others such as nearest neighbor . It was deemed more accurate in ” capturing ” ( filtering out ) the frequency-wise characteristics in the generation process in comparison to using the nearest neighbor . The only problem may be the fact that the number of the relevant architectural parameters grows excessively according to the output size , which ultimately would render the generation process non real-time . To summarize , based on these observations , the newly proposed PUGAN architecture would proceed to first train to learn the gross structure of the aural information distribution and fast produce the low resolution audio , then incrementally convert and enrich the output to a higher resolution efficiently instead of having to deal with the entire scale space with computationally heavy architecture simultaneously .

Summary: The authors detail PUGAN, architectural changes to models for raw waveform generation with GANs. They do a good job of motivating the challenge of raw audio generation with GANs and of methods for progressive training. PUGAN incorporates U-Net modules in the generator ("Bandwidth expansion"), sinc convolution as bandlimiting inputs to the generators, and the "style gan" type method of adding the noise at each level of the generator. Using listener studies and inception score, they show modest improvements over the state of the art (at time of submission), WaveGAN. Notably, their architecture is also more computation and parameter efficient.