Catch-A-Waveform: Learning to Generate Audio from a Single Short Example

1 Introduction . In recent years , deep models for audio generation have had an immense impact on a wide range of applications , including text-to-speech synthesis [ 12 , 38 , 15 , 7 ] , voice-to-voice translation [ 8 , 53 ] , music generation [ 33 , 11 ] , singing voice conversion [ 8 , 53 ] , timbre transfer [ 16 , 40 ] , bandwidthextension [ 29 , 6 ] , and audio inpainting [ 37 ] . Existing generative models require large datasets of training signals from the domain of interest . However , there are practical scenarios in which such datasets are extremely hard to collect , or are even nonexistent . Examples include a speaker that has only recorded a few sentences , an artist that had the chance to record only a few songs , or a unique jazz improvisation appearing in one particular recording . A natural question to ask , then , is whether large amounts of training data are a necessity for training a generative model . Here , we take this question to the extreme . We illustrate that capturing the essence of an audio source is possible from as little as a few tens of seconds from a single training recording . Specifically , we present a generative adversarial network ( GAN ) based model that can be trained on one short raw waveform and does not require pre-training or any other type of external supervision 1 . Once the 1code is available at https : //github.com/galgreshler/Catch-A-Waveform 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . model is trained , it is capable of generating diverse new signals that are semantically similar to the training recording , but contain new compositions and structures . Our model can handle different types of audio signals , from instrumental music to speech . For example , after training on 20 seconds of a saxophone solo , our model is able to generate new similar improvisations . The same can be done with a-capella rap , or old famous speeches , as exemplified in Fig.1 . Our model can also generate samples conditioned on the low frequencies of some signal ( be it the training signal or a similar one ) . This constraints the global structure of the generated signals , allowing to generate e.g . new versions of a Beatles song ( all audio samples mentioned in the paper can be found in our website ) . It is important to note that a short snippet of an audio signal is insufficient for learning language ( for speech ) or rules of harmony ( for music ) . Therefore , our generated signals lack the linguistic semantics or long-range harmonic structure that can be potentially achieved with externally-trained models . However , surprisingly , the coherence of our generated signals over short time scales , typically suffices for confusing listeners to believe they are real , as we confirm through extensive user studies . Besides generating random samples , we illustrate the utility of our approach in the common tasks of bandwidth extension , inpainting and denoising ( see Fig . 2 ) . We show that in the latter two tasks , no training signal whatsoever is required beyond the input itself . This allows handling sources for which no training data exist , like old recordings of famous musicians . In fact , our evaluation suggests that for the tasks of bandwidth extension and inpainting ( sections 4.3 and 4.4 ) , limiting the training to a single short signal is actually beneficial , and can lead to results that outperform models trained on hours of recordings . Our work is inspired by generative models for visual data , which have been recently explored in the context of learning from a single image [ 50 , 52 ] or a single short video [ 20 ] . Similarly to those works , we present a multi-scale GAN architecture that generates signals in their raw ( time domain ) representation . Audio signals , however , are very different from visual data ; they are of high temporal resolution ( usually at least 16,000 samples per second ) , they exhibit correlations at very long timescales , and they have diverse frequency contents . As we discuss , this necessitates dedicated architectures , losses , and adaptive selection of the multi-scale pyramid levels . 2 Related Work . Generative models for audio . Audio generation models have been extensively studied in the past few years . Some utilize autoregressive architectures [ 43 , 39 ] , including the computationally efficient inverse autoregressive flow ( IAF ) scheme [ 42 , 47 ] and other flow based models [ 27 , 48 , 25 , 49 ] . Others use GANs and variational autoencoders ( VAEs ) , which have been found effective for many applications , including text-to-speech [ 12 , 38 , 15 , 7 ] , unconditional generation [ 33 , 11 ] , singing voice conversion [ 8 , 53 ] , timbre transfer [ 40 ] , inpainting [ 13 , 37 ] , bandwith-extension [ 26 ] , and denoising [ 45 , 32 ] . Several pipelines also integrate classical signal processing blocks to obtain improved results [ 16 ] . All these models rely on large training sets with hours of recordings . In contrast , here we focus on settings where only a single short signal is available for training . Few shot audio learning . Audio generation models have also been taken to the few-shot regime , mainly in the context of voice cloning for speech [ 2 , 9 ] and singing [ 41 ] . In this setting , only a few examples are provided at test-time . However , a large training set is still used for learning to perform the task . Here , on the other hand , we study the use of a single short waveform for training . Internal generative learning . Exploiting the internal statistics of a single audio example by training a deep neural network ( DNN ) was recently explored for the tasks of audio restoration , source separation , audio editing , and ambient sound synthesis [ 62 , 55 ] . These methods , however , can not generate fake signals of complex structure ( like music or speech ) . In the visual domain , recent generative models , like SinGAN [ 50 ] and InGAN [ 52 ] , were developed for learning from a single natural image . These approaches were later extended to other domains , including videos [ 20 ] , medical imaging [ 61 ] , and 3D graphics [ 21 ] . Here we adapt some of these ideas to the audio domain . 3 Method . Consider a short sample x from a stationary audio source . Our goal is to learn a generative model that can draw new random samples x̃ from the source ’ s distribution . Our approach is inspired by the single image GAN ( SinGAN ) model [ 50 ] . Specifically , we aim at matching the distribution of length-T segments of x̃ to that of length-T segments of x , at multiple resolutions . Analysis pyramid . We start by constructing an analysis pyramid of the training signal , x0 = x , xn = ( x ∗ hn ) ↓dn , n = 1 , . . . , N , ( 1 ) where d1 < d2 < · · · < dN are down-sampling factors and h1 , . . . , hn are the corresponding antialiasing filters . This is illustrated at the top of Fig . 3 . Denoting the sampling rate of x by f s ( usually 16Khz in our experiments ) , we have that the sampling rate at the nth pyramid level is f sn = f s/dn . Similarly , we denote by x̃n the nth level of the multi-scale representation of the fake signal x̃ . Synthesis pyramid . The generation of a fake sample x̃ is performed sequentially by generating each of its pyramid levels conditioned on the previous one , from coarse to fine . Specifically , x̃N = GN ( zN ) , x̃n = Gn ( zn , ( x̃n+1 ) ↑αn ) , n = N − 1 , . . . , 0 , ( 2 ) where zn is white Gaussian noise , Gn is a convolutional neural network generator , αn = dn+1/dn is the resolution ratio between scales n+ 1 and n , and ( · ) ↑α stands for up-sampling by a factor of α using cubic interpolation [ 24 ] . The signal x̃0 at the end of this process is the generated fake sample x̃ . This synthesis pyramid is shown at the middle and bottom rows of Fig . 3 . All generators have the same receptive field , as measured in samples . This translates to larger effective receptive fields ( in seconds ) for the coarser levels than for the finer ones . As a result , the coarsest scale can capture the long-range dependencies that are typical of low frequencies of audio signals . Each subsequent generator , then , only needs to add a narrow band of frequencies to the signal generated at the previous scale ( see Fig . 4 ) . The higher the frequency band , the smaller the receptive field that suffices to achieve this goal . Following this understanding , we take the variance of zn to be proportional to the energy of x in the frequency band [ 12f s n , 1 2f s n−1 ] , which is at the responsibility of the generator Gn to synthesize . It is important to note that as opposed to images , audio signals tend to exhibit long-range dependencies even at the highest frequency bands . Therefore , we take the receptive field in samples to be three orders of magnitude larger than the resolution factor α between scales ( see below ) . This is as opposed to the SinGAN image model [ 50 ] , which uses only one order of magnitude . Training . We train our model in a coarse to fine manner as well . At each stage , a single generator in the pyramid is trained while the generators of all coarser levels are kept fixed . When training the nth level , the goal is to drive the distribution of length-T segments within x̃n to become as close as possible to the distribution of length-T segments within xn . To this end , we use a patch-GAN framework [ 31 , 22 ] , which employs a convolutional discriminator network Dn with receptive field T . The discriminator is tasked with classifying each of the overlapping length-T windows in its input as real or fake , so that its output is a classification sequence of the same length as the input ( minus T − 1 samples ) . The final score of the discriminator is the mean of this classification sequence . We specifically use the Wasserstein GAN loss [ 4 ] , Ladv ( Dn , Gn ) = E x∼Pxn [ Dn ( xn ) ] − E x̃n∼Px̃n [ Dn ( x̃n ) ] , ( 3 ) together with a gradient penalty [ 18 ] . Additionally , we pick a particular input at each scale , zrn , and enforce that its corresponding reconstructed signal , x̃rn = Gn ( z r n ) , be close to the real signal xn at that scale . This ensures that there is at least one point in the latent space of our model that maps to the real signal . We do this via a reconstruction loss , Lrec ( Gn ) = α1‖xn − x̃rn‖22 + α2 MSS ( xn , x̃rn ) , ( 4 ) where the second term is the multi-scale spectrogram ( MSS ) loss [ 3 , 42 ] , which penalizes for differences between spectograms ( thus disregarding phase ) . We use the particular MSS formulation of [ 11 ] ( see SM ) . For the reconstruction sequences , we choose { zrN , zrN−1 , ... , zr0 } = { z ? , 0 , ... , 0 } , where z ? is a fixed white Gaussian noise realization . Therefore , overall , we solve min Gn max Dn Ladv ( Dn , Gn ) + Lrec ( Gn ) , ( 5 ) where we alternate between performing one update step for Dn , which also involves minimizing the gradient penalty term , and one update step for Gn . In practice , we typically use only one of the terms in ( 4 ) ( setting the other coefficient to 0 ) , depending on the application ( see Sec . 4 ) . Architecture . The generators and discriminators at all scales have the same fully-convolutional architecture . We use stacked blocks of 8 dilated convolutions , followed by batch normalization and leaky ReLU with slope 0.2 . The dilation factor grows by a factor of 2 in each layer , which is known to be an effective way for increasing of receptive field [ 60 , 43 ] . At the end of the generator we use the gated activation unit [ 44 ] , which is an element-wise product of tanh and sigmoid , each fed by an extra non-dilated convolution . All of our convolution layers have a kernel size of 9 , which leads to a total receptive field of 2040 samples at each scale . We use weight normalization [ 51 ] , which we found to improve results and training stability . At the end of the trainable blocks , we add a fixed pre-emphasis ( PE ) filter with impulse response [ 1 , −0.97 ] , which amplifies the high frequencies , as common in similar tasks [ 58 , 59 ] . An illustration of the generator ’ s architecture is shown in Fig . 5 . Automatic scales selection . Different types of audio signals can have very different power spectra , as we illustrate in Fig . 6 . This suggests that the frequency bands of the pyramid should be adaptively chosen based on the spectrum of the training signal . However , to allow for efficient implementations of resampling techniques , we also want the sampling rates of all scales to be rational factors ( with small denominators ) of f s [ 14 , ch . 9 ] . We therefore use a predefined discrete set of potential sampling rates , and choose our bands adapitively only from this set . As can be seen in Fig . 6 , up to 2Khz , where most of the audible energy resides , the predefined scales grow at a factor of around 1.25 . The mid-range , 2-4Khz , typically contains less energy and so the scales are sparser there . Finally , to be able to capture the energy of non-vocal syllables in speech signals , the scales become denser again from 4Khz . In practice , the most significant effect is due to the automatic selection of the first band , which shapes the global structure of the signal . We therefore choose automatically only this band , such that it contains enough energy ( see SM ) . Fig . 6 shows typical selections of the first band for different types of audio signals . Additional spectra are presented in the SM .

Summary: They main goal of this article is to investigate if it's possible to build a (specific) generative audio model out of short audio snippet. They use a "progressive GAN"-ish model with an additional reconstruction loss, to explore this paradigm. Thei experiment with a fair amount of use cases, demonstrating some degree of success.