Catch-A-Waveform: Learning to Generate Audio from a Single Short Example

1 Introduction . In recent years , deep models for audio generation have had an immense impact on a wide range of applications , including text-to-speech synthesis [ 12 , 38 , 15 , 7 ] , voice-to-voice translation [ 8 , 53 ] , music generation [ 33 , 11 ] , singing voice conversion [ 8 , 53 ] , timbre transfer [ 16 , 40 ] , bandwidthextension [ 29 , 6 ] , and audio inpainting [ 37 ] . Existing generative models require large datasets of training signals from the domain of interest . However , there are practical scenarios in which such datasets are extremely hard to collect , or are even nonexistent . Examples include a speaker that has only recorded a few sentences , an artist that had the chance to record only a few songs , or a unique jazz improvisation appearing in one particular recording . A natural question to ask , then , is whether large amounts of training data are a necessity for training a generative model . Here , we take this question to the extreme . We illustrate that capturing the essence of an audio source is possible from as little as a few tens of seconds from a single training recording . Specifically , we present a generative adversarial network ( GAN ) based model that can be trained on one short raw waveform and does not require pre-training or any other type of external supervision 1 . Once the 1code is available at https : //github.com/galgreshler/Catch-A-Waveform 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . model is trained , it is capable of generating diverse new signals that are semantically similar to the training recording , but contain new compositions and structures . Our model can handle different types of audio signals , from instrumental music to speech . For example , after training on 20 seconds of a saxophone solo , our model is able to generate new similar improvisations . The same can be done with a-capella rap , or old famous speeches , as exemplified in Fig.1 . Our model can also generate samples conditioned on the low frequencies of some signal ( be it the training signal or a similar one ) . This constraints the global structure of the generated signals , allowing to generate e.g . new versions of a Beatles song ( all audio samples mentioned in the paper can be found in our website ) . It is important to note that a short snippet of an audio signal is insufficient for learning language ( for speech ) or rules of harmony ( for music ) . Therefore , our generated signals lack the linguistic semantics or long-range harmonic structure that can be potentially achieved with externally-trained models . However , surprisingly , the coherence of our generated signals over short time scales , typically suffices for confusing listeners to believe they are real , as we confirm through extensive user studies . Besides generating random samples , we illustrate the utility of our approach in the common tasks of bandwidth extension , inpainting and denoising ( see Fig . 2 ) . We show that in the latter two tasks , no training signal whatsoever is required beyond the input itself . This allows handling sources for which no training data exist , like old recordings of famous musicians . In fact , our evaluation suggests that for the tasks of bandwidth extension and inpainting ( sections 4.3 and 4.4 ) , limiting the training to a single short signal is actually beneficial , and can lead to results that outperform models trained on hours of recordings . Our work is inspired by generative models for visual data , which have been recently explored in the context of learning from a single image [ 50 , 52 ] or a single short video [ 20 ] . Similarly to those works , we present a multi-scale GAN architecture that generates signals in their raw ( time domain ) representation . Audio signals , however , are very different from visual data ; they are of high temporal resolution ( usually at least 16,000 samples per second ) , they exhibit correlations at very long timescales , and they have diverse frequency contents . As we discuss , this necessitates dedicated architectures , losses , and adaptive selection of the multi-scale pyramid levels . 2 Related Work . Generative models for audio . Audio generation models have been extensively studied in the past few years . Some utilize autoregressive architectures [ 43 , 39 ] , including the computationally efficient inverse autoregressive flow ( IAF ) scheme [ 42 , 47 ] and other flow based models [ 27 , 48 , 25 , 49 ] . Others use GANs and variational autoencoders ( VAEs ) , which have been found effective for many applications , including text-to-speech [ 12 , 38 , 15 , 7 ] , unconditional generation [ 33 , 11 ] , singing voice conversion [ 8 , 53 ] , timbre transfer [ 40 ] , inpainting [ 13 , 37 ] , bandwith-extension [ 26 ] , and denoising [ 45 , 32 ] . Several pipelines also integrate classical signal processing blocks to obtain improved results [ 16 ] . All these models rely on large training sets with hours of recordings . In contrast , here we focus on settings where only a single short signal is available for training . Few shot audio learning . Audio generation models have also been taken to the few-shot regime , mainly in the context of voice cloning for speech [ 2 , 9 ] and singing [ 41 ] . In this setting , only a few examples are provided at test-time . However , a large training set is still used for learning to perform the task . Here , on the other hand , we study the use of a single short waveform for training . Internal generative learning . Exploiting the internal statistics of a single audio example by training a deep neural network ( DNN ) was recently explored for the tasks of audio restoration , source separation , audio editing , and ambient sound synthesis [ 62 , 55 ] . These methods , however , can not generate fake signals of complex structure ( like music or speech ) . In the visual domain , recent generative models , like SinGAN [ 50 ] and InGAN [ 52 ] , were developed for learning from a single natural image . These approaches were later extended to other domains , including videos [ 20 ] , medical imaging [ 61 ] , and 3D graphics [ 21 ] . Here we adapt some of these ideas to the audio domain . 3 Method . Consider a short sample x from a stationary audio source . Our goal is to learn a generative model that can draw new random samples x̃ from the source ’ s distribution . Our approach is inspired by the single image GAN ( SinGAN ) model [ 50 ] . Specifically , we aim at matching the distribution of length-T segments of x̃ to that of length-T segments of x , at multiple resolutions . Analysis pyramid . We start by constructing an analysis pyramid of the training signal , x0 = x , xn = ( x ∗ hn ) ↓dn , n = 1 , . . . , N , ( 1 ) where d1 < d2 < · · · < dN are down-sampling factors and h1 , . . . , hn are the corresponding antialiasing filters . This is illustrated at the top of Fig . 3 . Denoting the sampling rate of x by f s ( usually 16Khz in our experiments ) , we have that the sampling rate at the nth pyramid level is f sn = f s/dn . Similarly , we denote by x̃n the nth level of the multi-scale representation of the fake signal x̃ . Synthesis pyramid . The generation of a fake sample x̃ is performed sequentially by generating each of its pyramid levels conditioned on the previous one , from coarse to fine . Specifically , x̃N = GN ( zN ) , x̃n = Gn ( zn , ( x̃n+1 ) ↑αn ) , n = N − 1 , . . . , 0 , ( 2 ) where zn is white Gaussian noise , Gn is a convolutional neural network generator , αn = dn+1/dn is the resolution ratio between scales n+ 1 and n , and ( · ) ↑α stands for up-sampling by a factor of α using cubic interpolation [ 24 ] . The signal x̃0 at the end of this process is the generated fake sample x̃ . This synthesis pyramid is shown at the middle and bottom rows of Fig . 3 . All generators have the same receptive field , as measured in samples . This translates to larger effective receptive fields ( in seconds ) for the coarser levels than for the finer ones . As a result , the coarsest scale can capture the long-range dependencies that are typical of low frequencies of audio signals . Each subsequent generator , then , only needs to add a narrow band of frequencies to the signal generated at the previous scale ( see Fig . 4 ) . The higher the frequency band , the smaller the receptive field that suffices to achieve this goal . Following this understanding , we take the variance of zn to be proportional to the energy of x in the frequency band [ 12f s n , 1 2f s n−1 ] , which is at the responsibility of the generator Gn to synthesize . It is important to note that as opposed to images , audio signals tend to exhibit long-range dependencies even at the highest frequency bands . Therefore , we take the receptive field in samples to be three orders of magnitude larger than the resolution factor α between scales ( see below ) . This is as opposed to the SinGAN image model [ 50 ] , which uses only one order of magnitude . Training . We train our model in a coarse to fine manner as well . At each stage , a single generator in the pyramid is trained while the generators of all coarser levels are kept fixed . When training the nth level , the goal is to drive the distribution of length-T segments within x̃n to become as close as possible to the distribution of length-T segments within xn . To this end , we use a patch-GAN framework [ 31 , 22 ] , which employs a convolutional discriminator network Dn with receptive field T . The discriminator is tasked with classifying each of the overlapping length-T windows in its input as real or fake , so that its output is a classification sequence of the same length as the input ( minus T − 1 samples ) . The final score of the discriminator is the mean of this classification sequence . We specifically use the Wasserstein GAN loss [ 4 ] , Ladv ( Dn , Gn ) = E x∼Pxn [ Dn ( xn ) ] − E x̃n∼Px̃n [ Dn ( x̃n ) ] , ( 3 ) together with a gradient penalty [ 18 ] . Additionally , we pick a particular input at each scale , zrn , and enforce that its corresponding reconstructed signal , x̃rn = Gn ( z r n ) , be close to the real signal xn at that scale . This ensures that there is at least one point in the latent space of our model that maps to the real signal . We do this via a reconstruction loss , Lrec ( Gn ) = α1‖xn − x̃rn‖22 + α2 MSS ( xn , x̃rn ) , ( 4 ) where the second term is the multi-scale spectrogram ( MSS ) loss [ 3 , 42 ] , which penalizes for differences between spectograms ( thus disregarding phase ) . We use the particular MSS formulation of [ 11 ] ( see SM ) . For the reconstruction sequences , we choose { zrN , zrN−1 , ... , zr0 } = { z ? , 0 , ... , 0 } , where z ? is a fixed white Gaussian noise realization . Therefore , overall , we solve min Gn max Dn Ladv ( Dn , Gn ) + Lrec ( Gn ) , ( 5 ) where we alternate between performing one update step for Dn , which also involves minimizing the gradient penalty term , and one update step for Gn . In practice , we typically use only one of the terms in ( 4 ) ( setting the other coefficient to 0 ) , depending on the application ( see Sec . 4 ) . Architecture . The generators and discriminators at all scales have the same fully-convolutional architecture . We use stacked blocks of 8 dilated convolutions , followed by batch normalization and leaky ReLU with slope 0.2 . The dilation factor grows by a factor of 2 in each layer , which is known to be an effective way for increasing of receptive field [ 60 , 43 ] . At the end of the generator we use the gated activation unit [ 44 ] , which is an element-wise product of tanh and sigmoid , each fed by an extra non-dilated convolution . All of our convolution layers have a kernel size of 9 , which leads to a total receptive field of 2040 samples at each scale . We use weight normalization [ 51 ] , which we found to improve results and training stability . At the end of the trainable blocks , we add a fixed pre-emphasis ( PE ) filter with impulse response [ 1 , −0.97 ] , which amplifies the high frequencies , as common in similar tasks [ 58 , 59 ] . An illustration of the generator ’ s architecture is shown in Fig . 5 . Automatic scales selection . Different types of audio signals can have very different power spectra , as we illustrate in Fig . 6 . This suggests that the frequency bands of the pyramid should be adaptively chosen based on the spectrum of the training signal . However , to allow for efficient implementations of resampling techniques , we also want the sampling rates of all scales to be rational factors ( with small denominators ) of f s [ 14 , ch . 9 ] . We therefore use a predefined discrete set of potential sampling rates , and choose our bands adapitively only from this set . As can be seen in Fig . 6 , up to 2Khz , where most of the audible energy resides , the predefined scales grow at a factor of around 1.25 . The mid-range , 2-4Khz , typically contains less energy and so the scales are sparser there . Finally , to be able to capture the energy of non-vocal syllables in speech signals , the scales become denser again from 4Khz . In practice , the most significant effect is due to the automatic selection of the first band , which shapes the global structure of the signal . We therefore choose automatically only this band , such that it contains enough energy ( see SM ) . Fig . 6 shows typical selections of the first band for different types of audio signals . Additional spectra are presented in the SM .

Summary: The authors demonstrate that a SinGAN model can be adapted for audio, training a model on a single instance of audio, and generating continuations and inpainting in both the time and frequency domain. They utilize a multi-scale decomposition to create a pyramid of generators and discriminators, and also force reconstructions for one of the points of the generator's latent space. They make several model accomodations to training on the audio domain, including a criteria for automatically tuning the selected scales of their pyramid to the low-frequency power content of the target audio. They show many applications for their technique, including unconditional temporal continuations, conditional regeneration of higher frequency content to create new song sections, bandwidth extension of low resolution audio, inpainting of audio segments, and implicit denoising through the reconstruction signal. Qualitative and quantitative results support their findings, including listening tests and an extensive page of examples.