Catch-A-Waveform: Learning to Generate Audio from a Single Short Example
Authors: Gal Greshler
Tamar Rott Shaham
Tomer Michaeli
Venue: NeurIPS 2021 Poster (2021)
Keywords: Audio Generation
Single Example
Raw Waveform
GAN
Bandwidth Extension
Audio Inpainting
Audio Denoising
Music Generation

Abstract: Models for audio generation are typically trained on hours of recordings. Here, we illustrate that capturing the essence of an audio source is typically possible from as little as a few tens of seconds from a single training signal. Specifically, we present a GAN-based generative model that can be trained on one short audio signal from any domain (e.g. speech, music, etc.) and does not require pre-training or any other form of external supervision. Once trained, our model can generate random  samples of arbitrary duration that maintain semantic similarity to the training waveform, yet exhibit new compositions of its audio primitives. This enables a long line of interesting applications, including generating new jazz improvisations or new a-cappella rap variants based on a single short example, producing coherent modifications to famous songs (e.g. adding a new verse to a Beatles song based solely on the original recording), filling-in of missing parts (inpainting), extending the bandwidth of a speech signal (super-resolution), and enhancing old recordings without access to any clean training example. We show that in all cases, no more than 20 seconds of training audio commonly suffice for our model to achieve state-of-the-art results. This is despite its complete lack of prior knowledge about the nature of audio signals in general.

Catch-A- :aan-----aavomm::: Learning to Generate Audio from a Single Short Example
Gal Greshler
Tamar Rott Shaham
Technion Israel Institute of Technology galgreshler0gmail com
Technion - Israel Institute of Technology
stamarot@campus techni on ac il
Tomer Michaeli
Technion Israel Institute of Technology
tomer m@ee. technion. il
Abstract
Models for audio generation are typically trained on hours of recordings. Here, we illustrate that capturing the essence of an audio source is typically possible from little as few tens of seconds from single training signal. Specifically, we present short audio signal from
GAN-based generative model that can be trained any domain (e.g. speech, music, etc.) and does not require pre-training or any other form of external supervision. Once trained, our model generate random samples of arbitrary duration that maintain semantic similarity to the training waveform, yet exhibit compositions of its audio primitives. This enables a long line interesting applications, including generating new jazz improvisations new a-cappella rap variants based single short example, producing coherent modifications to famous songs (e.g. adding a new verse to a Beatles song based solely the original recording). filling-in of missing parts (inpainting), extending the bandwidth of a speech signal (super-resolution) and enhancing old recordings without access any clean training example. We show that in most cases, no more than 20 seconds of training audio suffice for our model to achieve state-of-the-an results. This despite its complete lack of prior knowledge about the nature of audio signals in general.
Introduction
In recent years, deep models for audio generation have had immense impact a wide range of applications, including text-to-speech synthesis [12
38 15 7,, voice-to-voice translation [8] [3),,
timbre transfer [6 40), bandwidth- [37]. Existing generative models require large datasets of training signals from the domain of interest. However, there are practical scenarios in which such datasets are extremely hard to collect, or are even nonexistent. Examples include a speaker that has only recorded few sentences, artist that had the chance record only few songs, unique jazz improvisation appearing in one particular recording. natural question to ask, then, is whether large amounts of training data necessity for training generative model.
music generation 33 μ1. singing voice conversion
53
extension 22
6 and audio inpainting
Here, we take this question to the extreme. We illustrate that capturing the essence an audio source is possible from little as few tens of seconds from single training recording. Specifically, we present generative adversarial network (GAN) based model that can be trained on one short raw waveform and does not require pre-training any other type of external supervision | Once the
"code is available httt:///ggtthbboooogggggeeeeehhh
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Single training example (20 sec.)
Random sample (60 sec.)
Figure 1: --Warffoo.. We present a generative model that able to capture the statistics of single short audio recording (20 seconds in these examples). At inference, it can generate new diverse samples of arbitrary length, that exhibit new interesting compositions. The figure illustrates generation of new jazz improvisations and new freestyle rap variants. All examples can be listened to in our website
model is trained, it is capable of generating diverse new signals that are semantically similar to the training recording, but contain new compositions and structures. Our model can handle different types of audio signals, from instrumental music to speech. For example, after training on 20 seconds of saxophone solo, our model is able to generate new similar improvisations. The same can be done exemplified in Fig I Our model can also generate
with a-capella rap, or old famous speeches, samples conditioned on the low frequencies of some signal (be it the training signal or similar one). This constraints the global structure of the generated signals, allowing generate e.g. new versions of a Beatles song (all audio samples mentioned in the paper can be found in our website).
It is important to note that short snippet of an audio signal is insufficient for learning language (for speech) or rules of harmony (for music). Therefore, our generated signals lack the linguistic semantics or long-range harmonic structure that can be potentially achieved with externally-trained models. However, surprisingly, the coherence of our generated signals over short time scales, typically suffices for confusing listeners to believe they are real, as we confirm through extensive user studies.
Besides generating random samples, we illustrate the utility of our approach in the common tasks of bandwidth extension, inpainting and denoising (see Fig. 2 We show that in the latter two tasks, no training signal whatsoever required beyond the input itself. This allows handling sources for which no training data exist, like old recordings of famous musicians. In fact, our evaluation suggests that for the tasks bandwidth extension and inpainting (sections 4.3 and [4.4), limiting the training to single short signal is actually beneficial, and can lead to results that outperform models trained on hours of recordings.
Our work is inspired by generative models for visual data, which have been recently explored in the context of learning from single image [50] 52 or a single short video 20 Similarly to those works, we present a multi-scale GAN architecture that generates signals in their raw (time domain) representation. Audio signals, however, are very different from visual data; they are of high temporal resolution (usually at least 16,000 samples per second), they exhibit correlations at very long timescales, and they have diverse frequency contents, As we discuss, this necessitates dedicated architectures, losses, and adaptive selection of the multi-scale pyramid levels.
2 Related Work
Generative models for audio. Audio generation models have been extensively studied in the past
few years. Some utilize autoregressive architectures [4 39 including the computationally efficient inverse autoregressive flow (IAF) scheme 42 and other flow based models [27 25 49].
Others use GANs and variational autoencoders (VAEs), which have been found effective for many

---

Training signal
Low-band input
Full-band reconstruction
0.5
2
0.5
2
Noisy training signal (violin recording, 1903)
Denoised signal
10
15
20
10
15
20
Training signal with silent gap
Completed signal
6 [eec]
10
10
[[sce]
Figure 2: Applications. Our method can be used for a variety of tasks, including extending the bandwidth of a low-resolution signal, enhancing noisy signal (without any prior knowledge on the signal or the noise), and completing missing parts.
applications, including text-to-speech
12
38
15 unconditional generation
33
singing
voice conversion g (53, timbre transfer 40) inpainting 13 37), bandwithh-extension 26 and
denoising 45
32
Several pipelines also integrate classical signal processing blocks to obtain
improved results [16]. All these models rely on large training sets with hours of recordings. In
contrast, here focus settings where only single short signal is available for training.
Few shot audio learning. Audio generation models have also been taken to the few-shot regime, mainly in the context of voice cloning for speech [ 9 and singing [41]. In this setting, only few examples are provided at rest-rime. However, large training set is still used for learning to perform
the task. Here, the other hand, we study the use of single short waveform for training.
Internal generative learning. Exploiting the internal statistics of single audio example by training deep neural network (DNN) was recently explored for the tasks of audio restoration, source separation, audio editing, and ambient sound synthesis [62 [55] These methods, however, cannot generate fake signals of complex structure (like music speech). In the visual domain, recent generative models, like SinGAN (50) and InGAN (2), were developed for learning from a single natural image. These approaches were later extended to other domains, including videos 200, medical imaging [61]. and 3D graphics (11. Here we adapt some of these ideas to the audio domain.
Method
Consider a short sample x from stationary audio source. Our goal is to learn a generative model that can draw new random samples ă from the source's distribution. Our approach inspired by the single image GAN (SinGAN) model (50). Specifically, we aim at matching the distribution of length-T segments of ž to that of length-T segments of z, at multiple resolutions.
Analysis pyramid. We start by constructing an analysis pyramid of the training signal,
Zo I,
3r = (x * hn) Id.., n= 1, N,
(1)
where < d2 <... dy are down-sampling factors and hı,..., It, are corresponding anti- aliasing filters. This is illustrated the top of Fig. Denoting the sampling rate of by (usually 16Khz in our experiments), we have that the sampling rate at the nihh pyramid level ʃ" = f Similarly, denote by žn the wth level of the multi-scale representation of the fake signal ā.

---

Analysis pyramid
Training signal
AN-1
DN
Dz-1
Do
Generated signal
g
GN
GN-1
Ga
Noise
n2.
Synthesis pyramid
Figure 3: Model illustration. Our model is built from pyramid of generators that operate at gradually increasing sampling rates, each fed by the preceding onc. Adversarial training is performed sequentially in coarse-to-fine manner, using corresponding pyramid of discriminators.
Figure 4: Generation process. Our generation process gradually increases the frequency range of the signal. The receptive field of generators is same. This translates to larger effective receptive fields (shaded rectangles) at the lower sampling rates, which shape the global structure of the signal.
Synthesis pyramid. The generation of fake sample ă is performed sequentially by generating each of its pyramid levels conditioned on the previous one, from coarse to fine. Specifically,
iy (ZN))..
in = Gn (2n: (#n+1) 1a"), n n -
where is white Gaussian noise, Gn, is a convolutional neural network generator, = dd+11/d is the resolution ratio between scales n + 1 and n, and (.) ta stands for up-sampling by a factor of using cubic interpolation [24]. The signal To at the end of this process is the generated fake sample i. This synthesis pyramid is shown the middle and bottom rows of Fig. All generators have the same receptive field, as measured in samples. This translates to larger effective receptive fields (in seconds) for the coarser levels than for the finer ones. As a result, the coarsest scale can capture the long-range dependencies that are typical of low frequencies of audio signals. Each subsequent generator, then, only needs to add narrow band frequencies to the signal generated at the previous scale (see Fig. 4.. The higher the frequency band, the smaller the receptive field that suffices to achieve this goal. Following this understanding, we take the variance of 2n to be proportional to the energy of x in the frequency band [ fx f which is at the responsibility of the generator G, to synthesize. is important to note that as opposed to images, audio signals tend to exhibit long-range dependencies even the highest frequency bands. Therefore, we take the receptive field in samples to be three orders magnitude larger than the resolution factor a between scales (see below). This is opposed to the SinGAN image model [50], which uses only one order of magnitude.
Training. We train our model a coarse to fine manner as well. each stage, a single generator in the pyramid is trained while the generators of all coarser levels are kept fixed. When training the nth level, the goal to drive the distribution of length-7 segments within F, become as close possible to the distribution of length-T segments within 2n. To this end,
patch-GAN

---

Dilated conv-blocks
2n
tanh
PE filter
x
xxl
Xn
1+
Figure 5: Synthesis at a single scale. The generator at the rith scale gets an up-sampled version of the signal generated at the previous scale, (#n+1) 1o, which has frequency contents in the range [0, 1 f Together with a noise realization 2n, it generates a signal En with frequency contents in [0, // This is done with a residual architecture involving 8 dilated convolution blocks; 7 of the convolutional only. Dilation grows by factor of in each block. We add gated activation at the end of the generator, followed by fixed pre-emphasis filter.
form conv-BN-leakyReLJ.
framework 3 which employs convolutional discriminator network D,, with receptive field T. The discriminator tasked with classifying each of the overlapping length-T windows in its input as real or fake, so that its output is classification sequence of the same length as the input (minus T 1 samples). The final score of the discriminator the mean of this classification sequence. We specifically use the Wasserstein GAN loss
4].
Lad (Dn, G.) = E P-Pan
[Dn(
E [D,(),)),
(3)
together with a gradient penalty [18] Additionally, we pick a particular input at each scale, 2,,- and enforce that its corresponding reconstructed signal, o = (2,)), be close to the real signal 2,, at that scale. This ensures that there is at least one point in the latent space of our model that maps to the real signal. We do this via reconstruction loss,
(4) where the second term is the multi-scale spectrogram (MSS) loss 42 which penalizes for differences between spectograms (thus disregarding phase). We use the particular MSS formulation
Lee a1l|= - + MSS(Ins, D,,,,
of [11] (see SM). For the reconstruction sequences, we choose {=], .y....... 2f1 = f2", 0,., 0}.
where is fixed white Gaussian noise realization. Therefore, overall, solve
min max Laty (Dns (D,GG)) G.)) + (Gn),
(5)
where we alternate between performing one update step for Dn, which also involves minimizing the gradient penalty term, and one update step for G,. In practice, we typically use only one of the terms in (setting the other coefficient to 0), depending on the application (see Sec.
Architecture. The generators and discriminators at all scales have the same fully-convolutional architecture. We use stacked blocks of dilated convolutions, followed by batch normalization and leaky ReLU with slope 0.2. The dilation factor grows by factor of 2 in each layer, which is known to be an effective way for increasing of receptive field 60 At the end of the generator we use the gated activation unit [44), which is an element-wiss product of tanh and sigmoid, each fed by an extra non- dilated convolution. All of our convolution layers have kernel size of 9, which leads to total receptive field of 2040 samples at each scale. We use weight normalization [51). which we found to improve results and training stability. At the end of the trainable blocks, we add a fixed pre-emphasis (PF) filter with impulse response [1, 0.97). which amplifies the high frequencies, as common in similar tasks 588 59]. An illustration of the generator's 's architecture is shown in Fig. [
Automatic scales selection. Different types of audio signals can have very different power spectra, as we illustrate Fig. σ This suggests that the frequency bands of the pyramid should be adaptively chosen based on the spectrum of the training signal. However, to allow for efficient implementations of resampling techniques, we also want the sampling rates of all scales to be rational factors (with small denominators) of f [14] ch. 9]. We therefore use predefined discrete set of potential sampling rates, and choose our bands adapitively only from this set. As can be seen in Fig. ( up to 2Khz, where most of the audible energy resides, the predefined scales grow at a factor of around 1.25. The mid-range, 2-4Khz, typically contains less energy and so the scales are sparser there. Finally, to be able to capture the energy of non vocal syllables in speech signals, the scales become denser again
5

---

Pap&Rack Songs
10
Speech
Fiute
-10
-20
-30
160
200
250
320
400
800
640
800
7000
1250
2000
4000
5000
8000
7200
8000
[Hz]
Figure 6: Scales selection. The plot depicts the power spectral densities of three different audio
datasets (rock and pop songs (56). speech [23] and monophonic flute (63]. The dashed black lines show the predefined band partitions (note the logarithmic axis). The first band is adaptively chosen to contain enough energy. The bold colored lines show typical first scale chosen for each dataset.
from 4Khz. practice, the most sigmificant effect due the automatic selection of the first band, which shapes the global structure of the signal. We therefore choose automatically only this band, such that it contains enough energy (see SM). Fig. shows typical selections of the first band for different types of audio signals. Additional spectra are presented in the SM.
4 Experiments
We test our catch-a-waveform (CAW) method in several applications and evaluate it both qualitatively and quantitatively. Our training examples contain variety of audio types, including polyphonic rock and pop music, monophonic instrumental music, speech, and ambient sounds. Unless noted otherwise, all training signals have a sampling rate 16Khz. For training, we use the Adam optimizer (28) with (B1, 32) = (0.5, 0.999) and learning rate 0.0015, which we reduce by factor of 10 after two thirds of the epochs (we run total of 3000 epochs). Training on 25 second long signal takes about 10 hours on Nvidia GeForce RTX 2080. Inference is 60 times faster than real-time.
4.1 Unconditional generation
Monophonic music. We trained CAW models on monophonic music played by various instruments, including cello, violin, saxophone, trumpet, and electric guitar. Here, we used
O and (:2 10-1 ④ We trained on signals of length 25 to 100 seconds, and at test time generated signals of various lengths by simply injecting input noise signals of appropriate length (see SM for additional details). The generated signals sound like variations or naive improvisations on the original piece (see website).
Speech signals. We further trained CAW models on various human voice recordings, with lengths varying from to 40 seconds. These include short segments from speeches of American presidents Trump and Obama and a-capella rap. Here we used a₁ 10, generated random samples of lengths between 20 and seconds. As exemplified our wwebsite| the generated signals preserve the voice of the speaker, but exhibit new compositions of syllables, words, intonations and silent gaps. Note that since our model has no notion of language, the generated signals are not necessarily interpretable. The temporal coherence of the generated signals can be controlled by changing the receptive field of the model. As we illustrate in the website reducing the receptive field from 4 to 2 seconds (by removing become less coherent and makes the generated speech sound like mumbling. Increasing the receptive field to 8 seconds, on the other hand, preserves short sequences of words from the training signal.
0 in 4 At inference, we
convolutional block), causes the structure to
Human perception tests. In order evaluate the perceptual quality of our generated signals, we conducted auditory studies through Amazon Mechanical Turk (AMT). The studies were per- formed on solo signals of different instruments (saxophone, trumpet, violin, flute, clarinet, cello, accordion and distorted electric guitar) randomly chosen from the Medly and the solo-audio dataset [63]. For each instrument, we randomly cropped 25 second long segments from 7 different parts of the recording to serve as real signals (56 segments in to- tal) and used our method to generate a 10 second long fake version for each them. We per- formed two types of user studies: (i) a paired test, where the real signal and its fake version were played sequentially, and the user was asked to choose which the fake, and (ii) an unpaired test, where the user listened to single signal and had to determine whether it is real or fake.
dataset [34]

---

30
30
25
10
10
Figure 8: Similarity matrix. Distinct diagonal lines correspond to segments in the generated signal that were 'copied' from the real signal. Naively stitched signals show clear lines in the copied parts (left). In our generated signal (right) more lines be seen the model can maintain global temporal structure while combining higher frequencies from different temporal locations.
Each test opened with tutorial of 5-8 questions identi- cal to the structure of the main test, but with feedback to the user. We also had an additional version, (iii) an unpaired test with a paired tutorial. In this case users were exposed to examples of paired real and fake signals during the training phase, but the test itself was unpaired. In each the tests, we had 50 different users answer 25 questions each. The results summarized in Fig. 7 As can be seen, in all the studies the confusion rates relatively high (the ultimate rate being 50%). As pected, the confusion rate of the paired test is lower than the unpaired test, this setting less challenging. But there is no significant difference between the results of the two unpaired tests, suggesting that paired tutorial does not help the listener perform better discrimination.
Ungained test unpaired
Unpaite test paired catoral
Paicel test paized tutorial
tutorial
Figure 7: Real-vs.-Fake AMT studies. Users were asked to discriminate between our generated signals and real ones, both in paired and in unpaired tests. We present confusion rates and [5%, 95%) confidence intervals. As can be seen, the confusion rates in all cases close the ultimate rate, which is 50%.
Comparison to naive copy-and-paste. As model is trained on a very short signal, its ability to generate semantic contents is limited. For example, will most likely not generate syllables or notes that did not exist in the training signal. However, does that mean it naively copy and pastes segments from the training signal? To examine this, depict in Fig. g (right pane) similarity matrix between small overlapping patches of generated signal and the training signal. We specifically use the cosine similarity between the vectors of absolute values of the discrete Fourier transforms of the patches (see more details about the similarity matrix calculation in the SM). In this visualization, matching segments appear as diagonal lines. For comparison, we also show the same visualization for naively generated signal constructed by stitching together cropped segments from the training signal, with crop sizes randomly chosen between the receptive fields of our finest and coarsest scales. Stitching done using cross fading. We ensured that the naively stitched signals sound realistic by performing another unpaired user study, as in the left bar of Fig. 7 and got a confusion rate of 47.88%. While the similarity matrix of the naively stitched signal shows clear solid lines, in generated signal sometimes two or more parallel, weaker, lines one above the other. This suggests that our model often mixes several parts from the training signal. This happens thanks to our multi-scale architecture that allows each generated frequency band to contain contents from a different temporal location in the training signal.
observe
4.2 Conditional generation of music variations
Another interesting application is generating variations extensions of existing songs (e.g. adding new verse). do so, we first train our model on popular rock or pop song. Then, at inference time, we start the generation from the second coarsest scale by injecting the real (training) signal as input to that scale. This ensures that generated signal maintains the global structure of the real signal, as its low frequencies are constrained to be the same. But finer details, like the lyrics, are randomly generated. Here we take (1 = and 02 10-4 in 4 Also, to encourage large variability between different random samples, set the input noise in the second coarsest scale to have the same energy

---

Original verse (used for training)
Generated verse
It's been a hard
day's
night
It's been hard
day's day
and 've been working
like dog
chi get ya money
to ba log
It's been a hard
day's
night
It's been hard
all
day
should be sleeping
like log
you got me working
like a dog
Figure 9: Music variations. After training specific song, we can inject down-samplen sampled version of the song to the second coarsest scale of the model. This way, our model generates a signal having the same structure as the original song, but with randomly generated finer details, like lyrics. In this example we generate a new verse "A Hard Day's Night" by The Beatles, after training on its two first verses. Modified lyrics are shown in red.
Low-resolution audio
High-resolution audio (GT)
4
0
1.5
2.5
4.5
Extended by TFiLM 6
Extended by CAW (ours)
2 [K3R
0.5 1.5
2.5 [[xo]]
3.5
4.5
[sec]
A model trained on one short high-bandwidth signal (25 seconds in this case) can be used at test time to increase the bandwidth of any lowbbandwidth signal of the same speaker (by injecting it to coarse scale of the model). This results in sharper reconstructions than those obtained with TFiLM, which was trained on hours of examples.
Figure 10: Bandwidth extension. We use our model for speech handwidth-extensinn
as that of the real signal at that scale. Examples can be found in the website and in Fig. 9 which shows new verse generated by model the famous Beatles song "A Hard Day's Night".
4.3 Bandwidth extension
Bandwidth Extension (BE) is the task of reconstructing a bandwidh signal from its low- bandwidth version, and is usually demonstrated on speech [29] 6 26 19 55) and music [30 (54). To perform BE using CAW, we first train it on a high-bandwidth short audio example of specific speaker. At inference time, we can then inject any other low-bandws www-bandwidth signal of the same speaker to coarse scale of the model (we choose the scale whose sampling rate matches that of the input signal). Here we use (1) = 10 and (22 = 0 in 4 We then stitch the reconstructed higher frequencies generated by our model with the low frequency range the input signal to obtain our final full- bandwidth reconstruction. Figure 10 shows BE example, where the sampling rate of a speech signal increased from 4KHz to 16KHz, Our handwidth-extendde signals contain realistic high frequency contents, which makes them sound sharp (see examples and comparisons in our website including for the easier task extension from 8KHz to 16KHz).
We compare our BE results to the the state-of -hrt temporal FiLM (TFiLM) method [6] which requires a large training set to perform this task. We use the VCTK dataset, and report both the signal to noise ratio (SNRs) and the log spectral distance (LSD) [17 between the recovered signal and the ground-truth one, averaged over test set. LSD known to better correlate with human perception. We perform comparisons to several TFiLM variants, following the protocols of (6).

---

Table 1: Bandwidth extension quantitative evaluation. We compare our method TFiLM 6 using SNR (higher is better) and LSD (lower is better) both for multi-speaker test and sinle-speaker test. In all cases our model achieves better LSD scores, indicating of higher perceptual quality.
Multi speaker test
Single speaker test
TFiLM
6
CAW (ours)
TFiLM 66
CAW (ours)
Training set size [min]
25
240
600
0.4
30
0.4
SNR [dB] f
14.66
14.83
15.45
13.8
0.94
14.77
13.03 + 0.83
LSD 4
4.96
3.89
3.79
2.97
0.26
3.92
3.03 + 0.26
Single-speaker baseline. In this setting, we train a separate CAW model for each of 9 speakers, and then test each of the models on a set of held-out sentences of the same speaker. For TFiLM, we use 30 minutes of training data for each speaker, and for our model we use only 25 seconds. As can be seen in Table our model outperforms TFiLM in LSD, but achieves a slightly lower SNR (we report mean and standard deviation over 50 different trained models).
Multi-speaker baselines. Here, we train TFiLM on 99 speakers from the VTCK dataset and test on the remaining 9 speakers. We have three variants, corresponding to training sets of 25 minutes, 4 hours, and 10 hours. Our model is trained as in the Linle-speaker case. We use the same test set for evaluating both methods. As can be seen in Table [ our model is again superior in terms of LSD compared to all TFiLM variants, and is slightly worse in terms SNR.
4.4
Audio inpainting
Audio inpainting refers to the task of completing a missing part of given audio signal. It has been pre- viously addressed using classical signal processing methods I 5 35), graph-based approaches [46]
Table 2: Inpainting AMT study. Users chose between our model (trained on 12 seconds), GACELA (trained on & hours), and the ground- truth signals. The preference rates indicate that our results are at least comparable to GACELA, and are often confused to be real signals.
and neural networks 13 36 37/. Here, we address the long-img tin-iinnin task, where several hundred milliseconds are missing. We do this by training CAW with slight adaptations: (i) we calculate the loss with respect to only the valid parts of the sig- nal (excluding the gap), and (ii) we sample a new reconstruction noise realization for the missing part at each iteration. Here we use 01 = 10 and a2 = O in (4). After training, we take the completed part from the reconstruction, and stitch it with the input. As can be seen in Fig. our model coherently completes the missing part, and thanks to its relatively large receptive field, the completion smoothly fuses with the valid parts. Examples of completed rock songs can be found in our website
Study
Preference rate 55.3% + 2.4% 44.3% = 2.3%
Ours vs. GACELA Ours vs. Real
Human perception tests. We evaluated our results using AMT user preference test. We took 64 rock songs from the FMA-small dataset [10) extracted 12 second long segment from each, and masked a 750ms window. We compared our results with those of GACELA (37] a GAN based context encoder trained on roughly 8 hours of rock songs from the same dataset. In each query, raters listened to a 5-9 second long segment containing the missing gap, well as to our and to GACELA's completions. They could re-listen to all signals as many times as they wanted, and eventually had to pick the completion that sounded better. In total, 50 raters answered 20 queries each. As can be seen in Table 2 raters preferred our completions over 50% of the times, suggesting that the performance of our method is at least comparable to GACELA's. We also performed user study that compared our completions to the real signals. Interestingly, the preference rate for our completion was still relatively high (see Table 2). indicating that our results often indistinguishable from real signals.
4.5
Audio denoising
An interesting side effect of CAW's training process, is that it can be used for audio denoising. As explained in Sec. during training we enforce a certain noise hypothesis to generate a reconstruction of the training signal. We found that when training our model on a noisy signal, this reconstruction often preserves harmonic parts while suppressing ambient noise. This effect is more distinct when

---

Spectogram of signal with silent gap
Spectogram of signal inpainted by our model
0
10
2
10
[sece]
[seec]
Figure 11: Audio inpainting. Our model is able to complete mising silent gap in given signal without any additional information other than the signal with the gap itself. We train our model on the valid region of the signal excluding the missing part, learn its internal statistics, and then at test time we generate the missing gap, which results in a coherent realistic completion.
Noisy
Denoised
[sec]
[Secc]
Figure 12: Denoising. When trained on single noisy example, our model produces clean reconstruction. This enables to denoise e.g. old recordings, illustrated here on one of the first violin recordings, from 1903. Here, no ground-truth data is available for computing SNR. However, in controlled experiments with modern recordings of the same musical piece, we measure improvement of 1.5dB 4dB in SNR (see text for details).
using 01 = 10 and = O in 44. This enables to perform denoising without access to a any clean example for training, and without any prior knowledge about the noise distribution. This is in contrast to externally supervised approaches, which require many pairs of noisy-clean examples, egg. 332 As an example, we demonstrate denoising old recordings of the violinist Joseph Joachim from 1903 (for which obviously clean training examples can be collected), As seen in Fig. and on our website, the reconstructed signals are notably cleaner than the original ones. To took (clean)
obtain some quantitative measure of the denoising performance in this experiment, modern recording the same musical piece (Bach's 's Adagio), preformed by famous violinist Hilary Hahr We synthetically generated noisy versions of this recording using both white noise and old gramophone noise recording (see the SM and our websiite). For each noise type, we examined noise levels of 5dB and 10dB. For the white noise, denoising increased the SNR from 5dB 9.78dB and from 10dB > 11.53dB. For the gramophone noise, it increased SNR from 5dB - 6.89dB and from 10dB 11.56dB. We believe these results can be further improved in the future by optimizing the model for the specific task of denoising.
Conclusion and limitations
We presented a new GAN-based model for audio generation that can be trained on a single short example. Our model works learning from single waveform often has advantages over learning from large datasets. We believe this learning scheme be further developed general framework for training 'personalized' deep learning models, where personal small data would be sufficient for a variety of tasks. This approach, however, not free of limitations. Models trained on small data cannot learn high level semantics. Moreover, they can sometimes struggle even with low-level attributes. For example, our model is occasionally less successful in handling high-pitched speech signals, like that of female or child. Another challenging scenario is speech recorded in reverbrant environments (e.g. in large conference room), where our model tends transform the reverberations into high-pitcher noise. Examples for variety of such failure cases can be found in our website
waveforms, and is useful for variety of tasks. As illustrated,
Ptaken from http::/wwwww...u....cc./mw///tth
Staken from fwom
10

---

Acknowledgements This research was supported by the Israel Science Foundation (grant 852/17) and by the Technion Ollendorff Minerva Center.
References
[1] Adler, A., Emiya, V., Jafari, M.G., Elad, M., Gribonval, Plumbley, M.D.: Audio inpainting. IEEE Transactions on Audio, Speech, and Language Processing 20(3), 922-932 (2011)
[2] Arik. S., Chen, J.. Peng. K., Ping, W. Zhou, Y:: Neural voice cloning with few samples. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi N., Garnett, R. (ods.) Advances in Neural Information Processing Systems. vol. 31. Curran Associates, Inc. (2018)
[3] Ank, S.Ö., H., Diamos, G.: Fast spectrogram inversion using multi-head convolutional neural networks. TEFE Signal Processing Letters 26( 1), 94-98 (2018)
Arjovsky, M., Chintala, Bottou, L.: Wasserstein generative adversarial networks. In: International conference on machine learning. pp. 213. PMLR (2017)
[5] Bahat, Y. Schechner, Y.Y., Elad, M.: Self-comtent-based audio inpainting. Signal Processing 111, 61-72 (2015)
[6] Birbauu, S., Kuleshov, V., Enam, Z., Koh, P.W.W., Ermon, S.: Temporal FiLM: Capturing long-range sequence dependencies with feature-wise modulations. In: Advances in Neural Information Processing Systems (2019)
Binkkowki, M., Donahue, J., Dieleman, S., Clark, A., Elsen, E., Casagrande, N., Cobo, L.C., Simonyan, K.: High Fidelity Speech Synthesis with Adversarial Networks. International Conference on Learning Representations (2020)
[8] Chandna, P., Blaauw, M., Bonada, J., Gómez, E.: Wgansing: A multi-voice singing voice synthesizer based on the Wasserstein-Gan GANN, In: 2019 27th European Signal Processing Conference (EUSIPCO), pp. 1-5. IEEE (2019)
[9] Chen, Y.H., Wu, D.Y., Wu, T.H., Lee. H.y.: Again-VCa VC: one-shot voice conversion using activation guidance and adaptive instance normalization. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 5954-5958. IEEE (2021)
[10] Defferrard, M., Benzi, K., Vandergheynst, P., Bresson, X.: FMA: A dataset for music analysis. In: 18th International Society for Music Information Retrieval Conference. No. CONF (2017)
[11] Dhariwal, P., Jun, H., Payne, C., Kim, J.W., Radford, A., Sutskever, I.: Jukehox: A generative model for music. arXiv preprint arXiv:2005.00:: (2020)
[12] Donahue, C., McAuley, J. Puckette, M.: Adversarial audio synthesis. In: International Conference Learning Representations (2018)
[13] Ebner, P.P. Eltelt, A.: Audio inpainting with generative adversarial network. arXiv preprint arXiv:2003.07704 (2020)
[14] Eldar, Y.C.: Sampling theory. Cambridge University Press (2015)
[15] Engel, J., Agrawal, K.K., Chen, S., Gulrajani, I., Donahue, C., Roherts, A.: GANSynth: Adversarial neural audio synthesis. In: International Conference on Learning Representations (2018)
[16] Engel, J., Gu, C., Roberts, A., et al.: DDSP: Differentiable digital signal processing. In: International Conference on Learning Representations (2019)
[17] Gray, A., Markel, J.: Distance measures for speech processing. IEEE Transactions Acoustics, Speech, and Signal Processing 24(5). 380-391 (1976)
[18] Guirajani, L., Ahmed, F., Arjovsky. M., Dumoulin, V., Courville, A.: Improved training of Wasserstein GANs. Proceedings of the 31st International Conference on Neural Information Processing Systems. pp. 5769-5777777 (2017)
[19] Gupta, A., Shillingford, B., Assael, Y., Walters, T.C.: Speech bandwidth extension with WaveNet. In: 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 205-200 IEEE (2019)

---

[20] Gur, S., Benaim, S.. Wolf, L.: Hierarchical Patch VAE-GAN: Generating diverse videos from single sample. arXiv preprint arXiv:2006.1222) (2020)
[21] Hertz, A., Hanocka, R., Giryes, R., Cohen-Or, D.: Deep geometrie texture synthesis. ACM Transactions on Graphies (TOG) (9),,, 108-1 (2020)
[22] Isola, P., Zhu, J.Y. Zhou. T., Efros, A.A.: Image-to-inage translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125- 1134 (2017)
[23] Ito, K., Johnson, L.: The LJ speech datasct.
com/LJ-a Soo/LJ--ppeee Speech--atassee (2017)
[24] Keys, R.: Cubic convolution interpolation for digital image processing. IEEE transactions on acoustics, speech, and signal processing 29(6). 1153- 1160 (1981)
[25] Kim, J., Kim, S., Kong, J., Yoon, S.: Glow-TTS: A generative flow for text-to-specch via monotonic alignment search. Advances in Neural Information Processing Systems 33 (2020)
[26] Kim, S., Sathe, V. Bandwidth extension on raw audio via generative adversarial networks. arXiv preprint wxXvv1903.09900 .00027 (2019)
[27] Kim, S., Lee, S.G., Song, J., Kim, J., Yoon, S.: FloWaveNet: generative flow for raw audio. In: International Conference on Machine Learning. pp. 3370-3378. PMLR (2019)
[28] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (cds.) 3rd International Conference on Learning Representations, ICLR (2015)
[29] Kuleshov, V., Enam, S.Z., Ermon, S.: Audio super-resolution using neural nets. In: ICLR (Workshop Track) (2017)
[30] Lagrange, M., Gontier, F:: Bandwidth extension of musical audio signals with no side information using dilated convolutional neural networks. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 801-805. IEEE (2020)
[31] Li, C., Wand, M.: Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: European conference computer vision. pp. 702-716. Springer (2016)
[32] Li, Y., Gfeller. B., Tagliasacchi. M., Roblek, D.: Learning to denoise historical music. International Society for Music Information Retrieval (ISMIR) (2020)
[33] Liu, J.Y., Chen, Y.H., Yeh, Y.C., Yang, Y.H.: Unconditional audio generation with generative adversarial networks and cyele regularization. Proc. Interspeech 2020 1997- 2001 (2020)
[34] Lostanlen. V., Cella, C.E.: Deep convolutional networks on the pitch spiral for musical instrument recognition. International Society for Music Information Retrieval (ISMIR) (2016)
[35] Manilow, E., Pardo, B.: Leveraging repetition to do audio imputation. In: 2017 IEBE Workshop on
Applications Signal Processing to Audio and Acoustics (WASPAA). pp. 309-313. IEEE (2017)
[36] Marafioti, A., Holighaus, N., Majdak, P., Perraudin, N., et al.: Audio inpainting of music by means of neural networks. In: Audio Engineering Society Convention 146. Audio Engineering Society (2019)
[37] Marafioti, A., Majdak, P., Holighaus, N., Perraudin, N.: GACELA-A generative adversarial context encoder for long audio inpainting of music. IEEE Journal of Selected Topics in Signal Processing (2020)
[38] Marafioti, A., Perraudin, N., Holighaus, N., Majdak, P.: Adversarial generation of time- fime-frequency features with application in audio synthesis. In: International Conference on Machine Learning, pp. 430243354444 PMLR (2019)
[39] Mehri, S., Kumar, K.. Guirajani, I., Kumar, R., Jain, S., Sotelo, J.. Courville, A.C., Bengio. Y.: SampleRNN: An unconditional end-to-end neural audio generation model. In: 5th International Conference on Learning Representations, ICLR (2017)
[40] Michelashvili, M., Wolf, L.: Hierarchical fumbre-painting and articulation generation. International Society for Music Information Retrieval (ISMIR) (2020)
[41] Nercessian, S.: Zero-shot singing voice conversion. In: Proceedings of the International Society for Music Information Retrieval Conference (2020)
12

---

[42] Oord, A., Li, Y., Babuschkin, I. Simonyan, K. Vinyals. O., Kavukcuoglu. K., Driessche, G., Lockhart, E., Cobo, L.,, Stimberg, F., al.: Parallel WaveNet: Fast high-fidelity speech synthesis. In: International conference on machine learning, pp. 3918-996... PMLR (2018)
[43] Oord, A.v.d., Dieleman, S.. Zen, H., Simonyan, K.. Vinyals. O., Graves, A., Kalchbrenner. N., Senior, A.. Kavukeuoglu, K.: WaveNet: generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016)
[44] Oord, v.v.d., Kalchbrenner, N., Espeholt, L.,, Kavukeuoglu, K., Vinyals, O., Graves, A.: Conditional image generation with PixeICNN decoders, In: NIPS (2016)
[45] Pascual. S., Bonafonte, A., Serrà, J.: SEGAN: Speech Enhancement Generative Adversarial Network. Proc. Interspeech 2017 pp. 3642-3646 (2017)
[46] Perraudin, N., Holighaus, N., Majdak, P., Balazs, P.: Inpainting of long audio segments with similarity graphs. IEEE/ACM Transactions on Audio, Speech, and Language Processing 26(6). 1083-1094 (2018)
[47] Ping. W., Peng, K., Chen, J:: ClariNet: Parallel wave generation in end-to-end text-to-speech. In: Interna- tional Conference on Learning Representations (2018)
[48] Ping, W., Peng, K., Zhao, K., Song. Z.: WaveFlow: A compact flow-hased model for raw audio. In: International Conference on Machine Learning. pp. 7706-7716, PMLR (2020)
[49] Prenger, R., Valle, R., Catanzaro, B.: WaveGlow: flow-based generative network for speech synthesis. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 3617-3621. IEEE (2019)
[50] Rott Shaham, T., Dekel, T., Michaeli, T. SinGAN: Learning a generative model from a single natural image. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4570-4580 (2019)
[51] Salimans, T., Kingma, D.P. Weight Normalization: A simple reparameterizztioo to accelcrate training of deep neural networks. In: NIPS (2016)
[52] Shocher, A., Bagon, S., Isola, P., Irani, M.: InGAN: Capturing and remapping the "DNA" of a natural image. In: International Conference Computer Vision (ICCV). vol. 1, p. 2 (2019)
[53] Sisman, B., Li, H.: Generative adversarial networks for singing voice conversion with and without parallel data, In: Speaker Odyssey. pp. 238-244 (2020)
[54] Sulun, S., Davies, M.E.: On filter generalization for music bandwidth extension using deep neural networks. TEFE Journal of Selected Topics in Signal Processing (2020)
[55] Tian, Y., Xu, C., Li, D.: Deep audio prior. arXiv preprint arv:: 1tt::199...00 191299929999 (2019)
[56] Tzanetakis, G., Cook, P.: Musical genre classification of audio signals. IEEE Transactions on speech and audio processing 10(5), 293-302 (2002)
[57] Wang, M., Wu, Z., Kang. S., Wu, X., Jia, J., Su, D., Yu, D., Meng, H.: Speceh super-resolution using parallel WaveNet. In: 2018 11th International Symposium on Chinese Spoken Language Processing (ISCSLP). pp. 260-264. IEEE (2018)
[58] Wright, A., Damskägg, E.P., Välimäki, V., et al.: Real-time black-box modelling with recurrent neural networks. In: 22nd International Conference Digital Audio Effects 19) (2019)
[59] Wright, A., Välimäki, V.: Perceptual loss function for neural modeling of audio systems. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 251-255 IEEE (2020)
[60] Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. International Conference on Learning Representations (ICLR) (2016)
[61] Zhang, P., Zhong, Y., Tang, X., Deng, Y., Li, X.: Learning diagnosis of COVID-19 from single radiological image. arXiv preprint arxiv:2006. 12220 (2020)
[62] Zhang. Z., Wang, Y.. Gan, C., Wu, J., Tenenbaum, J.B., Torralba, A., Freeman, W.T.: Deep audio priors emerge from harmonic convolutional networks. In: International Conference on Learning Representations (ICLR) (2020)
[63] Zhousl16: solo audio, http://www..aggll...m///ooooll//
audio (2019)
13


Reviews:
[{"review_id": "qJkAO4-KPq", "reviewer": "Reviewer_Qtuh", "summary": "The authors demonstrate that a SinGAN model can be adapted for audio, training a model on a single instance of audio, and generating continuations and inpainting in both the time and frequency domain. They utilize a multi-scale decomposition to create a pyramid of generators and discriminators, and also force reconstructions for one of the points of the generator's latent space. They make several model accomodations to training on the audio domain, including a criteria for automatically tuning the selected scales of their pyramid to the low-frequency power content of the target audio. They show many applications for their technique, including unconditional temporal continuations, conditional regeneration of higher frequency content to create new song sections, bandwidth extension of low resolution audio, inpainting of audio segments, and implicit denoising through the reconstruction signal. Qualitative and quantitative results support their findings, including listening tests and an extensive page of examples. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality / Significance\n\n\n-- Straightforward extension of previous SinGAN work to a new domain, so novelty of the approach is limited.\n\n++ However, the authors do a good job of adapting the technique to the audio domain, and demonstrating that it is possible, which is not a forgone conclusion. Further, the breadth of creative applications they demonstrate are of broad interest to the community and show the further potential of the original SinGAN architecture.\n\n++ In particular, the authors provide several clever adaptations of the model (such as generating new song sections by conditionally sampling higher frequencies, and denoising by using the reconstruction signal) that provide new insight into the domain and are not at all straightforward to expect from this model.\n\n\nQuality / Clarity\n\n++ The paper is clearly written, laid out in a logical manner, with ample figures and examples to demonstrate the many examples and applications proposed. \n\n++ Quantitative results are presented, including listening tests for some of the more qualitative examples such as the quality of continuations. \n\n-- While it is helpful to have the paired / unpaired / tutorial splits in Table 1, the study is lacking a reasonable baseline comparison, such as extending the audio by concatenating (with crossfades) audio chunks from the original audio. A pairwise comparison, or even mean opinion score comparison, would be informative for the quality of the technique rather than the current Turing test of \"Is this audio fake?\"\n\n-- More quantitative metrics for the audio denoising experiments would help to understand the quality of the denoising. There are many standard denoising benchmarks where noise is added to clean signals and SNR can be computed.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "++ It is great that limitations of the quality of the generation are considered in both the main text and the supplement, and the examples provided are very informative. \n\n-- That said, there seem to be stronger limitations to the technique in the audio domain that are less perceptually relevant in the image domain. In particular, certain high-level structure such as language (both in english and in terms of musical notes) are not captured by training on a single example. For example the song continuations created garbled pseudo-language mixes of phonemes because they haven't seen words not present in the original clip. Similar for the musical note motifs of continuations, they are more \"smoothed concatenations\" of the original segments than a true improvisation would consider (there's only so much high-level structure that can be present in a small clip of audio). This is a fundamental limitation of this technique that is less relevant for SinGAN on images as the application of continuing the texture of a mountain skyline of fleet of hot air balloons doesn't require generating novel high-level structure (say a building on the hillside or airplanes flying among the balloons). That said, generating new song segments or improvisations does seem to require new high-level structure (new words, new motifs etc.) and this technique will be limited in these applications unless it is modified to incorporate this outside knowledge from elsewhere. Further highlighting and exploration of these limitations would give a better sense for what this technique does and does not do.\n\n+/- There are not any discussions of Societal Impact, which would be a problem if the model were more convincing at speech continuation, as voice cloning is worth considering as a negative impact. That said, the limitations mentioned above about high-level structure and control limit the possible impact of this particular work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "UbSt1lPXwar", "reviewer": "Reviewer_Bdte", "summary": "This paper presents a strategy for learning to generate convincing audio using only a short audio example as training data. The method involves iteratively upsampling the audio and shaping whitre noise, a procedure which lends itself to variable-length unconditional synthesis as well as several downstream conditional generation tasks such as bandwidth extension and inpainting.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is an excellent paper and the results are quite impressive. I have no doubt that this will be an influential paper within the field of deep audio synthesis. The writing is also very clear and easy to follow, and the method is simple and intuitive. Additionally, the paper appears to have put forth (but not advertised) a solution to a major issue with past work on audio synthesis: the ability to generate audio of arbitrary length with continuity across receptive field chunks.\n\nIronically, while the paper primarily advertises its strengths at unconditional generation, I am generally more convinced by the _conditional_ generation tasks like music variation generation, bandwidth extension, inpainting, etc. The unconditional generation sounds much more like recombination than generalization. This isn’t necessarily a major criticism as recombination of audio with local continuity is difficult to achieve, as small discontinuities which occur from naive recombination can be perceptually devastating. However, it does sound like the model is just stitching together short windows of audio, not unlike what can be accomplished with traditional granular audio synthesis (in ML parlance, granular synthesis might be called a “retrieval-based” method).\n\nOne analysis that would be very useful to see is a similarity matrix between the ground truth and the generated waveform in some audio feature space. I suspect such a figure would make it much easier (compared to listening) to determine the temporal correspondence between the generated result and the original training excerpt, which would either help confirm or discredit my hypothesis that the model is stitching together copied segments of the training data. However, even if it is the case that it is mostly copying, I reiterate that this is still nontrivial, and I would prefer to see the limitation addressed rather than swept under the rug.\n\nRegardless of potential limitations in the unconditional setting, the number of conditional generation tasks that this model excels at is quite impressive. The proposed method appears ready for immediate application, and will be of interest to audio engineers, musicians, and sound artists alike.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors explicitly address some limitations of their method in the main text of the paper. The societal implications are not discussed however. Some discussion of this would be nice to see for the camera ready - I think there are a lot of applications of this work that could benefit society.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "CYSNW5hEd9j", "reviewer": "Reviewer_r27W", "summary": "They main goal of this article is to investigate if it's possible to build a (specific) generative audio model out of short audio snippet.\nThey use a \"progressive GAN\"-ish model with an additional reconstruction loss, to explore this paradigm. Thei experiment with a fair amount of use cases, demonstrating some degree of success.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows that it is possible to build a (specific) generative audio model out of short audio snippet. However, the artefacts I can hear (in their great webpage with examples!) are the ones I personally associate with the first generation of waveform-based models – that were trained with little data and with small capacity models. Hence, while the model seems to work (up to some extend) it synthesises audio with a fair amount of artefacts. In line with that, one comment:\n- \"For the task of bandwidth extension, only a single 20 second long full-bandwidth training recording typically suffices for obtaining high quality results.\" -> Under my view, those are not high quality speech renditions – please rephrase. That said, the article \"Bandwith extension is all you need\" might be of relevance for the authors.\n\nRegarding the \"structure\" of the new renditions:\n- \"new signals that are semantically similar to the training recording, but contain new compositions and structures\" -> Under my view, the resulting music lacks of \"structure\" – in the same way the speech signals are \"not necessarily interpretable\" becuase the \"model has no notion of language\". For this reason, I think this model does not shine for music or speech signals – but the ambient sounds are super convincing.\n- \"generate new realistic improvisations\" -> Similarly, I don't think the improvisations are realistic because they lack structure and have artefacts. Please, rephrase.\n- I also miss a discussion on why several examples \"repeat\" a variation of the training audio at the beginning of the rendition.\n\nComments related to the perceptual tests:\n- \"are often confused to be real\" -> That's interesting, because your perceptual tests (more-or-less) support this idea, but when I hear to the audio samples myself I think the synthesised audio has artefacts (what makes them clearly distinguishable from the originals).\n- Have you considered running a statistical test to see if there is a statistical difference against (50%, 50%) no preference?\n- \"In fact, our evaluation suggests that limiting the training to a single short signal is actually beneficial, and can outperform models trained on hours of recordings.\" -> I'm not sure how you back this claim.\n\nRelated to the model:\n- \"we take the variance of zn to be proportional to the energy of x in the frequency band\" -> Have you considered injecting less noise, also? The audios sound very noisy. What if you just sample the zn noise at the coarsest level? Or to sample with a dropout layer?\n- I enjoyed the conditional generation and bandwith extension experiments, is a smart way to use the model at hand to build \"personalized deep learning models\".\n- Have you considered pre-training the model with more data? And then finetuning for the specific use case?\n\nMinor comments:\n- \"This is despite its complete lack of prior knowledge about the nature of audio signals in general.\" -> I'm not sure I understand what you mean here.\n- I enjoyed the failure cases! Thanks for sharing – it was very useful.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: Its main limitation is its synthesis quality – the model introduces noticeable artefacts.\n\nSocietal impact: If further developed, this model could allow \"personalised deep learning models\" – what can allow for private models that could be trained on the client side with just few data. That said, the computational requirements might be challinging – since \"training on a 25 second long signal takes about 10 hours on Nvidia GeForce RTX 2080\".", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a way to build a specific generative audio model out of short audio snippet, via a GAN-based approach. While the proposed approach is combining existing techniques in the field (as it's mostly derived from SinGAN), reviewers agree that the adaptation to speech is novel enough. 

A shortcoming of the approach is the artifacts generated by the model - however, a majority of the reviewers found that the quality is good enough.

The authors should include the changes promised in the rebuttal - in particular, it was mentioned during the discussion period that overselling points on the efficacy and quality of the approach should be toned down.

Decision: Accept (Poster)