Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors demonstrate that a SinGAN model can be adapted for audio, training a model on a single instance of audio, and generating continuations and inpainting in both the time and frequency domain. They utilize a multi-scale decomposition to create a pyramid of generators and discriminators, and also force reconstructions for one of the points of the generator's latent space. They make several model accomodations to training on the audio domain, including a criteria for automatically tuning the selected scales of their pyramid to the low-frequency power content of the target audio. They show many applications for their technique, including unconditional temporal continuations, conditional regeneration of higher frequency content to create new song sections, bandwidth extension of low resolution audio, inpainting of audio segments, and implicit denoising through the reconstruction signal. Qualitative and quantitative results support their findings, including listening tests and an extensive page of examples. 
main_review: Originality / Significance


-- Straightforward extension of previous SinGAN work to a new domain, so novelty of the approach is limited.

++ However, the authors do a good job of adapting the technique to the audio domain, and demonstrating that it is possible, which is not a forgone conclusion. Further, the breadth of creative applications they demonstrate are of broad interest to the community and show the further potential of the original SinGAN architecture.

++ In particular, the authors provide several clever adaptations of the model (such as generating new song sections by conditionally sampling higher frequencies, and denoising by using the reconstruction signal) that provide new insight into the domain and are not at all straightforward to expect from this model.


Quality / Clarity

++ The paper is clearly written, laid out in a logical manner, with ample figures and examples to demonstrate the many examples and applications proposed. 

++ Quantitative results are presented, including listening tests for some of the more qualitative examples such as the quality of continuations. 

-- While it is helpful to have the paired / unpaired / tutorial splits in Table 1, the study is lacking a reasonable baseline comparison, such as extending the audio by concatenating (with crossfades) audio chunks from the original audio. A pairwise comparison, or even mean opinion score comparison, would be informative for the quality of the technique rather than the current Turing test of "Is this audio fake?"

-- More quantitative metrics for the audio denoising experiments would help to understand the quality of the denoising. There are many standard denoising benchmarks where noise is added to clean signals and SNR can be computed.


limitations_and_societal_impact: ++ It is great that limitations of the quality of the generation are considered in both the main text and the supplement, and the examples provided are very informative. 

-- That said, there seem to be stronger limitations to the technique in the audio domain that are less perceptually relevant in the image domain. In particular, certain high-level structure such as language (both in english and in terms of musical notes) are not captured by training on a single example. For example the song continuations created garbled pseudo-language mixes of phonemes because they haven't seen words not present in the original clip. Similar for the musical note motifs of continuations, they are more "smoothed concatenations" of the original segments than a true improvisation would consider (there's only so much high-level structure that can be present in a small clip of audio). This is a fundamental limitation of this technique that is less relevant for SinGAN on images as the application of continuing the texture of a mountain skyline of fleet of hot air balloons doesn't require generating novel high-level structure (say a building on the hillside or airplanes flying among the balloons). That said, generating new song segments or improvisations does seem to require new high-level structure (new words, new motifs etc.) and this technique will be limited in these applications unless it is modified to incorporate this outside knowledge from elsewhere. Further highlighting and exploration of these limitations would give a better sense for what this technique does and does not do.

+/- There are not any discussions of Societal Impact, which would be a problem if the model were more convincing at speech continuation, as voice cloning is worth considering as a negative impact. That said, the limitations mentioned above about high-level structure and control limit the possible impact of this particular work.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper presents a strategy for learning to generate convincing audio using only a short audio example as training data. The method involves iteratively upsampling the audio and shaping whitre noise, a procedure which lends itself to variable-length unconditional synthesis as well as several downstream conditional generation tasks such as bandwidth extension and inpainting.
main_review: Overall, this is an excellent paper and the results are quite impressive. I have no doubt that this will be an influential paper within the field of deep audio synthesis. The writing is also very clear and easy to follow, and the method is simple and intuitive. Additionally, the paper appears to have put forth (but not advertised) a solution to a major issue with past work on audio synthesis: the ability to generate audio of arbitrary length with continuity across receptive field chunks.

Ironically, while the paper primarily advertises its strengths at unconditional generation, I am generally more convinced by the _conditional_ generation tasks like music variation generation, bandwidth extension, inpainting, etc. The unconditional generation sounds much more like recombination than generalization. This isn’t necessarily a major criticism as recombination of audio with local continuity is difficult to achieve, as small discontinuities which occur from naive recombination can be perceptually devastating. However, it does sound like the model is just stitching together short windows of audio, not unlike what can be accomplished with traditional granular audio synthesis (in ML parlance, granular synthesis might be called a “retrieval-based” method).

One analysis that would be very useful to see is a similarity matrix between the ground truth and the generated waveform in some audio feature space. I suspect such a figure would make it much easier (compared to listening) to determine the temporal correspondence between the generated result and the original training excerpt, which would either help confirm or discredit my hypothesis that the model is stitching together copied segments of the training data. However, even if it is the case that it is mostly copying, I reiterate that this is still nontrivial, and I would prefer to see the limitation addressed rather than swept under the rug.

Regardless of potential limitations in the unconditional setting, the number of conditional generation tasks that this model excels at is quite impressive. The proposed method appears ready for immediate application, and will be of interest to audio engineers, musicians, and sound artists alike.

limitations_and_societal_impact: The authors explicitly address some limitations of their method in the main text of the paper. The societal implications are not discussed however. Some discussion of this would be nice to see for the camera ready - I think there are a lot of applications of this work that could benefit society.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: They main goal of this article is to investigate if it's possible to build a (specific) generative audio model out of short audio snippet.
They use a "progressive GAN"-ish model with an additional reconstruction loss, to explore this paradigm. Thei experiment with a fair amount of use cases, demonstrating some degree of success.
main_review: The paper shows that it is possible to build a (specific) generative audio model out of short audio snippet. However, the artefacts I can hear (in their great webpage with examples!) are the ones I personally associate with the first generation of waveform-based models – that were trained with little data and with small capacity models. Hence, while the model seems to work (up to some extend) it synthesises audio with a fair amount of artefacts. In line with that, one comment:
- "For the task of bandwidth extension, only a single 20 second long full-bandwidth training recording typically suffices for obtaining high quality results." -> Under my view, those are not high quality speech renditions – please rephrase. That said, the article "Bandwith extension is all you need" might be of relevance for the authors.

Regarding the "structure" of the new renditions:
- "new signals that are semantically similar to the training recording, but contain new compositions and structures" -> Under my view, the resulting music lacks of "structure" – in the same way the speech signals are "not necessarily interpretable" becuase the "model has no notion of language". For this reason, I think this model does not shine for music or speech signals – but the ambient sounds are super convincing.
- "generate new realistic improvisations" -> Similarly, I don't think the improvisations are realistic because they lack structure and have artefacts. Please, rephrase.
- I also miss a discussion on why several examples "repeat" a variation of the training audio at the beginning of the rendition.

Comments related to the perceptual tests:
- "are often confused to be real" -> That's interesting, because your perceptual tests (more-or-less) support this idea, but when I hear to the audio samples myself I think the synthesised audio has artefacts (what makes them clearly distinguishable from the originals).
- Have you considered running a statistical test to see if there is a statistical difference against (50%, 50%) no preference?
- "In fact, our evaluation suggests that limiting the training to a single short signal is actually beneficial, and can outperform models trained on hours of recordings." -> I'm not sure how you back this claim.

Related to the model:
- "we take the variance of zn to be proportional to the energy of x in the frequency band" -> Have you considered injecting less noise, also? The audios sound very noisy. What if you just sample the zn noise at the coarsest level? Or to sample with a dropout layer?
- I enjoyed the conditional generation and bandwith extension experiments, is a smart way to use the model at hand to build "personalized deep learning models".
- Have you considered pre-training the model with more data? And then finetuning for the specific use case?

Minor comments:
- "This is despite its complete lack of prior knowledge about the nature of audio signals in general." -> I'm not sure I understand what you mean here.
- I enjoyed the failure cases! Thanks for sharing – it was very useful.
limitations_and_societal_impact: Limitations: Its main limitation is its synthesis quality – the model introduces noticeable artefacts.

Societal impact: If further developed, this model could allow "personalised deep learning models" – what can allow for private models that could be trained on the client side with just few data. That said, the computational requirements might be challinging – since "training on a 25 second long signal takes about 10 hours on Nvidia GeForce RTX 2080".
review_title: null

Meta-review: The paper introduces a way to build a specific generative audio model out of short audio snippet, via a GAN-based approach. While the proposed approach is combining existing techniques in the field (as it's mostly derived from SinGAN), reviewers agree that the adaptation to speech is novel enough. 

A shortcoming of the approach is the artifacts generated by the model - however, a majority of the reviewers found that the quality is good enough.

The authors should include the changes promised in the rebuttal - in particular, it was mentioned during the discussion period that overselling points on the efficacy and quality of the approach should be toned down.

Decision: Accept (Poster)