Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

The authors adapt the recent trend of work on denoising diffusion probablistic models to the task of conditional or unconditional waveform generation.
Using the same principles as in (Ho et al 2020), as well as a Wavenet-like non causal model, the authors provide state of the art results for both tasks, as evaluated on spoken digits dataset (for unconditional and conditional generation) and on the LJ speech dataset (for deep vocoding). 
The model proposed is faster to evaluate than WaveNet, and has less parameters than WaveGlow. It achieves a slightly higher MOS than WaveFlow for a comparable model size. The generation speed is comparable to previous methods.

## Review

The paper is clear, well structured and the authors provide many experiments to validate their approach.
While serious, the paper does lack novelty, as the method is completely taken from (Ho et al. 2020). The architecture is similar to Wavenet, but non causal. Note that this exact same non causal wavenet architecture has already been used for source separation [Rethage et al. 2018, Lluis et al. 2019].

One limitation is that unconditional, or weakly conditioned generation (i.e. not conditioned on a mel-spectrogram) is only evaluated on single digit generation, which is relatively limite. While the samples shows an improvement over WaveNet, it seems the proposed architecture would still struggle to generate longer sequences, like an entire sentence for instance.
It would be interesting to add WaveGlow or WaveFlow to the SC09 comparison.

Overall I recommend acceptance as the paper show that denoising diffusion process can be used for waveform generation, even though the paper does not bring further novelty.

References:
Rethage et al. 2018: A Wavenet for Speech Denoising
Lluis et al. 2019: End-to-end music source separation: is it possible in the waveform domain?
review_title: Review of DiffWave

final_score: rating: 7: Good paper, accept
review_content: The paper develops a speech synthesis model using denoising diffusion processes, a generative model framework recently demonstrated in image generation (Ho et al. 2020).    The application is straightforward and there is little if any theoretical difference from the Ho et al. paper.    I didn't check the proofs included in the appendix, but they along with the learning and sampling procedure seem to be already developed in (Ho et al. 2020).  The authors should take care to be very clear about the mathematical developments that are directly taken from the prior literature, and what developments are introduced in this paper.   
Nevertheless the experiments on this application is valuable, and makes significant progress on a problem that has proven surprisingly difficult to solve in an efficient way.  
The experiments and demos are convincing, and the results could be considered highly competitive in conditional generation and  state of the art for class-conditional and non-conditional generation.  

The writing at times could use improvement.   In the abstract, line 1, "we propose DiffWave, a versatile Diffusion probabilistic model for conditional and unconditional Waveform generation".   I don't like this style of capitalizing things in a sentence that are not proper nouns.  If you want to introduce an abbreviation derived from a term or phrase, a widely accepted conventional method is to italicize the phrase and define the acronym the first time it is used, as in "\emph{diffusion waveform} (DiffWave) model".   Later you have "DiffWave produces high-fidelity audios in Different Waveform generation tasks":  Why is "Different Waveform" capitalized?   DiffWave has already been defined relative to "diffusion waveform".   If this is supposed to be cute, it's not.   
Defining the acronym in the abstract is OK, but not necessary.  In any case, you still have to define it again in the body of the paper, since the abstract is considered a standalone summary of your document. 
Also "audios" is not a word.  Please use "audio signals". This is repeated throughout the paper. 
Other examples:  
"This avoids the ... issues *stemmed from the joint training"   stemmed -->  stemming
" for generating very long waveform" :  waveform --> waveforms



review_title: An interesting paper showing good results on applying denoising diffusion processes to spectrogram inversion. 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper describes a neural vocoder based on a diffusion probabilistic model. The model utilizes a fixed-length markov chain to convert between a latent uncorrelated Gaussian vector and a full-length observation. The conversion from observation to latent is fixed and amounts to adding noise at each step. The conversion from latent to observation reveals slightly more of the observation from the latent at each step via a sort of cancellation. This process is derived theoretically based on maximizing the variational lower bound (ELBO) of the model and follows Ho et al. (2020) who derived it for image generation. Thorough experiments show that the model produces high quality speech syntheses on the LJ dataset (MOS comparable to WaveNet and real speech) when conditionally synthesizing from the true mel spectrogram, while generating much more quickly than WaveNet. Perhaps more interesting and surprising, however, is that it generates very high quality and intelligible short utterances with no conditioning, and also admits to global conditioning, e.g., with a digit label.

The paper is very clearly written, with the description of the model going into sufficient detail in the main body of the paper for the reader to understand it without getting bogged down in the less immediately relevant details. The experiments are thorough and well executed, comparing with listening tests to many state of the art neural vocoders for the conditional task. It describes a thorough evaluation of the unconditional generation task, which is in general difficult, but in this case was constrained in such a way as to make it feasible and informative, using reasonable metrics that clearly show the advantages of the proposed approach. The literature review is thorough and comes at a point in the paper where the reader understands the proposed approach and can appreciate the nuances of the differences between the approaches. Audio samples are provided on a companion website and demonstrate the effectiveness of the approach along with some additional interesting properties of the model not even mentioned in the paper (denoising most impressively, interpolation between speakers is less convincing). 

The paper has two minor weaknesses. First is that it does not make a clearer distinction between the concurrent work from Chen et al (2020) along similar lines, although presumably this paper was not released prior to submission of the current paper. An extended comparison would be welcome in a camera ready version of this paper. Second, that it doesn't explicitly state the real-time factor of WaveNet generation in the results discussion on page 6, which is presumably much smaller than 1. This section compares to WaveNet in terms of quality and WaveFlow in terms of speed, slightly being slightly worse in both comparisons, but better in the opposite, partially missing, comparisons.

Overall, this paper makes a strong contribution to the field of neural vocoding and to the field of representation learning more generally for long-duration intricately structured signals (i.e., speech).
review_title: State of the art neural vocoder that is fast, small, accurate, and works with little supervision.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Well-written paper with strong, well-presented results. The MOS results attain or surpass the best WaveNet results. The presentation is on the whole clear. 

I was a bit confused by the core model description at first. In particular, the index t on the samples x_t is not a time-index, correct? Rather, it's just a step in the diffusion process? At first I thought it was a time-index, and so the model seemed very much like an AR model, leaving me very confused.

Some additional things I liked about the work:

Compelling contrast with existing methods (WaveNet, VAE, GANs).
Use of multiple metrics in the evaluation, 5 objective metrics in addition to MOS, e.g. Tables 2 & 3, with details on the metrics provided in an Appendix.
Use of multiple models as reference models, WaveNet, WaveGlow, WaveFlow, Clarinet, WaveGAN, in addition to the proposed model.
Focus on unconditional generation, which AIU has not received that much attention in the community
Where I am unsure is the originality of the work. I personally am not aware of the diffusion approach having been applied to TTS, but this is not my primary area of expertise. Obviously, if there is related work in TTS with diffusion models, this should be cited.

Also, what I don't see in the Conclusion is any discussion of the weaknesses and challenges for the model going forward. The paper would be strengthened by having a more balanced conclusion.

Some caveats regarding my review:

I am not familiar with the specific datasets used, so cannot fully appreciate the significance of the results reported.
I did not check the math in detail; the notation overall seemed clear and consistent to me (though see my first set of comments).

I have a few more specific comments.

Throughout the paper, "... audios ..." : "audio" is not usually used as a plural noun. 

E.g. "We randomly generate 1,000 audios" --> "We randomly generate 1,000 audio waveforms"?

"Notably, the quality of audios ... " --> "Notably, the quality of audio ... "

"Note that, the quality of ground-truth ...": nit, no comma after "that".
review_title: Well-written paper with strong, well-presented results.

final_score: rating: 7: Good paper, accept
review_content: The Diffusion Probabilistic model is gaining popularity as a generative model. Diffwave explores the same for speech synthesis tasks. They show very good results i.e. matching autoregressive Wavenet on the conditional and outperforming baselines on the unconditional audio waveform synthesis tasks.

The paper is clearly written, and easy to follow. The work does not provide any novel machine learning or generative modeling insights. However, the work is significant for speech synthesis applications since it shows great results, with a small foot-print network with a very new method. This work can be expected to spark a plethora of follow-up works for speech synthesis and other real-valued time-series modeling tasks.

Pros:
1. Very good results on conditional neural vocoding task.
2. State-of-the-art results on unconditional speech synthesis task

Cons:
1. Lack of novelty in terms of insights/approach
2. Motivation and ablation-study for various design choices are missing

*Further Comments*: An ablation study that establishes the impact of various hyper-parameters/components (e.g. choice of diffusion step embedding function, more detailed analysis of width/depth of the network, etc.) would help the readers get a lot more value out of the paper. A qualitative study of samples, specifically, pointing out any bias that underlies generative modeling via the diffusion process would be great.
review_title: Paper showing the applicability of Diffusion Probabilistic model for speech synthesis tasks

Meta-review: I join all five reviewers in recommending acceptance.

There was some discussion about a comparison with WaveGrad (Chen et al., 2020), a contemporaneous work that explores a similar modelling approach for speech generation. While I agree that such a comparison is a useful addition to the manuscript, I do not think it is reasonable to request anything beyond an acknowledgement and citation of the work from the authors as a condition for acceptance. Further discussion and comparison experiments could be valuable, but I believe that should not factor into the final decision. My position is most similar to Reviewer 4's in this sense. The current version of the manuscript briefly discusses the differences between WaveGrad and DiffWave, which I think is more than sufficient. (As an aside, another difference potentially worth discussing is that the "noise schedule" for WaveGrad can be adapted at inference time, enabling a trade-off between inference speed and sample quality, which I believe is not possible for DiffWave in its current form.)

There was some debate about the weakly conditioned generation results; I believe they are a nice addition to the paper, although it would have been suitable for publication without them. They certainly do not detract from it, and might inspire further work in weakly conditioned audio generation (e.g. music). There were also concerns about the clarity of writing, which I believe the authors have addressed in the current version of the manuscript.

This work stands out because it applies a relatively fresh idea in generative modelling to a domain of great practical importance, which has long been dominated by traditional likelihood-based models, with compelling results. While this implies a limited degree of technical novelty, I do not think that is grounds for rejection, and in fact I would argue that making new ideas work well for practical problems is just as important.
Decision: Accept (Oral)