Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This speech synthesis paper proposes to learn the forward diffusion process and reach better ELBO values by doing that, as well as facilitate inference with fewer steps while keeping good generation quality. The experiments were conducted on LJSpeech and VCTK datasets. The paper compared the proposed method to some strong baselines, including WaveNet, GANs and two recent diffusion methods: WaveGrad and DiffWave. The metrics were STOI, PESQ and human evaluation MOS. The MOS results for the proposed method show no significant difference from the ground truth (same as DiffWave) starting from 7 steps while being and an order of magnitude faster than the best baseline DiffWave.
main_review: Pros:

1. The idea to find the optimal forward process is interesting and novel (modulo concurrect work like VDM https://arxiv.org/abs/2107.00630 which attempts to reach the same goal).
2. The experimental results persuade that the method indeed works well with few inference steps.

##########################################################################

Cons:

1. There are a lot of technical details which make the paper hard to read. It would profit from moving some parts into the supplementary material. In particular, Section 3 first lists 2 pages of propositions and theorems - and only later the reader understands why those were necessary. It might be better to first state what is being done in simple language/terms, and only then explain it mathematically - or even move the math into the supplementary.
2. The comparison methodology to DiffWave is a bit strange: the paper correctly states that confidence intervals for both proposed method and DiffWave cover ground truth under t-test, but why not perform paired test to determine which method is better? I.e. show two generations and ask the rater which is more realistic. Also, there exist paired tests with less strict assumptions than t-test. For example, Wilcoxon signed-rank test doesn't assume normal distribution of scores: https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test.

##########################################################################

Questions during rebuttal period: 

Please address the cons above.

#########################################################################

Minor suggestions and typos: 

(1) Parallel WaveNet is worth mentioning as one of the earlier attempts to apply distillation for speeding up generation.

(2) "Bilateral" in the paper name is slightly confusing: it made me think about https://en.wikipedia.org/wiki/Bilateral_filter, which is unrelated to the proposed method.
summary_of_the_review: I am inclined to accept this paper, because it contains novel ideas and the experiments are solid. Writing would profit from simplification though.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper aims at improving generative denoising diffusion models for audio synthesis. In particular, it proposes a method to learn a noise prediction or scheduling network that optimally chooses the noise scales (this is, the scales of the noise that is injected during the iterative synthesis process) at inference time. This allows for faster synthesis without much reduction in quality. The method, called Bilateral Denoising Diffusion Model (BDDM), relies on a slightly modified training objective compared to previous works, in which case learning the noise schedule corresponds to optimizing the objective towards a tighter bound on the data log-likelihood. The main score model is learned in a similar fashion like previous diffusion models, such that the objectives for the score model and the noise scheduling network decompose into two separate objectives that can be optimized essentially one after another. Experimentally, BDDM achieves audio generation with a quality almost as good as the current state-of-the-art, while requiring much less synthesis steps. It also outperforms other methods for accelerated sampling from diffusion models for audio synthesis.
main_review: **Strengths**:
- The idea proposed by the paper, this is, learning a noise scheduling network based on an adapted objective such that learning this network corresponds to explicitly tightening the lower bound on the data log-likelihood, is new to the best of my knowledge (although related methods exist). Since the noise schedule is learnt specifically for synthesis with fewer steps, the work addresses an important problem of denoising diffusion models, their slow sampling speed.
- The experimental results are promising. The method achieves almost state-of-the-art results, while requiring much fewer synthesis steps, and also seems to be superior to several previous approaches for accelerating sampling in these types of models.

**Weaknesses and Questions/Suggestions**:

Overall, I found the paper difficult to follow and lacking in clarity. I have several questions and concerns:
- While the overall approach seems to be novel, it is closely related to [1]. [1] also learns the noise schedule and additionally also shows how one can perform synthesis with fewer steps. I think this suggests a more thorough, quantitative comparison to [1]. In particular, it would be interesting to learn the noise schedule in a similar manner as [1] and then also reduce the number of steps similarly ([1] works on images, but the method is general). I would be curious which method performs better. The current ablation is a bit inconclusive in that regard, as [1] uses a significantly different noise parametrization, which may not as easily collapse.
- The paper has no thorough Related Work section, which has become a standard part of many papers to appropriately position the work in the literature. I would recommend including such a section in the paper.
- Beginning of section 3; "...which is used to reversely sample $x_n \sim q_\phi(x_n|x_{n+1}, x_0)$ using a scheduling network $\phi$.": Why do we even need an additional scheduling network here? When conditioning on both $x_{n+1}$ and $x_0$, as indicated in the distribution, the ideal denoising distribution, or posterior, is tractable [2].
- In Figure 1, both $q_\phi(x_n|x_{n-1})$ and $q_\phi(x_n|x_{n+1}, x_0)$ have $\phi$ indices and depend on the learnt scheduling network. However, these are different distributions (one defines a forward process, the other the backward posterior). It is unclear, which distribution exactly is learnt here and how the forward and backward processes depend on the learnt distribution. I would suggest to clarify.
- Below Proposition 2; "In this regard, the proposed lower bound $F^{(t)}_{score}(\theta)$ allows us to consider only one $t$ at each training step for efficient training, which is practically more advantageous than the standard ELBO in Eq. (1) that entails computing a sum of T KL terms." -> In DDPM and related methods, we never actually calculate the whole sum either. Rather, $t$ is randomly sampled within a mini batch. Furthermore, my understanding is that BDDM in fact uses a DDPM objective to learn the score network as well. Hence, the advantage is not really clear to me.
- Above Proposition 2, $\pi(x_t)$ is defined as the "prior" at $t$, which, I think, we can interpret as the marginal distribution over all possible $x_t$ at $t$. But now in Proposition 3, we assume $\pi(x_{t-1})=q_\beta(x_{t−1}|x_t, x_0)$, which corresponds to the posterior over $x_{t−1}$, given specific conditionings $x_t$ and $x_0$. This seems very different compared to the $\pi(x_t)$ described above Prop. 2 and might even be contradictory. I think this needs clarification.
- What specifically is $p_{\theta^{*}}(x_{1:t−1}|x_0)$ in Proposition 3? My understanding is that $p_{\theta}$ generally is the learnt score model that defines a distribution of the form $p_\theta(x_{t−1}|x_t)$. How does this distribution then predict $x_{1:t−1}$ given $x_0$, as indicated?
- Theorem 1: In practice, it is highly unlikely that we ever have $\theta=\theta^*$. Hence, I believe the $\theta\neq\theta^*$ case is the only practically relevant one. But in this case, it isn't clear to me why the bound will be tighter than the regular ELBO (because the $\mathcal{L}_{step}$ term isn't there). The proof says it follows from Proposition 2, but it isn't clear. The proposition does not explicitly compare the derived bound with the regular ELBO bound from DDPMs in terms of how tight the bounds are, unless I am missing something.
- Eq. 19 is supposed to define a linear noise schedule, but has $t$ in the denominator. Is there an error?
- [3] also learns a noise schedule. I believe it also deserves to be discussed and cited.

[1] Nichol and Dhariwal, "Improved Denoising Diffusion Probabilistic Models", 2021.

[2] Ho et al., "Denoising Diffusion Probabilistic Models",  2020.

[3] Kingma et al. "Variational Diffusion Models", 2021.
summary_of_the_review: Overall, the paper presents a new idea for learning the inference noise schedule in generative diffusion models and the experimental results are promising. However, closely related methods exist [1] and a more detailed comparison isn't presented (it's not one of the baselines). My main concern is that I found the presentation and writing of the paper lacking in clarity and difficult to follow, as indicated by my questions above. This also makes it difficult to check the derivations and develop a solid intuition for the method. Related to this, the motivation for the specific approach and many steps in the derivations are a bit unclear. In conclusion, I think the paper is not ready for publication in its current form. However, the method does seem promising nonetheless. Hence, I would be willing to raise my score if the weaknesses and questions can be addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to train a scheduling network in addition to the score network for fast sampling of diffusion models. To train such network a new loss is proposed. Results are presented for the task of neural vocoding (single-speaker and multi-speaker).
main_review: Strengths:
- Although I just did a quick pass, I think the derivations in Appendix A are correct. 
- Results in Table 1 seem convincing.
- Results for VCTK are nice. I'd suggest adding them to the main paper.
- I found the ablation of ELBO vs BDDM appropriate.

Weaknesses:
- I had difficulty in understanding the paper. I think it is not accessible for a wide audience.
- I personally find that the paper lacks motivation in every step. High-level interpretations are missing and quite often one has the impression that the corresponding formula "just appears" in there, somehow randomly or out of nowhere. After carefully reading it for hours I still do not understand the main intuition behind the proposed approach: Why this way?
- The paper discusses and proposes generic approaches in a general setting, but then only empirically evaluates them in the task of speech vocoding, which is in a quite particular domain (speech) and, more importantly, corresponds to a strongly-conditioned task (that is, a task where the conditioning carries a lot of information; for instance, it is known that tasks like that require much less sampling steps for diffusion models). It leaves a lot of doubt of whether the proposed approach works with, say, unconditional image generation, or even a less strongly-conditioned task like category-based generation.
- I'm not convinced about the significance of the improvement between the methods presented in Tables 1 and 2 (I'm talking about perceptual significance, not statistical significance here). Furthermore, for some methods and measures (and specially in Table 2), confidence intervals are quite wide, leaving the reader with the doubt of whether the approach is worth this complicated development when compared to, for instance, DDIM.

Other/random:
- I think the role of $\tau$ is not discussed enough.
- I'd have appreciated more discussion on the role of the architecture for the scheduling network.
- Why GS becomes "prohibitively" for $N>6$ and not $N>5$ or $N>10$? Even with a reduced set of synthesis utterances?
summary_of_the_review: I think the proposal of the paper is sound and the developments are correct. However, the choice of the task for which to provide empirical evidence is in a single domain (speech) and very specific (extremely strong conditioning). I personally find the paper hard to follow and lacking motivation/intuition.

[Update: After authors' response and improved manuscript I decided to raise my score from 6 to 8]
review_title: null

Meta-review: This work suggests an extension of diffusion-based generative models, where both the forward and reverse process have learnable parameters (rather than just the reverse process). This is then applied to speech synthesis, with high-fidelity audio generated in very few sampling steps compared to what is typical for this class of models. The proposed model is specifically compared to other diffusion-based approaches for speech synthesis in terms of inference speed.

Reviewers highlighted the novelty of the idea and the convincing experimental results. Concerns were raised about the accessibility and clarity of the presentation (structure, too many technical details), lack of a related work section, and the methodology used to compare the proposed model against baselines. The authors have attempted to address these issues, and two reviewers raised their scores as a result. All reviewers now recommend acceptance.

I am therefore recommending acceptance as well, but I would like to encourage the authors to polish the presentation further, in order to make the work maximally accessible to a wide audience.
Decision: Accept (Poster)