Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies diffusion probabilistic models, and derives the optimal mean and variance (as functions of the expected data score) for the reverse process. Authors then propose to plug in a Monte Carlo estimate of the the variance for the reverse process and experimentally show how this leads to improved results with trained and/or pretrained models in terms of FID and NLL. In addition, authors combine their approach with recent work that optimizes for "knot" locations given a fixed number of knots for faster sampling. 
main_review: **novelty, significance**

The result on optimal variance is new, to the best of my knowledge. The authors also do a reasonable job in convincing me that a Monte Carlo estimate of it is useful for inference. Given the recent progress and interest on DPMs, I think this work will also have reasonable significance and influence. 

**presentation**

Authors do a reasonable job on the writing and presenting experimental results. Past works are adequately and appropriately cited, to the best of my knowledge. 

**Technical quality and correctness**

One thing I'll add here is that once the Monte Carlo estimate of $\sigma_t^2$ is plugged into the bound computation, it seems we end up with a stochastic lower bound of the ELBO (assuming the loss is concave in $\sigma_t^2$). The important thing here to note perhaps is that bias is introduced. To put it more concretely, say the quantity being estimated is $\mathbb{E}[ f(\sigma_t^2) ]$, where I've written the bound as the expectation of the loss $f$ as a function of $\sigma_t^2$. The estimator $f(\hat{\sigma}_t^2)$ is now a stochastic lower bound on the original quantity by Jensen's, since 
$$\mathbb{E} [f(\hat{\sigma}_t^2)] \le \mathbb{E} [f( \mathbb{E} [ \hat{\sigma}_t^2 ] ) ]  = \mathbb{E} [f( \sigma_t^2)].$$

I think there should be some discussion about this. I'm assuming $f$ is concave in $\sigma_t^2$, mostly reasoning from past bounds, but authors should perhaps make parts of the discussion more precise. 

**Experimental results**

Authors do a reasonable job in evaluating their method. One particular point I didn't get is how $M$ (number of samples for estimating the expected score) is chosen. The are a couple of potential issues here.
- Selecting $M$ requires additional hyperparameter tuning, potentially; the tuning procedure should be reported. 
- How results depend on $M$ isn't entirely clear just from reading the main text (maybe there's some discussion in the appendix, but I didn't have time to read all content in the appendix). Ideally, some discussions should appear in the main text.
- Large $M$ incurs more compute cost during inference -- while this seems less an issue when inference is run on GPUs (since most scenarios, I'd guess, there's enough cores to parallelize the Monte Carlo samples), this could be an issue for CPU inference. How does the run-time in practice compare in this case? Note while practical systems don't tend to run training on CPUs, inference on CPUs is still quite common. 
summary_of_the_review: Authors study choosing the optimal variance for the reverse process in DPMs and propose to Monte Carlo estimate it for improved inference.  Technical quality, writing, and experiments are mostly good with the two minor caveats I described above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a theoretically grounded method for estimating *optimal* reverse process variances for DDPMs and DDIMs. This method can be applied to a trained DDPM / DDIM after the fact and lead to improved likelihoods and faster sampling (when used together with optimal trajectory search). The proposed method and theoretical insights also perform strongly in practice across a range of models and datasets.
main_review: **Strengths**
* Strong theoretical motivation and strong empirical results to support it.
* Nicely written, does a great job putting prior work in the context of the new insights on optimal reverse mean and variance.
* Despite the abundance of theory / derivations, the paper still remains accessible.

**Weaknesses**

I am convinced by the paper in its current form. But if I had to list something:
* It would be great to see applications of this method to other data modalities, such as for example speech / sound generation.
* The performance of Analytical-DPMs on methods that learn forward process variance schedules (e.g. VDMs) would also be great to see.
summary_of_the_review: The paper provides valuable insights into optimal reverse process variance of DDPMs and DDIMs, and makes connections between the proposed optimal variance and previous handcrafted choices, etc. The improved understanding of these model classes could have been sufficient to recommend acceptance. However, the empirical results, especially those around faster sampling are also strong and convincing. DDPMs / DDIMs achieve high sample quality and it's primarily their sampling speed that prevents practical application off this model class in real-world systems. This works makes a significant step towards enabling faster sampling for this model class.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed a modification of the diffusion probabilistic models (DPMs) called analytic-DPM that is based on an analytic estimate of the optimal reverse variance. Using this analytic estimate, the proposed method can achieve fast and performant inference through the Monte Carlo method and pre-trained score-based model. The derivation of the optimal reverse analytic mean and variance is proven to be associated with the score function. Upper bounds and lower bounds are provided for the optimal reverse variance, the relationship between the data covariance and the score function is shown. Experimental results are provided by comparing the proposed method with existing variants of DPMs, through both negative log-likelihood and FID as metrics. The experimental results suggest that the proposed method can potentially provide better performance more efficiently compared to alternatives.
main_review: Strength:
1. the paper derives the optimal reverse variance for diffusion probabilistic models as well as its lower and upper bound. This leads to more efficient DPMs with better performance compared to existing DPM variants. It also leads to new insights into DPMs such as why the way the reversed variance is chosen by existing work is not ideal.

2. The paper is clearly presented. The technical results reported in the paper are non-trivially obtained. The relationship between the paper and existing works is well discussed and well-motivated.

3. Experimental results compared to other variants of DPM are well discussed. It also demonstrates the advantage of the proposed method compared to existing DPM variants.

Weakness:
1. Although proposition 1 establishes the relationship between data covariance and score function, it is unclear to me what is the practical implication of proposition 1.

2. In the experiment, it is unclear to me whether timestep is a good metric to measure efficiency in Table 3. Does each method spend roughly the same time at each timestep?

3. While the authors compare the proposed method with other existing variants of DPMs. Is there any reason why the comparison between the proposed method and other classes of generative models such as GAN should be conducted or not?


summary_of_the_review: Correctness: I think the paper is mostly correct. I am not sure if using timestep is a good metric to demonstrate the efficiency of the proposed method. Since efficiency is a major aspect of the proposed method, it would be desirable to make a clarification on this issue.

Novelty and significance: I think the derivation of the optimal reverse variance is novel and insightful. The lower bound and upper bound of the optimal reverse variance is also useful in practice. While the proposed method is performant compared to existing DPM variants, these existing variants achieve better or comparable FID or negative log-likelihood with enough timesteps. These strong baselines suggest the (potentially limited) headroom left for improvement for the proposed method.

Overall, I think the paper solves an interesting problem in DPM.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies an estimate of the reverse process of a diffusion probabilistic model (DPM). The reverse process is usually estimated by minimizing the KL divergence between the forward and the reverse process. Authors present that the optimal mean and variance of the reverse process have analytic forms w.r.t. the score function. The form of the optimal mean coincides with the parameterization in the previous work and justifies a reweighted variant of the variational bound proposed in the recent work. Different from the handcrafted strategies employed in other work, the authors propose a novel estimation for the variance and analyze its bias. To reduce the bias, bounds of the optimal reverse variance are analyzed and the estimate is clipped based on the bound. Furthermore, the KL divergence between the forward and the optimal reserve process also has an analytical form and as a result, the authors propose the optimal trajectory which has minimal KL value. Finally, the authors present the relationship between the score function and the data covariance matrix and assess the proposed approach in the experiments. The experiment results show that the analytic results improve the efficiency, likelihood, and sample quality.
main_review: Strengths:

1. The paper is clearly written and well organized. Contents are easy to follow. There are many technical details for readers to understand the results, such as the derivation of Theorem 1.

2. The analytic results are interesting and novel. According to the introduction and the related work sections, the optimal forms of the reverse process of DPM didn't appear in the previous DPM work and I believe it will help people in the field of DPM better understand this type of models. The bias analysis and the bound of the variance are helpful to understand the estimate and improve the estimate performance. The authors also make detailed discussions on these results, which are very helpful.

3. The experiment results are strong. The authors not only validate their analytic forms but also present the outperformance of their methods. The experiment results show significant improvements in their methods.

Weaknesses:

1. The DPM is a Gaussian-distribution-based simple model and similar analytic results can exist in similar models. The Gaussian model considered in this paper has a simple Markov property, which as a result has decomposable probability. The optimization is via forward KL divergence. Therefore, for example, the classical result on the expectation propagation algorithm with the Gaussian process can simplify the derivation: the decomposable form of the probability guarantees the forward KL decomposable, and minimizing the forward KL is equivalent to moment matching. So, I am concerned that the contribution is not as much as what was introduced in the paper. I also believe that such kind of connection will be helpful to figure out the possible further directions along this line and also save energy to get new results. I suggest authors add some discussions on the connection to other Gaussian models and their results.
summary_of_the_review: The paper contributes interesting analytic results to the DPM models and the methods perform well in practice. However, the DPM is a simple Gaussian model and similar results can appear in other similar models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In Sec. 2, the authors review the general framework of Diffusion Probabilistic Models~(DPMs) with non-Markovian forward processes from [1] which has DDPMs [2] and DDIMs [1] as special cases. They emphasize that the variance of the reverse process, such as in DDPM and DDIM, are generally hand-crafted.

In Sec. 3, the authors present their main theorem: the optimal variance of the reverse process at step $n$ is a linear function of the expected squared norm of the score at step $n$. As a post-processing scheme of learned models, they then propose to replace the commonly used hand-crafted variances by an estimator of their optimal variance: in particular, the score is replaced by the learned score model and the expectation is replaced by a Monte Carlo estimate (with $M$ samples) thereof. In the remainder of this section, the authors derive bounds for the optimal variance and state that in practice their estimator is clipped according to these bounds.

In Sec. 4, the authors repeat the derivation of an optimal variance and bounds thereof when only a subset (of size $K$) of the $N$ timesteps are used for inference as is very common in the literature [1, 3, 4]. Furthermore, they show how their estimator in this setting can be used to still compute the optimal subset according to the commonly used DP algorithm introduced in [4].

In Sec. 6, they apply their post-processing scheme to models trained on CIFAR-10, CelebA 64 and ImageNet 64 and show that it generally leads to improved likelihood and FID scores. They show that their post-procecssing scheme consistently outperforms suboptimal hand-crafted choices (such as in DDPM and DDIM).

References:

[1] Song et al. Denoising Diffusion Implicit Models. ICLR 2021.

[2] Ho et al. Denoising Diffusion Probabilistic Models. NeurIPS 2020.

[3] Dhariwal & Nichol. Diffusion Models Beat GANs on Image Synthesis. arXiv:2105.05233.

[4] Watson et al. Learning to Efficiently Sample from Diffusion Probabilistic Models. arXiv:2106.03802.
main_review: Strengths:
- Paper has single well-executed idea (optimal variance can be computed as a function of score) 
- Improvements over generally fair base lines are achieved by only post-processing; no additional training needed
- Paper is generally well-written and straightforward to read
- I did not check all Lemmas (appendix) in detail, however, it seems that their proofs are generally very rigorous and detailed.
- The bounds in Theorem 2 are quite nice. Given that DPMs are often used for images, the specific bound for the data distributions supported in the $d$-dimensional cube are quite applicable.

Weaknesses:
- For $K<100$, FID scores of the proposed method greatly rely on a trick of clipping the variance of the step $n=2$ (can be seen in appendix G.4). This seems to be a crucial element and should be discussed more in the main paper. In particular, I would like to see how this clipping compares to the bounds (and the clipping) of the optimal score wrt Theorem 2.
- The post-processing method can be quite expensive. For example, on ImageNet the best values are achieved using $MK= 400000$ additional ($M=100, K=N=4000$) function evaluations. For even trajectories (ET), the obvious solution would be to simply use $K \ll N$ (results for this are shown in paper), however, if I am not mistaken, for the optimal trajectory (OT), computed using the DP algorithm from [4], $MN$ functions have to be evaluated for any $K$ (see appendix B and eq (14)). Therefore, in my opinion, the comparison of DDPM and Analytic=DDPM in the OT setting of Table 1 is unfair.

Suggestions:
- It would be helpful to understand how often the estimator is actually clipped. I suggest to compute $R$ (maybe $R=100$) estimators for, say $M=[1,10, 50, 100, 500, 1000]$, and plot he number of the ratio of estimators that was clipped over $M$. This could be done for a few different instantiations of $n$ (I guess there will be more clipping for $n$ being small).
- the estimator $J$ in (14) is biased even when the correct score is known ($J$ is the log of an unbiased (Monte Carlo) estimator); it might be nice to mention this fact.
- It would be nice to see even lower $M$ in the ablations in G.2. Instead of only indicating variance by plotting the estimator in Figure 3, it would be nice to see the estimated variance of the sampler directly.
- Please state the number of Monte Carlo samples used for results in Tables 1,2,3.
- To me it seems that the optimal variance could also be used for training, using for example only $M=1$. I would be curious to see how this performs compared to using the optimal variance only as a guide for post-processing. I greatly encourage the authors to try this (in case there are no major problems I am missing)
summary_of_the_review: Overall, I vote for accepting. The paper provides an important insight in DPMs and shows improved results for pretrained models by a simple post-processing technique. My major concern about the paper is that OT [4] does not work well in combination with their method, which in my opinion makes the work slightly less significant. I hope the authors can address this concern in the rebuttal period.

**Post discussion period update:**
I strongly vote for accepting (and also changed the correctness score from 3 to 4). All of my concerns, questions, and suggestions have been addressed by the authors in the discussion period. I thank the authors for this productive reviewing process.
review_title: null

Meta-review: This paper presents an analytic approach for estimating the optimal reverse variance schedule given a pre-trained score-based model. The experimental results demonstrated the efficacy of the proposed method on several datasets across different sampling budgets. Given the recent interest in score-based generative models, I believe that the paper will find applications in various domains. I am pleased to recommend it for acceptance.
Decision: Accept (Oral)