Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper characterizes the marginal log-likelihood (MLL) as an integral of the minimum mean squared error (MMSE) between the input and the optimal denoiser output under the Gaussian noise channel (integrated over all values of signal-to-noise ratio (SNR)). The main technical tools are a pointwise generalization of the I-MMSE relation and the thermodynamic integration trick. The paper applies the result to estimating data likelihood under diffusion models. Empirical versions of the MLL characterization are developed and shown to yield slightly higher test likelihoods than existing variational methods.

strength_and_weaknesses: STRENGTHS
- Useful extension of the classical I-MMSE relation of Guo et al.
- Exact equality between the MLL and a function of the Gaussian MMSE (for a choice of base Gaussian), which sheds light on the success of MSE in variational objectives for diffusion models. 
- The above equality is derived for both continuous and discrete inputs. The latter admits a very simple equality of -log P(x) = 0.5 int_0^{infty} MMSE(x, snr) d(snr), which implies the result on Shannon entropy from Guo et al. 
- The overall approach of deriving an exact relation rather than a variational objective is refreshing. 

WEAKNESSES
- This is a (mostly) theory paper. There is no (substantial) empirical contribution. 
- The experiments section could be more informative (e.g., the "denoising MSE objective" used for finetuning is never explicitly defined).
- Performance of the empirical bound is not addressed, though I think it's beyond the scope of the paper. It would be interesting to discuss this issue in the context of the general difficulty of entropy estimation.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well written. The presented perspective and insights seem novel and substantial.
summary_of_the_review: The paper presents a new exact relation between MLL and MMSE under Gaussian noise channels, and applies it to the likelihood estimation under diffusion models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new theoretical perspective on diffusion models
by using an exact information-theoretical expression for the data
likelihood in terms of minimum mean squared error estimators.  This
allows to refine likelihood bounds of existing diffusion models by
considering the MSE of the denoisers at different SNRs.

strength_and_weaknesses: **Strengths**
- The paper is well written and establishes an interesting novel
 perspective (to the best of my knowledge) for the formalism of
 diffusion models, in terms of information and MMSE estimators.
- The method allows to refine likelihood bounds for existing models
  while being significantly more efficient in terms of required steps to compute the likelihood bounds.
- The paper proposes a principled approach to improving the bounds by
  ensembling models with the best MSE at different SNRs.

**Weaknesses/Questions**
- The resulting refined bounds seem to be mixed in terms of sometimes
  increasing or decreasing the value from the variational bounds. I
  wonder if the authors can comment further on practical implications
  that could be derived from this study, e.g. for improving diffusion
  models?
- The authors mention categorical variables in the introduction, which
  is an interesting case in light recent discrete diffusion approaches
  [Austin et al. 2021, VQ-diffusion  Gu et al 2022]. However it
  seems to me that due to the Gaussian channel model this method is
  not amenable to non-ordinal discrete variables such as VQ encodings.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, original and good quality. The appears to contain sufficient details for reproducibility.
summary_of_the_review: This paper presents an interesting and novel theoretical perspective
into diffusion models, which form a valuable contribution.
This contribution could be stronger if there were clear practical
implications to improve diffusion models based on this perspective.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses an unrecognized relation between denoising diffusion probabilistic models and Information with Minimum Mean Square Error estimators (I-MMSE). By showing the equivalence of the KL divergence $KL[p(z_\gamma|x)\|p(z_\gamma)]$ and the pointwise MMSE, this work presents an exact relation between the data probability densities and the optimal denoising functions, which reveals new perspective for improving diffusion models such as recognizing important SNRs. Finally, a practical implementation of the proposed density estimator achieves good test NLLs.
strength_and_weaknesses: Strength:

- The paper presented a highly intriguing relationship between DPM and I-MMSE, which has the potential to have a significant impact on current research on diffusion-based generative models.
- The paper is generally well-written, organized, and easy to follow.

Questions:
- The KL terms in the original ELBO (as in DDPM) also involve MSE terms. Is there any close relation between the ELBO and the actual implementation of the proposed method (e.g. Eq.(14))? (The discussion in "Variational bound connection" seem to be not clearly enough.)
- In Section 6, it is mentioned that estimating Eq.(10) does not require a sequential estimation of the scores at different SNRs. However, I am uncertain as to whether this is an advantage over Song et al. (2020), as estimating MMSE at each SNR requires estimating an expectation over all Gaussian noise. If only one Monte Carlo sample is used to estimate each MMSE, then it is surely the proposed method is more efficient than Song et al. (2020) due to the parallelism, but this seems to produce a much larger variance. Moreover, regarding the parallelism, the original DDPM ELBO can also be evaluated parallelly if we are able to estimate the expectations (appeared in KL terms) at all SNRs. Can you please explain more about this?
- Eq.(8) looks a bit like the form of normalizing flows, does this mean that MMSE terms correspond to some kind of log determinant of Jacobian of invertible transformations?

Minors:
- In Eq.(3) the "arg" should be "argmin".

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and clear. The discovered connection and the technical contribution in the paper are novel and of good quality.
summary_of_the_review: Overall, this paper presents a very interesting explanation of diffusion models with I-MMSE. The technical contribution is solid.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes using information theory tools to derive the diffusion model's optimization object without using the variational approximation. It turns out the ELBO lower bound actually matches. This paper also discusses the problem of discrete probability estimator. Experiments are done to verify the findings.
strength_and_weaknesses: Strength:
1. Using the information-theoretic approach to derive the object of the diffusion model seems interesting. Significantly, the finding of ELBO should be exact is interesting.

2. Discussion of the related theoretical property of the main result is complete.

Weakness:

1. The result that ELBO should be exact is actually not new. Existing works [1, 2] have also had such result based on the Girsanov theorem and some SDE analyses. This slightly decreases the impact of the result. Moreover, a discussion comparing this work and the existing works are necessary: I was wondering which result should be general. 

2. The work in [1,2] also discusses the issue of training a discrete probability estimator but seems to using a different approach that avoids rounding and inexact approximation.

3. The sentence under equ (12): In that case, we should be able to return the exact, correct value for \hat{x} = x, leading to zero error with high probability. Could you elaborate on how this can be true during the sampling phase without the high probability that our model will be close to the discrete state at the end?

4. The experiment section is weak. I believe there should be some more intuitive/practical way to demonstrate the improvement: how about the change of some standard metric such as FID? Does your experiment supports the 4 findings you list in page 5?

[1] Let us Build Bridges: Understanding and Extending Diffusion Generative Models

[2] First Hitting Diffusion Models

===============================================================
After reading AC's comments. 

Thanks AC for outputting extra information and pointing out that some presentations in this paper confused me on the `tightness of VLB`. Indeed, VLB is tight when we choose the optimal posterior is a standard result and I didn't notice that this paper also requires such optimal posterior. 

Since this significantly reduces the soundness and novelty, I downgrade my initial score from 6 -> 5.
clarity,_quality,_novelty_and_reproducibility: Clarity:
The clarity is clear but I feel the writing should be improved. This paper took me more time to read compared with other submissions due to its organization. I feel it is better to start with some background: the existing result/ preliminary knowledge of some important derivation and the problem.  And then begin to give some technical introduction to the information-theoretic approach. Also highlighting the important findings given by the theory in a short and simple sentence at the beginning of the section might be helpful.

Novelty:
overall, the novelty is sound.

Reproducibility:
No code is given and thus not accessible. But I believe it is reproducible. 

summary_of_the_review: Please see above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Ratings: 8/6/6/5
Confidences: 3/3/4/4

By using information-theoretical tools, the authors derive an exact expression for the data likelihood of diffusion models in terms of minimum mean squared error estimators. This allows for a more accurate estimation of density, and the proposed method also enables the simultaneous modeling of both continuous and discrete probabilities with no additional cost. Empirical results show that the proposed method yields slightly higher test likelihoods than existing variational methods.

The strengths of this paper include its novel perspective on diffusion models. The ensembling approach to improving bounds is also a strength. Weaknesses include the mixed empirical results and the limitations of the Gaussian channel model for non-ordinal discrete variables. There were various questions from reviewers, which were addressed in the rebuttal. My recommendation is to accept.
justification_for_why_not_higher_score: Somewhat weak empirical results.
justification_for_why_not_lower_score: Interesting perspective on diffusion models.
Decision: Accept: poster