Information-Theoretic Diffusion
Authors: Xianghao Kong
Rob Brekelmans
Greg Ver Steeg
Venue: ICLR 2023 poster (2023)
Keywords: diffusion
density models
information theory

Abstract: Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation.  We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory that connect Information with Minimum Mean Square Error regression, the so-called I-MMSE relations. We generalize the I-MMSE relations to \emph{exactly} relate the data distribution to an optimal denoising regression problem, leading to an elegant refinement of existing diffusion bounds.  This new insight leads to several improvements for probability distribution estimation, including a theoretical justification for diffusion model ensembling. Remarkably, our framework shows how continuous and discrete probabilities can be learned with the same regression objective, avoiding domain-specific generative models used in variational methods.

Published as conference paper at ICLR 2023
INFORMATION-THEORRN DIFFUSION
Xianghao Kong
Rob Brekelmans Vector Institute
Greg Ver Steeg
University of California Riverside
University of California Riverside
ABSTRACT
Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided Al art generation. We introduce new mathematical foundation for diffusion models inspired by classic results in information theory that connect Information with Minimum Mean Square Error regression, the so-called I-MMSE relations. We generalize the I-MMSE relations to exactly relate the data distribution to an op- timal denoising regression problem, leading to an elegant refinement of existing diffusion bounds. This new insight leads to several improvements for probability distribution estimation, including a theoretical justification for diffusion model ensembling. Remarkably, our framework shows how continuous and discrete proba- bilities can be learned with the same regression objective, avoiding domain-specific generative models used in variational methods. Code to reproduce experiments is provided at ht httpps: / //gitttt giithub... com faom/xxh.000 ITdi fusion and sim- plified demonstration code is at https: hithhbb...h..g.. com grw.geseeeeeeeeee/eeeeeeeeee.eee InfoDi f wfwusisnsinnno
INTRODUCTION
Denoising diffusion models (Sohl-Dickstein et al., 2015) incorporating recent improvements (Ho al., 2020) now outperform GANs for image generation (Dhariwal Nichol, 2021), and also lead to better density models than previously state-of-the-ard autoregressive models (Kingma et al., 2021). The quality and flexibility of image results have led to major new industrial applications for automatically generating diverse and realistic images from open-ended text prompts (Ramesh et al., 2022; Saharia et al., 2022; Rombach al., 2022). Mathematically, diffusion models can be understood in a variety of ways: classic denoising autoencoders (Vincent, 2011) I) with multiple noise levels and a new architecture (Ho et al., 2020), as VAEs with a fixed noising encoder (Kingma et al., 2021), as annealed score matching models (Song & Ermon, 2019), as non-equilibrium process that tractably bridges between a target distribution and a Gaussian (Sohl-Dickstein et al., 2015), or stochastic differential equation that does the same (Song et al., 2020; Liu et al., 2022).
In this paper, we call attention to a connection between diffusion models and a classic result in information theory relating the mutual information the Minimum Mean Square Error (MMSE) estimator for denoising Gaussian noise channel (Guo et al., 2005). Research on Information and MMSE (often referred as I-MMSE relations) transformed information theory with new representations of standard measures leading to elegant proofs of fundamental results (Verdá & Guo, 2006). This paper uses a generalization of the I-MMSE relation to discover an exact relation between data probability distribution and optimal denoising regression. The informmation-theorttie formulation of diffusion simplifies and improves existing results. Our main contributions are as follows.
A new, exact relation between probability density and the global optimum of a mean square error denoising objective of the form, - log p(æ) = 1/2 f mmse(a: 7)d7 + constant terms, with useful variations in Eq. (8), (9) and (11). Because this expression is exact and not a bound, we can convert any statements about density functionals such as entropy into statements about the unconstrained optimum of regression problems casily solved with neural nets.
The same optimization problem can also be exactly related to discrete probability mass, Eq. (12). In contrast, the variational diffusion bound requires specifying separate discrete decoder term in the generative model and associated variational bound. Our unification of discrete and continuous probabilities justifies empirical work applying diffusion to categorical variables.

---

Published as conference paper at ICLR 2023
In experiments, show that approach can take pre- pre-trained discrete diffusion models and re- inteererree them as continuous density models with competitive log-likelihoods, Our approach allows us fine-tune and ensemble existing diffusion models achieve better NLLs.
FUNDAMENTAL POINTWISE DENOISING RELATION
Let p(z,, xx) be Gaussian noise channel with y V + and (0,1), where represents the Sognal-to-Noose Ratio (SNR) and p(x) is the unknown data distribution. This channel has exploding variance the SNR increases but will that the variance of this channel can be normalized arbitrarily without affecting results. Our convention matches the information theory literature and significandly simplifies proofs.
The seminal result of Guo et al. (2005) connects mutual information with MMSE estimators,
d
(<::z)) 1/2 mmse(7).
(1)
d;
Here, the MMSE refers the Minimum Mean Square Error for recovering in this noisy channel,
mmnse(7) = min Ep(x,,.)) x) [||= - (( =(x,,7)
refer to as the denoising function. The optimal denoising function á' corresponds to the conditional expectation, which can be seen using variational calculus from the fact that the squared error a Bregman divergence (Banerjce et al. (2005) Prop. 1),
i (Zy: 7) = arg min ms = Busp(ul))))) [xc] .(z,,7)
The analytic solution is typically intractable because it requires sampling from the posterior distribu- tion of the noise channel.
Recent work on variational diffusion models writes a lower bound on log likelihood explicitly in terms of MMSE (Kingma et al., 2021), suggesting a potential connection to Eq. (1). However, the precise nature of the connection
clear because the variational diffusion bound is an incquality
while Eq. (2) is an equality, and the variational bound is formulated pointwise for log p(æ) at single æ, while Eq. (2) is an expectation.
We now introduce point-wise generalization of Guo et al's result that is the foundation for all other new results in this paper.
d DKLLp(pzz p(z,)) 1/2 mmso(æ 7)
(4)
dn
The marginal is p(z,) = /pzz||/)p((n and the pointwise MMSE is defined as follows,
mmss((;; (x, 7) = (x), lze) [||= - (zy,
(5)
Pointwise MMSE is just the MMSE evaluated at single point x, and Ep(a)) [mmso(x; 7)1 = mmss(()) Taking the expectation with respect to æ of both sides Eq. (4) recovers Guo et al's famous result, Eq. (1). Our proof of Eq. (4) uses special properties of the Gaussian noise channel and repeated application of integration by parts, with a detailed proof given in Appendix A.1. This result can also be seen a special case of Theorem 5 from Palomar & Verdú (2005) by replacing their general channel with our channel written in terms of 7 and then using the chain rule. In the rest of this paper, we show how this more general version of the I-MMSE relation can be used to reformulate denoising diffusion models.
3 DIFFUSION AS THERMODYNAMIC INTEGRATION
We first use Eq. (4) to derive an expression for log- log-likelihood that resembles the variational bound. However, using results from the information theory literature (Verdú & Guo, 2006), find that the expression can be significantly simplified and certain terms that are typically estimated can be calculated analytically.
2

---

Published as conference paper at ICLR 2023
Our derivation is inspired by the recent development of thermodynamic variational inference (Masrani et al., 2019; Brekelmans et al., 2020) which shows how to construct path connecting a tractable distribution to some target, such that integrating over this path recovers the log likelihood for the target model. The paths between distributions can be generalized in a number of ways (Masrani et al., 2021; Chen et al., 2021). Typically, though, these path integrals are difficult to estimate because they require expensive sampling from complex intermediate distributions. A distinctive property of diffusion models is that the Gaussian noise channel, which transforms the target distribution into a standard normal distribution, can be easily sampled at any intermediate noise level.
We use the fundamental theorem of calculus to evaluate a function two points in terms of the integral of its derivative, ( dr d. ((y) f(m) f(70). This approach is known as "thermody-- namic integration" in the statistical physics literature (Ogata, 1989; Gelman & Meng, 1998), where evaluation of the endpoints corresponds to a difference in the free energy or log partition function, and the derivatives of these quantities may be more amenable to Monte Carlo simulation.
Consider applying the thermodynamic integration trick to the following function, f(x,7) Dxllp(z,, x)) p(zy)]. As before, we have p(z), the data distribution, and Gaussian noise channel, 2y v + e with different iggnalto-noise ratios y
d Lmm dr d7 f(x,7) DKL DKL(p(Z,, (x) p(zz )] D DRz(p(Zz x)) p(2)))l
= = -Dklpp(z,, (z) p( - Ep a)) [log p + log p(x).
In the second line, expanded the KL divergence and used Bayes rule. Next, re-arrange and Eq. (4) to re-write the integrand
log p(x) DKI(p(Zn (z) p(zx))
(T)) 1 log p(
1/2
mmse(æ; r)dy (6)
70
Prior loss
Reconstruction loss
Diffusioo loss
Comparing a particular variational bound for diffusion models in the continuous time limit (Eq. 15 from Kingma el al. (2021)), we see this expression looks similar (see App. A.7 for more details). However, our derivation so far is exact and we haven't 't introduced any variational approximations. Prior and reconstruction loss terms are stochastically estimated in the variational formulation, but we show this unwise and unnecessary. Consider the limit where ?1 - O. In that case, the prior loss will be zero. The reconstruction term, for continuous densities, becomes infinite in the limit of large "ro. For this reason, recent diffusion models only consider reconstruction for discrete random variables, P( ().) Estimation of conflicting divergent terms and the need for separate prior and reconstruction objectives can be avoided, as we now show.
Simple and exact probability density via MMSE
We consider applying thermodynamic
integration to a slightly different function, and expand the range of integration to y E [0, oo). Consider sending samples from either the data distribution p(z) standard Gaussian PG (x) Nx; 0, I)
through our Gaussian noise channel. We denote the MMSE for the channel with Gaussian input as mmseg and write its marginal output distribution as pz,) = p(2,) pe (ax ddx. Finally, we define the function f(x 7) as
f(x, 7) = DKI(p(z, [x) pa(z))] - DKu(p(z
p(zz))-
In the limit of zero SNR, we get lim, f (xx = 0. In the high SNR limit we use the following result proved in App. A.2,
lim f(x, log p(x)
(x)
Combining this with Eq. (4), we can write the log likelihood exactly in terms of the log likelihood of a Gaussian and one dimensional integral.
f0 dy dr
- llg p(x) = log (ogPa(x) -
((0,1)
= log log - 1/2 fo dy (mmsoc((2 mmso(x, 7))
(8)
(z,) logr)(zz)) = (KL(p(z.)) (x) (2,)] is the gap an upper bound Ep(,)) [Dxz [p(z, [=)))lp((( mutual information I [( x,). where the upper bound uses the marginal distribution induced by the Gaussian source pa (z.,) instead of the data p(z.).
Note that For (f [f 7)] = Fepce...) (ppcc.,,,)loop(Ee
3

---

Published as conference paper at ICLR 2023
This expresses density in terms of Gaussian density and correction that measures how much better we can denoise the target distribution than we could using the optimal decoder for Gaussian source data. The density be further simplified by writing out the Gaussian expressions explicitly and simplifying with an identity given in App. A.3.
-logp(x) = d/2 log(2me) 2/2 /- dy
( j 1 1177
mmse(
This expression shows that density be written solely in terms of the global optimum of particular regression problem, the denoising MSE. This is convenient because neural networks excel unconstrained optimization of MSE loss functions. If we take the expectation of llg p(xx) using Eq. (9), we recover a relatively recently discovered representation of the differential entropy. h(p), in terms of MMSE (Verdú & Guo, 2006). This highlights advantage of approach, as all density functionals can be rewritlen in terms of the solution of an unconstrained regression problem.
hp)) Bp(m)) log p(=)] log 2nee 1/2 {-" d7 ( mmse(1))
(10)
(177
Since the first integral in Eq. (9)-(10) does not depend on æ. it is tempting to absorb it into constant. However, the first integral diverges, and the second integral typically divergences as well only the difference converges. This observation will help improve density estimation, by noticing that only the gap between MMSEs is important and that the gap becomes small high and low values of SNR.
d
mmseg(() mmse(y)
mmse(yy
mmsec(y) d/(1+y)
mmsc(y)
2
6
10
12
2
8
10
12
(SNR)
(SNR)
Figure 1: The integral of the gap between MMSE curves for data from the target distribution versus data from a Gaussian distribution is used in Eq. (10) to get an exact expression for the entropy, or expected Negative Log Likelihood (NLL), of the data.
We show example MMSE curves for denoising with Gaussian input non-Gaussian inputs in Fig. 1. The goal is to integrate the gap between these two curves, one analytic (for Gaussians) and one given by the data. Note that the gap could in principle be positive or negative. However, we know that Gaussians have the maximum possible MMSE, compared to any distribution with the same variance or covariance, when used as an input to a Gaussian noise channel (Shamai & Verdu, 1992). Therefore, if we always compare to Gaussians with the same variance or covariance as the data, we can guarantee that the gap is positive. In practice, if the gap ever becomes negative, which we show can occur for models appearing in the literature in Sec. 5, it means that the discovered estimator is sub-optimal and we can fall back to the optimal Gaussian estimator to improve results this region.
We also see that for large and small SNR values, the gap becomes small. At low SNR, the noisy data approximately Gaussian. At high SNR, the noise goes to zero, so a linear (Gaussian) denoiser is sufficient in either case. Recognizing that the important signals come from intermediate SNR values can help save computation over approaches that indiscriminately optimize over the whole curve, as discuss in Sec. 4.
Standardizing data to have unit variance to ensure that the MMSE gap is always positive may be inconvenient,
derive variation of Eq. (9) that takes the scale into account. Instead of using (x) = A(x;; 0, I), we can take the base measure to be a Gaussian, (a = N(x;; μa, Σ), with mean and covariance that match the data. Letting λ1,-.., , Xa be the eigenvalues of the covariance

---

Published as conference paper at ICLR 2023
matrix Σ, we derive a compact expression in App. A.4,
1/2 / dr Σ 7 E mmsc(?)
Ep(a)] log Gaussian entropy
1/2 det (2
(11)
Deviation from Caussianity20 20
This expression tells precisely which part of the reconstruction mean square error curve actually important, namely the part that deviates most from Gaussianity. Note that if the eigenvalues of Σ are not feasible to estimate, we can use a diagonal covariance matrix for the base Gaussian and still preserve the desired property that the gap between the data MMSE and Gaussian MMSE is non-negative (Shamai Verdu, 1992).
Finally, we make several remarks to relate our exact expression for the likelihood in Eq. (9) to existing work in the diffusion literature.
Since the MSE minimization is performed at each 7, reweighting terms with different (Kingma al., 2021) should not have an effect as long as we achieve the global minima each 7.
Several heuristics have been suggested for improving the "noise schedule". which corresponds to points for evaluating the MMSE integral in our formulation (Ho et al., 2020; Kingma et al., 2021; Nichol & Dhariwal, 2021). Our formulation highlights the importance of sampling from SNRs where there is a large gap, and we suggest a simple and effective strategy for this in Sec. 4.
The literature diffusion models also suggests more complex denoising distributions that also model the covariance (Nichol & Dhariwal, 2021). Modeling the covariance of the denoising model is unnecessary in our approach, as our exact expressions never require it.
Compare Eq. (9) to the exact, continuous density expression in Eq. 39 of Song et al. (2020), which combines neural ODE flows and diffusion models. The form of that expression is:
- log -log((x(0)) = log + /." V - g(x(((()), t)dt.
The first step to using this expression is to solve a differential equation for the full trajectory, att), that depends many evaluations of learned denoising diffusion (or score) model. Then, estimating the integral is highly non-trivial as is complex and its divergence is expensive to compute, necessitating some stochastic approximations. See Sec. 6 for additional discussion.
Discrete probability estimator Modeling probability distributions over continuous versus discrete random variables typically requires rather different approaches. An unusual feature of I-MMSE relations is that they naturally handle both types of random variables. In this subsection, consider that æ E C zá, with a domain that discrete but numeric num-niimrii discrete variables can be handled via appropriate mapping function (Guo et al., 2005)). We will use capital P(x:) to denote probability (rather than probability density) over this discrete domain. Note that the Gaussian noise channel, 2, v + is still continuous with associated probability density, p(zy/z)
Re-doing the analysis in Eq. (6) leads to the same expression, but replacing p(xr) with P(æ). The first two terms in Eq. (6) go to zeto,
lim To
D (p(,, (xz) p(Z,,)] + Ep(x le)--log log P(æ) O.
+0
This leads to the following form for the discrete probability.
- log P(x) = 1/2 mnnse(xx r)
(12)
At large SNR, z,, very concentrated around a discrete point specified by æ. In that case, we should be able to recover the true value from the slightly perturbed value with high probability. Sec. 4 derives tail bounds used for approximating this integral numerically. Taking the expectation of this general result recovers equation for the Shannon entropy from Section VI of Guo et al. (2005),
H(x) = Σ P(x) log P(x) E-- P()) log P(x))
1/2 /
mmme (7)7.
(13)
Conveniently, our discrete and continuous probability representations nearly identical they they rely on the same integral and optimization and differ only by constant terms. We discuss comparisons to the variational bounds in App. A.7 and numerical implementation details in Sec. 4.
5

---

Published as conference paper at ICLR 2023
4 IMPLEMENTATION
While we start with exact expressions for density and discrete probability in Eqs. (9) and (12), in practice there will be two sources of error when implementing our approach numerically. First, must parametrize our cstimator as neural network that may not achieve the global minimum mean square error and, second, we are forced to rely on numerical integration.
While the likelihood bounds in Eqs. (6), (8), and (9) are exact, evaluating each mmse(a: x. 7) term requires access to the optimal denoising function or conditional expectation & (zy, 7) By( 2z.) [zr].
MMSE Upper Bounds
Using a suboptimal denoising function á ((z,,7) 7) instead of á" (zy, 7) to
approximate the MMSE, we obtaín an upper bound whose gap can be characterized as
Ex ||
* 2z,,, 1] Bp(x,
||
xt ( 1)
Bp(x,) [|| (2, t y 7)
mins((>) (7)
estimation gap
In App. A.6, we derive this upper bound and its gap using results of Banerjee et al. (2005) for general Bregman divergences. This translates to an upper bound on the Negative Log Likelihood (NLL),
[- dy Σ
(ET1A> 7+1/ - Ep( [|| 2(2y, 1)1
< 1/2 log (2t((2=e))))))
As noted in Fig. 1, only the difference mmsea(1) mmse(?) 0 contributes our expected NLL bound. The non-negativity of this difference is guaranteed by the results of Shamai & Verdu (1992), but may not hold in practice due to our suboptimal estimators of the MMSE, Ep(m,z,)lla
Restricting Range of Integration
(zz, 7) mmse(7).
In regions 7 S Yo 2 71 where the difference in mean square error terms appears to be negative, we can simply define z (2) = &; (z., 7), where &; (z,, 7) is the optimal, linear decoder for Gaussian with the same covariance as the data (App. A.5). For this choice of decoder, the integrand
becomes zero so that we may drop the tails of the integral outside of the appropriate range y C [To, 7)
Ep(z) log p(x)] $ 1/2 log det 1/2 (") dy E 7-/4
(EATIA
Ep( ||| 2(,,,
Parametrization We parametrize a(2,, 7) = (z, - É(Zz, 7))/ definition,
(.. With this arrange to that the error for reconstructing the noise, e, is related to the eтror for reconstructing the original image, E(zy, 7) c v7(x z( i(z, 7)). Our neural network then implements é, a function that predicts the noise. The inputs to this network are (z,. Note that we can equivalently parametrize our network to use the inputs (z,/VI+ +V++3 +1 7, This choice preferable that the variance of the noisy image representation bounded, while the SNR is
unchanged. Numerically, it is better to work with log SNR values, so we change variables, a = log which leads the following form
Ep(=) log p(x)] 1/2 log dete(2eeeeee - 1/2 / da (fz(a) - Ex.. (aleel((zzz E(2,, 7) (14)
Here, fx(a) = Eá= (ax + log Xi), using the traditional sigmoid function, (tt) 1/(1 + e-²).
Numerical Integration The last technical issue to solve is how to evaluate the integral in Eq. (14) numerically. We use importance sampling to write this expression as an expectation over some distribution, q((x), for which we can get unbiased estimates via Monte Carlo sampling. This leads to the final specification of our loss function Ep(=m] log p(x)] < L, where
L = 1/2 log (2 1/2 Ey(a) [1/q(a) (f>((a) En.. [lle - É(2,, (12]]]
(15)
All that remains is to choose (a). We use our analysis the Gaussian case to motivate this choice. Note that fx (=) is a mixture of CDFs for logistic distributions with unit scale and different means. Mixtures of logistics with different means are well approximated by a logistic distribution with larger scale (Crooks, 2009). So we take q(a) to be truncated logistic distribution with mean a Mean, (- log 2,)) and scale / + 3/7² Var, Az), based moment matching to the

---

Published as a conference paper at ICLR 2023
mixture of logistics implied by fx) Samples can be generated from uniform distribution using the quantile function, a = μl + s log (1 [1- t),t ~ L[In, ti]. The quantile range is set so that [M- 4s, jr + 4s]. The objective is estimated with Monte Carlo sampling, providing a stochastic, unbiased estimate of our upper bound in Eq. (14). Numerical integration alternatives include the trapezoid rule (Lartillot & Philippe, 2006; Friel al., 2014; Hug et al., 2016) a Riemann sum, due to the monotonicity of mmse( 7) (Kingma et al. (2021) Fig. 2, Masrani et al. (2019)).
Comparing between continuous and discrete probability estimators Recent diffusion models are trained assuming discrete data. We can measure how well they model continuous data by viewing continuous density estimation as the limiting density of discrete points (Jaynes, 2003). Treating p(æ) as uniform density in some d-dimensional box or bin of volume A around each discrete point leads to the relation, [E|- log x=] = Ei log P(æ= E bin)/(A)) = E - log P(ax E bin) d log A In the other direction, when we use a continuous density estimator to model discrete density, we use uniform dequantization (Ho et al., 2019).
Direct discrete probability estimate likelihood for discrete probabilities.
Finally, we derive practical upper bound negative log
mmsc(y)d7 1/2 f mmse( 1/2
(f,² + ({m)) mmss (y)d
E| log P(=)] 1/2
()) (")
El- log P(x)] s 1/2
E || #(z, 1) dy c("o, m)
(16)
In App. B.1, we analytically derive upper bounds for the left and right tail of the integral, expressed using c(7o: 71). We also get an upper bound from using a denoiser that is not necessarily globally optimal. The discrete and continuous density estimators differ only by constants, therefore we can use the same importance sampling estimator for the integral, as in Eq. 15.
5 EXPERIMENTS
Our results establish an exact connection between the data likelihood and the solution to a regression problem, the Gaussian denoising MMSE. If we integrate the MSE curve of a particular denoiscr and get worse bound, it simply means that the denoiser does not achieve the MMSE. Interestingly, variational diffusion models optimize an objective that can be made quite similar with appropriate choices for variational distributions (App. A.7). Therefore, we can take previously trained diffusion models and evaluate alternate likelihood bounds using the methods described above. We can attempt to improve these bounds by directly optimizing the denoising MSE. Finally, a straightforward implication of our results is that we would like the minimum MSE denoiser at each SNR value; therefore, if we have different denoisers with lower error at different SNR values, we can combine them to get density estimator which outperforms any individual one. Ensembling diffusion models that "specialize" in different SNR ranges can likely be used to construct new SOTA density models.
For the following experiments and consider a pre-trained DDPM model (Ho et al., 2020) and an Improved DDPM we refer to as IDDPM (Nichol & Dhariwal, 2021). We use 4000 diffusion steps for calculating variational bounds. For our IT based methods, the comparable parameter is how many log SNR samples, a ~ q(a). we use for evaluation. For continuous density estimation 100 points were sufficient, while for the discrete estimator we used 1000 points (as discussed in App. B.5). More details on models and training can be found in App. B.3.
the CIFAR-10 dataset, scaled have pixel values in [-1, 1].
Continuous Density Estimation with Diffuusion We first explore the results of continuous probability density estimation on test data using variational bounds versus our approach. Recent diffusion based models treat the data as discrete, bounding log P(xx), however, we could also use diffusion models to give continuous density estimates by interpreting the last denoising step as providing Gaussian distribution over œ. For comparison, we can also take the variational bound for discrete data and create a density estimate by making density uniform within each bin. The results shown in Fig. 2. For our estimator, the shaded integral between the Gaussian MMSE and the denoising MSE is used, and this provides noticeably improved density estimates. Note that for our bound calculated using the IDDPM model, we throw away the part of their network that estimates the covariance and still achieve improved results, validating our assertion that variational modeling of covariance is unnecessary.
7

---

Published as conference paper at ICLR 2023
Table 1: BB| logp(ar) on Test Data (bpd)
-
Variational Bounds
Model
Disc.*
Cont.
Cont.
4/2
MMSE, NIO.
DDPM
-3.62
-3.44
-3.56
MUSE, for Nu, 2)
IDDPM
-4.05
-3.58
-4.09
d4
IDOPM
DDPM-luned 1D0PM tuned
(DPM((une)
-3,55
-3.51
-3.84
IDDPM(tune)
-3.85
-3.55
-4.28
Ensemble
.4.29
(lg SNR)
Figure (Left) MSE curves for different denoisers, used in estimating Negative Log Likeli- hood (NLL). (Right) Continuous NLL estimates for diffusion models using variational bounds and Information- Theoretic
- (IT) bounds (ours). Variance estimates are shown App. B.5. *Uses a
discrete estimator made continuous by assuming uniform density in each bin.
Continuous density estimators applied intrinsically discrete pixel data, like CIFAR-10, can in principle lead to very low NLLs if the model learns to put large mass on discrete points. In our case, we are comparing the same model so the comparison would still be meaningful. Furthermore, we can see that the diffusion models are not putting probability mass on discrete points by comparing the MSE curves to denoisers where discretization is purposely introduced in Fig. 3.
Discrete Probability NLL bounds for diffusion models in recent work are for discrete probability estimates. We compare to our estimate based on an exact relation between MMSE and discrete likelihood. Eqs. (12) and (16), and to continuous NLL estimator treated as discrete estimator with uniform dequantization. The diffusion architectures we tried do not naturally concentrate predictions grid points high SNR, so we had to explicitly add rounding via "soft discretization" function (described in App. B.4), which leads to much lower error at high SNR, as seen in Fig. 3. Further improvements from ensembling the best models different SNRs are described below.
Fine-tuning Fig. 2 and 3 show that we can improve log likelihoods by fine-tuning existing models using our regression objective derived in Eq. 15, rather than the variational bound. In particular, we see that the improvements for continuous density estimation come from reducing eπтоr at high SNR levels. The inability of existing architectures fine-tuning improves the continuous but not the discrete estimator in Table 2. The final discrete estimates are improved by including a soft discretization nonlinearity and using ensembling, as described next. See App. B.3 for additional experimental details.
exploit discreteness the data is the reason
Ensembling Finally, we propose to ensemble different denoising diffusion models by choosing the denoiser with the lowest MSE at each SNR level in Eq. (14), Since estimator depends only on estimating the smallest MSE at each 7, our likelihood estimates will benefit from combining models whose relative performance differs across SNR regions. In Fig. and 3, report improved results from ensembling DDPM (Ho et al., 2020), IDDPM (Nichol & Dhariwal, 2021), finc-tuned versions, and rounded versions (in the discrete case). For discrete estimators, rounding is useful high SNR but counterproductive at low SNR. The ensemble MSE (shaded region of Fig. 3) gives a better NLL bound than any individual estimator.
6 RELATED WORK
The developments in diffusion models that have enabled new industrial applications (Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022) build on rich intellectual history of ideas spanning many fields including score matching(Hyvvrrin) & Dayan, 2005), denoising autocncoders (Vincent, 2011), nonequilibrium thermodynamies and stochastic differential equations (Song et al., 2020; Huang et al., 2021). The observation that part of the variational diffusion loss could be written an integral of MMSEs was made by Kingma et al. (2021). To the best of our know ledgledge, the connection between I-MMSE relations in information theory and diffusion models has so far gone unrecognized.
et al., 2015), neural ODEs (Chen et al., 2018),
8

---

Published as conference paper at ICLR 2023
Table 2: EB| log P(x)) on Test Data (bpd)
DOPM
IDCPM
DOPM tuned
DDCMMruned
Var.
PoundiDDPM)
mundd[DDPM)
Model
Disc.
Disc.
Cont. **
roundiDDPM luned)
4/2
mundj(00PMHumeed)
DDPM
3.37
3.68
3.51
IDDPM
2.94
3.16
3.17
d4
(DPM(tune)
3.45
3.48
3,41
IDDPM(tune)
3.14
3.15
3.18
Ensemble
2.90
3.16
(og SNR)
Figure 3: (Left) MSE curves for different denoisers, used estimating Negative Log Likelihood (NLL), shown with or without rounding using a soft discretization function (App. B B.4). (Right) Discrete NLL estimates for diffusion models using variational bounds and information-taen -hheoretccc bounds (ours). Variance estimates are shown in App. B.5. *Using the denoiser with soft discretization. **Indicates continuous estimator made discrete via uniform dequantization.
Song et al. (2020) introduced different way to use diffusion to model probability densitics via differential equations. This approach requires solving differential equation of a dynamic trajectory for each sample. Our estimate can be computed in a single step by evaluating the score model at each SNR parallel, but the neural probability flow ODE will require thousands of sequential score function evaluations to solve. Concurrent work uses stochastic differential equations along with Girsanov's theorem, stochastic version of the change of variables formula, show that optimizing regression objective is sufficient to exactly match a stochastic process to some target density (Liu al., 2022; Ye al., 2022). The results are used to improved sampling, but they use standard variational bounds for density estimation. Our work, in contrast, exactly relates regression to density estimation but does not address sampling. The form their results for discrete distributions suggests deeper connection between their work and this one.
Our work focused on density modeling, so approaches that forgo density modeling to improve image generation, for instance by doing diffusion in latent space (Rombach et al., 2022), were not considered. Bansal et al. (2022) observe that denoising diffusion models using many types of noise can lead to good image sampling. however, our results highlight the special connection between Gaussian denoising and density modeling.
Recent work has studied applicability of diffusion to discrete and categorical variables (Austin et al., 2021; Gu et al., 2022; Chen et al., 2022). Our exact relation between denoising and discrete probability provides theoretical justification for this promising line of research. Finally, this paper gives theoretical justification for ensembling diffusion models, to achieve the best MSE at different SNR levels. We from concurrent work that this strategy already being successfully employed (Balaji ct al., 2022).
7 CONCLUSION
Variational methods are powerful, but require many choices in designing variational approximations. More complex choices for the variational distributions could lead tighter bounds, and fair amount of work has explored this possibility. Interestingly. though, the most successful refinements of diffusion models have led to objectives that are more similar to the denoising MSE, and the results in this paper finally make it clear that regression is all that is needed for exact probability estimation, for generalized the classic I-MMSE relation from information theory to introduce this simple, exact relationship between the data probability distribution and optimal denoiscrs, - llo p(x) = 1/2 fe mmse( xz, 7) dr + constant terms. This result pares away the unnecessary ingredients in the variational approach, and the simple and evocative form of the result can inspire many improvements and generalizations to be explored in future work.
both continuous and discrete variables.
9

---

Published as conference paper at ICLR 2023
REFERENCES
Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their
Applications, 12(3):313-326, 1982.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusson models in discrete state-spaces. Advances in Neural Information Processing Systems,
2021.
Yogesh Balaji, Seungjun Nah, Xun Huang. Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:221 1.01 324, 2022.
Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, and Joydeep Ghosh. Clustering with bregman divergences. Journal of machine learning research, 6(10), 2005.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208. 09392, 2022.
Rob Brekelmans, Vaden Masrani, Frank Wood, Greg Ver Steeg, and Aram Galstyan. All in the exponential family: Bregman duality in thermodynamic variational inference. In International Conference on Machine Learning (ICML), 2020.
Junya Chen, Danni Lu, Zidi Xiu, Ke Bai, Lawrence Carin, and Chenyang Tao. Variational inference with holder bounds. arXiv preprint arXiv:2111.0294 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 2018.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self conditiniii arXiv preprint arXiv:2208. 042002, 2022.
Gavin E Crooks. Logistic approximation to the logistic-nomee normal integral. Technical Report Lawrence Berkeley National Laboratory. 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis, Advances in Neural Information Processing Systems, 34, 2021.
William Feller. On the thcory of stochastic processes, with particular reference to applications. In Selected Papers I, pp. 769-798. Springer, 2015.
Nial Friel, Merrilee Hurn, and Jason Wyse. Improving power postcrior estimation of statistical evidence. Statisties and Computing, 24(5):709 723, 2014.
Andrew Gelman and Xiao-Li Meng. Simulating normalizing constants: From importance sampling to bridge sampling to path sampling. Statistical science, pp. 163 163,185,11 1998.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for toxt-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Dongning Guo, Shlomo Shamai, and Sergio Verdú. Mutual information and minimum mean-square error in gaussian channels. IEEE transactions on information theory, 1164):1211112882 2005.
Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722-2730. PMLR, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Adyances in Neural Information Processing Systems, 33:6840-6851, 2020.
10

---

Published as conference paper at ICLR 2023
Chin-War Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective diffusion-based generative models and score matching. Advances in Neural Information Processing Systems, 34: 22863-22876, 2021.
Sabine Hug, Michael Schwarzfischer, Jan Hasenauer, Carsten Marr, and Fabian Theis. An adaptive scheduling scheme for calculating bayes factors with thermodynamic integration using simpson's 's rule. Statistics and Computing, 26(3):663 677, 2016.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Edwin Jaynes. Probability theory: the logic of science. Cambridge university press, 2003.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.00630, 2021.
Nicolas Lartillot and Hervé Philippe. Computing bayes factors using thermodynamic integration. Systematic biology, 55(2):195-200 2006.
Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion generative models. arXiv preprint arXiv:2208. 146999, 2022.
Vaden Masrani, Tuan Anh Le, and Frank Wood. The thermodynanic variational objective. In Advances in Neural Information Processing Systems, pp. 11521-11530, 2019,,
Vaden Masrani, Rob Brekelmans, Thang Bui, Frank Nielsen, Aram Galstyan, Greg Ver Steeg, and Frank Wood. q-paths: Generalizing the geometrie annealing path using power means, In Conference on Uncertainty Artificial Intelligence (UAI), 2021.
Radford M Neal. Annealed importance sampling. Statistics and computing, 11(2):125 1399 2001.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Frank Nielsen. What is an information projection. Notices of the AMS, 65(3):321-324, 2018.
Yosihiko Ogata. monte carlo method for high dimensional integration. Numerische Mathematik, 1372)1137-157, 1989.
Daniel Palomar and Sergio Verdú. Gradient of mutual information in linear vector gaussian channels. IEEE Transactions Information Theory, 52(1): 2005.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text- conditional image generation with clip latents. arXiv preprint 220xv:2204.06125. 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Scyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-inng ext-to--maae diffusion models with deep language understanding. arXiv preprint 1wXiv:2205.11487, 2022.
Shlomo Shamai and Sergio Verdu. Worst-case power-constrained noise for binary-input channels. IEEE Transactions Information Theory, 38(5) 1992.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv: / 15030030333333 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2023
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 133466, 2020.
Scrgio Verdú and Dongning Guo. simple proof of the powerr incquality, IEEE Transactions on Information Theory, 52(5):2165-2) 2166, 2006.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compula- tion, 23(7):66611177) 2011.
Liming Wang. Miguel Rodrigues, and Lawrence Carin. Generalized bregman divergence and gradient of mutual information for vector poisson channels. In 2013 IEEE International Symposium Information Theory, pp. 454-458. IEEE, 2013.
Liming Wang, David Edwin Carlson, Miguel RD Rodrigues, Robert Calderbank, and Lawrence Carin. bregman matrix and the gradient of mutual information for vector poisson and gaussian channels. IEEE Transactions Information Theory, 60(5):26112260 2014.
Mao Ye. Lemeng Wu, and Qiang Liu. First hitting diffusion models. arXiv preprint arXiv:2209001170. 2022.
12


Reviews:
[{"review_id": "eclt3NbHD_Y", "reviewer": "Reviewer_pxZh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discusses an unrecognized relation between denoising diffusion probabilistic models and Information with Minimum Mean Square Error estimators (I-MMSE). By showing the equivalence of the KL divergence $KL[p(z_\\gamma|x)\\|p(z_\\gamma)]$ and the pointwise MMSE, this work presents an exact relation between the data probability densities and the optimal denoising functions, which reveals new perspective for improving diffusion models such as recognizing important SNRs. Finally, a practical implementation of the proposed density estimator achieves good test NLLs.", "summary_of_the_review": "Overall, this paper presents a very interesting explanation of diffusion models with I-MMSE. The technical contribution is solid.", "strength_and_weaknesses": "Strength:\n\n- The paper presented a highly intriguing relationship between DPM and I-MMSE, which has the potential to have a significant impact on current research on diffusion-based generative models.\n- The paper is generally well-written, organized, and easy to follow.\n\nQuestions:\n- The KL terms in the original ELBO (as in DDPM) also involve MSE terms. Is there any close relation between the ELBO and the actual implementation of the proposed method (e.g. Eq.(14))? (The discussion in \"Variational bound connection\" seem to be not clearly enough.)\n- In Section 6, it is mentioned that estimating Eq.(10) does not require a sequential estimation of the scores at different SNRs. However, I am uncertain as to whether this is an advantage over Song et al. (2020), as estimating MMSE at each SNR requires estimating an expectation over all Gaussian noise. If only one Monte Carlo sample is used to estimate each MMSE, then it is surely the proposed method is more efficient than Song et al. (2020) due to the parallelism, but this seems to produce a much larger variance. Moreover, regarding the parallelism, the original DDPM ELBO can also be evaluated parallelly if we are able to estimate the expectations (appeared in KL terms) at all SNRs. Can you please explain more about this?\n- Eq.(8) looks a bit like the form of normalizing flows, does this mean that MMSE terms correspond to some kind of log determinant of Jacobian of invertible transformations?\n\nMinors:\n- In Eq.(3) the \"arg\" should be \"argmin\".\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aVFrJLK7Mbf", "reviewer": "Reviewer_Ky8k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new theoretical perspective on diffusion models\nby using an exact information-theoretical expression for the data\nlikelihood in terms of minimum mean squared error estimators.  This\nallows to refine likelihood bounds of existing diffusion models by\nconsidering the MSE of the denoisers at different SNRs.\n", "summary_of_the_review": "This paper presents an interesting and novel theoretical perspective\ninto diffusion models, which form a valuable contribution.\nThis contribution could be stronger if there were clear practical\nimplications to improve diffusion models based on this perspective.\n", "strength_and_weaknesses": "**Strengths**\n- The paper is well written and establishes an interesting novel\n perspective (to the best of my knowledge) for the formalism of\n diffusion models, in terms of information and MMSE estimators.\n- The method allows to refine likelihood bounds for existing models\n  while being significantly more efficient in terms of required steps to compute the likelihood bounds.\n- The paper proposes a principled approach to improving the bounds by\n  ensembling models with the best MSE at different SNRs.\n\n**Weaknesses/Questions**\n- The resulting refined bounds seem to be mixed in terms of sometimes\n  increasing or decreasing the value from the variational bounds. I\n  wonder if the authors can comment further on practical implications\n  that could be derived from this study, e.g. for improving diffusion\n  models?\n- The authors mention categorical variables in the introduction, which\n  is an interesting case in light recent discrete diffusion approaches\n  [Austin et al. 2021, VQ-diffusion  Gu et al 2022]. However it\n  seems to me that due to the Gaussian channel model this method is\n  not amenable to non-ordinal discrete variables such as VQ encodings.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RdY7HvM7oQ", "reviewer": "Reviewer_q1V3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper characterizes the marginal log-likelihood (MLL) as an integral of the minimum mean squared error (MMSE) between the input and the optimal denoiser output under the Gaussian noise channel (integrated over all values of signal-to-noise ratio (SNR)). The main technical tools are a pointwise generalization of the I-MMSE relation and the thermodynamic integration trick. The paper applies the result to estimating data likelihood under diffusion models. Empirical versions of the MLL characterization are developed and shown to yield slightly higher test likelihoods than existing variational methods.\n", "summary_of_the_review": "The paper presents a new exact relation between MLL and MMSE under Gaussian noise channels, and applies it to the likelihood estimation under diffusion models.", "strength_and_weaknesses": "STRENGTHS\n- Useful extension of the classical I-MMSE relation of Guo et al.\n- Exact equality between the MLL and a function of the Gaussian MMSE (for a choice of base Gaussian), which sheds light on the success of MSE in variational objectives for diffusion models. \n- The above equality is derived for both continuous and discrete inputs. The latter admits a very simple equality of -log P(x) = 0.5 int_0^{infty} MMSE(x, snr) d(snr), which implies the result on Shannon entropy from Guo et al. \n- The overall approach of deriving an exact relation rather than a variational objective is refreshing. \n\nWEAKNESSES\n- This is a (mostly) theory paper. There is no (substantial) empirical contribution. \n- The experiments section could be more informative (e.g., the \"denoising MSE objective\" used for finetuning is never explicitly defined).\n- Performance of the empirical bound is not addressed, though I think it's beyond the scope of the paper. It would be interesting to discuss this issue in the context of the general difficulty of entropy estimation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "256QLf7Tu8", "reviewer": "Reviewer_j334", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes using information theory tools to derive the diffusion model's optimization object without using the variational approximation. It turns out the ELBO lower bound actually matches. This paper also discusses the problem of discrete probability estimator. Experiments are done to verify the findings.", "summary_of_the_review": "Please see above.", "strength_and_weaknesses": "Strength:\n1. Using the information-theoretic approach to derive the object of the diffusion model seems interesting. Significantly, the finding of ELBO should be exact is interesting.\n\n2. Discussion of the related theoretical property of the main result is complete.\n\nWeakness:\n\n1. The result that ELBO should be exact is actually not new. Existing works [1, 2] have also had such result based on the Girsanov theorem and some SDE analyses. This slightly decreases the impact of the result. Moreover, a discussion comparing this work and the existing works are necessary: I was wondering which result should be general. \n\n2. The work in [1,2] also discusses the issue of training a discrete probability estimator but seems to using a different approach that avoids rounding and inexact approximation.\n\n3. The sentence under equ (12): In that case, we should be able to return the exact, correct value for \\hat{x} = x, leading to zero error with high probability. Could you elaborate on how this can be true during the sampling phase without the high probability that our model will be close to the discrete state at the end?\n\n4. The experiment section is weak. I believe there should be some more intuitive/practical way to demonstrate the improvement: how about the change of some standard metric such as FID? Does your experiment supports the 4 findings you list in page 5?\n\n[1] Let us Build Bridges: Understanding and Extending Diffusion Generative Models\n\n[2] First Hitting Diffusion Models\n\n===============================================================\nAfter reading AC's comments. \n\nThanks AC for outputting extra information and pointing out that some presentations in this paper confused me on the `tightness of VLB`. Indeed, VLB is tight when we choose the optimal posterior is a standard result and I didn't notice that this paper also requires such optimal posterior. \n\nSince this significantly reduces the soundness and novelty, I downgrade my initial score from 6 -> 5.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster