Variational Diffusion Models
Authors: Diederik P Kingma
Tim Salimans
Ben Poole
Jonathan Ho
Venue: NeurIPS 2021 Poster (2021)
Keywords: Diffusion Models
Generative Models
Density Estimation

Abstract: Diffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affirmative, and introduce a family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks. Unlike other diffusion-based models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to a remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VLB estimator, leading to faster optimization. Combining these advances with architectural improvements, we obtain state-of-the-art likelihoods on image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to use the model as part of a bits-back compression scheme, and demonstrate lossless compression rates close to the theoretical optimum.

Variational Diffusion Models
Diederik P. Kingma", Tim Salimans', Ben Poole, Jonathan Ho Google Research
Abstract
Diffusion-baseee generative models have demonstrated capacity for perceptually impressive synthesis, but can they also be great likelihood- models? We answer this in the affirmative, and introduce family of diffusion-based generative models that obtain state-of-the-ae likelihoods on standard image density estimation benchmarks. Unlike other diffusion-basee models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-ting time VLB is invariant to the noise schedule, except for the signal-to-nois ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VL.B estimator, leading faster optimization. Combining these advances with architectural improvements, obtain state-of-the-ar likelihoods image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to use the model as part of bits-back compression scheme, and demonstrate lossless compression rates close the theoretical optimum.
1 Introduction
Likelihood-base generative modeling is a central task in machine learning that is the basis for a wide range of applications ranging from speech synthesis [Oord et al., 2016], translation [Sutskever et al., 2014], to compression [MacKay, 2003), to many others. Autoregressive models have long been the dominant model class this task due their tractable likelihood and expressivity, shown in Figure 1. Diffusion models have recently shown impressive results in image [Ho et al., 2020, Song al., 2021b, Nichol and Dhariwal, 2021] and audio generation [Kong al., 2020, Chen al., 2020] in terms perceptual quality, but have yet to match autoregressive models on density estimation benchmarks. In this paper we make several technical contributions that allow diffusion models to challenge the dominance of autoregressive models in this domain. Our main contributions are as follows:
We introduce flexible family of diffusion based generative models that achieve new state- of the-art likelihoosss on standard image density estimation benchmarks (CIFAR-10 and ImageNet). This is enabled by incorporating Fourier features into the diffusion model and using learnable specification of the diffusion process, among other modeling innovations.
We improve our theoretical understanding of density modeling using diffusion models by analyzing their variational lower bound (VLB), deriving a remarkably simple expression in terms of the signal-to-non to-isee ratio of the diffusion process. This result delivers new insight
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

CIFAR-10 without data augmentation
ImageNet State-of-the-art models each of the past years (lower is better)
State-of-thearr: models in each of the past years (lower is better)
3,05
Aurinngraaiiv models
3.1
Autorsatess models
3.60
dimemsionn Qbacs (A0)
Goted
3.0
3.55
Image Transtommer
SPN
2.3
3.50
Spanse Transformer
Sparse Transformer
2.8
Sparse
Routing
VDM (ours)
2.7
VOM
3.40
2.6
3.35
2016-2017
2018
2019
2020
2021
2016-2017
2018
2019
2020
2023
Year
Year
(a) CIFAR-I0 without data augmentation
(b) ImageNet 64x64
Figure 1: Autoregressive generative models were long dominant in standard image density estimation benchmarks. In contrast, we propose family of diffusion basodd generative models, Variational Diffusion Models (VDMs), that outperforms contemporary autoregressive models in these benchmarks. See Table for more results and comparisons.
into the model class: for the continuous-time (infinite-depth) setting we prove novel invariance of the generative model and its VLB to the specification of the diffusion process, and show that various diffusion models from the literature are equivalent up trivial time-dependent rescaling of the data.
2
Related work
Our work builds on diffusion probabilistic models (DPMs) "Sohl-Dickstein et al., 2015], or diffusion models in short. DPMs can be viewed type of variational autoencoder (VAE) [Kingma and Welling, 2013, Rezende et al., 2014], whose structure and loss function allows for efficient training of arbitrarily deep models. Interest in diffusion models has recently reignited due to their impressive image generation results [Ho et al., 2020, Song and Ermon, 2020].
Ho et al. [2020] introduced a number of model innovations to the original DPM, with impressive results on image generation quality benchmarks. They showed that the VLB objective, for diffusion model with discrete time and diffusion variances shared across input dimensions, is equivalent multi- scale denoising score matching, up to particular weightings per noise scale. Further improvements proposed by Nichol and Dhariwal [2021], resulting in better likelikelihood scores. Gao et al. [2020] show how diffusion can also be used to efficiently optimize meergy-based models (EBMs) towards close approximation of the eggliielloood objective, resulting in high- tigh-fideityy samples even after long MCMC chains.
Song and Ermon [2019] first proposed learning generative models through multi-scale denoising score matching objective, with improved methods Song and Ermon [2020]. This was later extended to continuous-time diffusion with novel sampling algorithms based on reversing the diffusion process [Song et al., 2021b].
Concurrent to our work. Song et al. [2021a], Huang et al. [2021]. and Vahdat et al. [2021] also derived variational lower bounds to the data likelihood under continuous-tima diffusion model. Where we consider the infinitely deep limit of standard VAE, Song et al. [2021a] and Vahdat al. [2021] present different derivations based on stochastic differential equations. Huang al. [2021] considers both perspectives and discusses the similarities between the two approaches. An advantage of our analysis compared these other works that we present an intuitive expression of the VLB in terms of the signal-to-noise ratio of the diffused data, leading to much simplified expressions of the discrete-time and continuous time loss, allowing for simple and numerically stable implementation. This also leads to results the invariance of the generative model and its VLB

---

to the specification of the diffusion process. We empirically compare to these works, as well as others, in Table I.
Previous approaches to diffusion probabilistic models fixed the diffusion process; in contrast optimize the diffusion process parameters jointly with the rest of the model. This turns the model into type of VAE [Kingma and Welling, 2013, Rezende et al.. 2014]. This is enabled by directly parameterizing the mean and variance of the marginal g(z, (zo). where previous approaches instead parameterized the individual diffusion steps q ((zt+elZz)) In addition, our denoising models include several architecture changes, the most important of which is the use of Fourier features, which enable us to reach much better likelihoods than previous diffusion probabilistic models.
3 Model
We will focus on the most basic case of generative modeling, where we have dataset of observations of x, and the task is to estimate the marginal distribution p(x). As with most generative models, the described methods can be extended to the case of multiple observed variables, and/or the task of estimating conditional densities p(xly). The proposed varial-v model consists a diffusion process (Section 3.1) that we invert to obtain hierarchical generative model (Section 3.3). As we will show, the model choices below result in surprisingly simple variational lower bound (VLB) of the marginal likelihood, which use for optimization of the parameters.
3.1 Forward time diffusion process
Our starting point is Gaussian diffusion process that begins with the data x, and defines a sequence of increasingly noisy versions of x which we call the latent variables z,, where t runs from t = 0 (least noisy) to = (most noisy). The distribution of latent variable z, conditioned on x, for any I E [0, 1] is given by:
(2,1x) = N (uxx,oo1)
(1)
where Og and of are strictly positive scalar-valued functions of t. Furthermore, let us define the signal-to-noise ratio (SNR):
SNR(t) = /7
We assume that the SNR(t) is strictly monotonically decreasing in time, i.e. that SNR(t) SNR(s) for any t t>> s. This formalizes the notion that the z, is increasingly noisy as we go forward in time. We also assume that both a and of are smooth, such that their derivatives with respect to time t are finite. This diffusion process specification includes the variance-preservee by [Sohl-Dickstein et al., 2015, Ho et al., 2020] as a special case, where a₂ = /I o7. Another special case is the variance-exploding diffusion process used by [Song and Ermon, 2019, Song et al., 2021b], where at = 1. In experiments, we use the variance-preserving version.
diffusion process as used
The distributions qzz (z,) for any t > are also Gaussian, and given in Appendix A. The joint distribution of latent variables (zg, Zz, z.) at any subsequent timesteps S s < t < u S 1 is Markov: verify through Bayes rule that (2,, 24, x), for any 0 S a < < < 1, is also Gaussian. This distribution also given in Appendix A.
(Zn, 22, 2,) q(z..) (z:)- Given the distributions above, it is relatively straightforward
3.2 Noise schedule
In previous work, the noise schedule has a fixed form (see Appendix H, Fig. 4a). In contrast, we learn this schedule through the parameterization
of = sigmoid(?,,
where Yn (t) is monotonic neural network with parameters n7, detailed in Appendix H.
Motivated by the equivalence discussed Section 5.1, we use a, = /I V1-o" of our experiments for both the discrete-time and continuous-tim models, i.c. parseric-pesservigg diffusion processes, It is straightforward verify that of and SNR(t), as a function of (t),,, then simplify to:
a = SNR(t) exp(-7,(t))
(4)
(5)
3

---

3.3 Reverse time generative model
We define our generative model by inverting the diffusion process of Section 3.1, yielding a hierarchi- cal generative model that samples sequence of latents 2,, with time running backward from / to t t= O. We consider both the case where this sequence consists of a finite number of steps T', as well as a continuous time model corresponding T + oo. We start by presenting the discrete-time case.
Given finite T, we discretize time uniformly into timesteps (segments) of width r = 1/T. Defining s(i) (i 1) and t(i) i/T, our hierarchical generative model for data is then given by:
p(x) -f (pzz))((x1z
II
With the variance preserving diffusion specification and sufficiently small SNR(1), we have that [<z1/x) = (Zz 0,I). We therefore model the marginal distribution of Z1 as spherical Gaussian:
p(z1) (2z;; 0, 0)
We wish to choose model p(x|zo) that is close the unknown (x|22)) Let 24 and 20,i be the i-th elements of x, Zo, respectively. We then use factorized distribution of the form:
p(x|zo) II p(2.)20,s)
(8)
where we choose p(z;) 20,s) x ((20,1244) which normalized by summing over all possible discrete
values of z; (256 in the case of 8-bit image data). With sufficiently large SNR(O). this becomes very close approximation to the true q(x(zo). the influence of the unknown data distribution q(x)
is overwhelmed by the likelihood q(Zul x). Finally, we choose the conditional model distributions as
p(z, (2:) g(2, /2,, x (o(zi; ()),
Lee the same q(z, (2,,xx x), but with the original data replaced by the output of denoising model &a (z:; 1) that predicts x from its noisy version 2₁ Note that in practice we parameterize the denoising model function of a noise prediction model (Section 3.4), bridging the gap with previous work on diffusion models [Ho et al., 2020]. The means and variances of p(z. (2)) simplify to a remarkable degree; see Appendix A.
3.4 Noise prediction model and Fourier features
We parameterize the denoising model in terms of noise prediction model éø(Z;; t):
X (z(z))) t) = (ze ozčo(Z;; t))/oa:
(10)
where éo(z;; t) is parameterized as a neural network. The noise prediction models we use in experiments closely follow Ho et al. [2020], except that they process the data solely at the original resolution. The exact parameterization of the noise prediction model and noise schedule is discussed in Appendix B.
Prior work on diffusion models has mainly focused the perceptual quality of generated samples, which emphasizes coarse scale patterns and global consistency of generated images. Here, we optimize for likelihood, which sensitive to fine scale details and exact values of individual pixels. To capture the fine scale details of the data, we propose adding a set of Fourier features to the input of our noise prediction model. Let x be the original data, scaled to the range [-1, 1]. and let z be the resulting latent variable, with similar magnitudes. We then append channels sin TZ) and cos(2"xz), where runs over range of integers {Nmin: .... nmaz). }. These features are high frequency periodic functions that amplify small changes in the input data Z₁; see Appendix C for further details. Including these features in the input of our denoising model leads to large improvements in likelihood as demonstrated in Section 6 and Figure 5, especially when combined with learnable SNR function. We did not observe such improvements when incorporating Fourier features into autoregressive models.

---

3.5
Variational lower bound
We optimize the parameters towards the variational lower bound (VLB) of the marginal likelihood, which is given by
- (o) - Vx) = Dx -DKu(((z())ll((e))
l- + (r(
(11)
Prior loss
Reconstruction
Diffusion loss
The prior loss and reconstruction loss be (stochastically and differentiably) estimated using standard techniques; see [Kingma and Welling, 2013]. The diffusion loss, (x(x), is more complicated, and depends on the hyperparameter T that determines the depth of the generative model.
4 Discrete-time model
In the case of finite T, using s(i) = ()-1))/, i/T. the diffusion loss is:
T
Cx( = Σ
Eyleaco (x) Dx (9( Ze()) ())))(2(()) k())
(12)
i=1
In appendix we show that this expression simplifies considerably, yielding:
Lr(x) T
Be..iio.) [sNN(ss) SNN (() ||x (z( 1013
(13)
where U} is the uniform distribution the integers {1...., T). and o,x Øgc. This is the general discrete- time loss for any choice of forward diffusion parameters (0x, oz). When plugging in the specifications of or a₂ and &g (z,; that use in experiments, given in Sections 3.2 and 3.4, the loss simplifies to:
Cx(x) T
EenN -ULL)) (exp((T)))) (()()) 1) ||e Eo (zr; 1)
(14)
where 2, sigmoid(- (m7 (1) x + (t)) In the discrete-time case, we simply jointly optimize and 0 by maximizing the VLB through Monte Carlo estimator of Equation 14.
Note that exp(.) has numerically stable primitive expm1 (.) in common numerical computing packages; see figure 6. Equation 14 allows for numerically stable implementation in 32-bit or lower-precision floating point, in contrast with previous implementations of discrete-time diffusion models (e.g. [Ho et al., 2020]), which had to resort to 64-bit floating point.
4.1 More steps leads lower loss
A natural question to ask is what the number of timesteps T should be, and whether more timesteps is always better in terms of the VLB. Appendix we analyze the difference between the diffusion loss with T' timesteps, (x)), and the diffusion loss with double the timesteps, (2r(x), while keeping SNR function fixed. We then find that if our trained denoising model &e is sufficiently good, have that (x) Cr(x), i.e. that our VLB will be better for larger number of timesteps. Intuitively, the discrete time diffusion loss is an upper Riemann sum approximation of an integral of strictly decreasing function, meaning that finer approximation yields a lower diffusion loss. This result illustrated in Figure 2.
Continuous-time model: T - o
Since taking more time steps leads to a better VLB, we now take T - oc, effectively treating time f as continuous rather than discrete. The model for p(2:) can in this case be described as continuous time diffusion process [Song et al., 2021b] governed by stochastic differential equation; see Appendix D. In Appendix E we show that this limit the diffusion loss Cr(x) simplifies further. Letting SNR (t) dSSR((t/dd) dt, we have, with z, 0& + OrE:
1 E. BeoNion L SNR'(t) ||x (z t)|1 dt,
Loo (x)
(15)
2
2 Be..Nio.). (SNR'()) ||x xo
(16)
5

---

MSE
MS0
SNR
SNR
SNR
Figure 2: Illustration of the diffusion loss with few segments T (left), more segments (middle), and infinite segments ' (continuous time, right). The continuous-time loss (Equation 18) is an integral of mean squared error (MSE) over SNR, here visualized as a black curve. The black curve strictly decreasing when the model is sufficiently well trained, so the discrete-time loss (Equation 13) is an upper bound (an upper Riemann sum approximation) this integral that becomes better when segments are added.
This is the general continuous-timm loss for any choice of forward diffusion parameters (ox, at). When plugging in the specifications of ot a and (o(z;;t) that we use experiments, given in Sections 3.2 and 3.4, the loss simplifies to:
Loo = ( (t)) ||c 1)11
(17)
2
where (t) (t)/t). We use the Monte Carlo estimator of this loss for evaluation and optimization.
5.1 Equivalence of diffusion models continuous time
The signal-to-noise function SNR(() is invertible due to the monotonicity assumption in Section 3.1. Due to this invertibility, we can perform change of variables, and make everything a function of 0 = (t) instead of 1, such that = SNR -0 (v). Let a and og be the functions a and oe evaluated at SNR-1 (w)), and correspondingly let Z = Q + agc. Similarly, we rewrite our noise prediction model as Xe (z, v) X (z, SNR (w)). With this change of variables, our continuous-tiin -time loss in Equation 15 can equivalently be written as:
SNR
(x) = -EenNioo
||x- Xo( v)(l de,
(18)
SNR
where instead of integrating w.r.t. time we now integrate w.r.t. the signal-to-noise ratio v, and where SNR S1R( and SNR (NR
What this equation shows us is that the only effect the functions a<tt) and (t) have on the diffusion loss is through the values SNR(t) af oof at endpoints f 0 and f 1. Given these values SNR and SNR the diffusion loss is invariant to the shape of function SNR(t) between = 0 and = 1. The VLB is thus only impacted by the function SNR(t) through its endpoints SNR and SNR
Furthermore, we find that the distribution p(x) defined by our generative model also invariant to the specification of the diffusion process. Specifically, let pa denote the distribution defined by the combination let n (x) be the distribution defined through different specification {a a x8). where both specifications have equal SNR SNR as shown in Appendix G, then have that *x() p" (x) x (z, v) X6 ((a," /a,"" )z, v). The distribution on all latents Z is then also the same under both specifications, up to trivial rescaling. This means that any two diffusion models satisfying the mild constraints set in 3.1 (which includes e.g. the variance exploding and variance preserving specifications considered by Song et [2021b]), can thus be seen as equivalent continuous time.
diffusion specification and denoising function {agt y x }. and similarly
5.2 Weighted diffusion loss
This equivalence between diffusion specifications continues to hold even if, instead of the VLB, these models optimize weighted diffusion loss of the form:
SNR
Loo (x, w) = BE-N((O.)
w(v) ||x (0 u2),
(19)
SNR

---

Model
Type
CIFARIO no data aug.
CIFARIO data aug.
ImageNet 32x32
ImageNet 64x64
(Bits per dim on test set)
Previous work
ResNet VAE with IAF [Kingma et al., 2016]
VAE VAE
3.11
Very Deep VAE [Child, 2020]
2.87
3.80
3.52
NVAE [Vahdat and Kautz, 2020]
VAE
2.91
3.92
Glow [Kingma and Dhariwal, 2018]
Flow
3.35
4.09
3,81
Flow++ [Ho et al., 2019a]
Flow
3.08
3,86
3,69
PixeCCNN [Van Oord al., 2016]
AR
3.03
3.83
3.57
PixeICNN+ [Salimans al., 2017]
AR
2.92
Image Transformer [Parmar et al., 2018]
AR
2.90
3.77
SPN [Menick and Kalchbrenner, 2018)
AR
3.52
Sparse Transformer [Child et al.. 2019]
AR
2.80
3,44
Routing Transformer [Roy al., 2021]
AR
3.43
Sparse Transformer+ + DistAug [Jun et al., 2020]
AR
2.53(4)
DDPM [Ho et al., 2020]
Diff
(.)
EBM-DRL [Gao et al., 2020]
Diff
3.18(C)
Score SDE [Song et al., 2021b]
Diff
2.99
Improved DDPM [Nichol Dhariwal, 2021]
Diff
2.94
3.54
Concurrent work
CR-NVAE [Sinha and Dieng, 2021]
VAE
2.51
LSGM [Vahdat al., 2021]
Diff
2.87
ScoreFlow [Song et al., 2021a] (variational bound)
Diff
(.90
3,86
ScoreFlow [Song ct al., 2021a] (cont. norm. flow)
Diff
2.83
(.80(C)
3.76
Our work
VDM (variational bound)
Diff
2.65
(.49
3.72
3,40
Table 1: Summary of our findings for density modeling tasks, in terms of bits per dimension (BPD) on the test set. Model types are autoregressive (AR), normalizing flows (Flow), variational autoencoders (VAE), diffusion models (Diff). Our results were obtained using the continuous-time formulation of our model. CIFAR- 10 data augmentations are: (A) extensive, (B) small translations, or (C) horizontal flips. The numbers for VDM are variational bounds, and likely be improved by estimating the marginal likelihood through importance sampling, or through evaluation of the corresponding continuous normalizing flow done by Song al. [2021a].
which e.g. captures all the different objectives discussed by Song et al. [2021b], see Appendix K. Here, w(v)) is weighting function that generally puts increased emphasis on the noisier data compared to the VLB, and which thereby certain metries like FID and Inception Score. For the models presented in this paper, we further use w(v) = 1 as corresponding to the (unweighted) VLB.
sometimes improve perceptual generation quality measured by
5.3 Variance minimization
Lowering the variance of the Monte Carlo estimator the continuous time loss generally improves the efficiency of optimization. We found that using low-discrepancy sampler for f, explained in Appendix 1.1, leads to a significant reduction variance. addition, due to the invariance shown in Section 5.1 for the continous-time case, we optimize the schedule between its endpoints w.r.t. to minimize the variance of our estimator of loss, as detailed in Appendix The endpoints of the noise schedule simply optimized w.r.t. the VLB.
Experiments
We demonstrate our proposed class of diffusion models, which we call Variational Diffusion Models (VDMs), on CIFAR-10 [Krizhevsky et al., 2009] dataset, and the downsampled Im- ageNet [Van Oord et al., 2016, Deng et al., 2009] dataset, where we focus on maximizing likelihood. For our result with data augmentation we used random flips, 90-degree rotations, and color channel swapping. More details of our model specifications are in Appendix B.

---

the KIN
Figure 3: Non cherry-picked unconditional samples from our Imagenet 64x64 model, trained in continuous time and generated using T 1000. The model's hyper-parameters and parameters optimized w.r.t. the likelihood bound, so the model is not optimized for synthesis quality.
6.1 Likelihood and samples
Table shows our results on modeling the CIFAR-10 10 dataset, and the downsampled ImageNet dataset. We establish new offe--of-the-rr terms of test set likelihood on all considered benchmarks, by sigmificant margin. Our model for CIFAR-10 10 without data augmentation surpasses the previous best result of 2.80 about 10x faster than takes the Sparse Transformer to reach this, in wall clock time on equivalent hardware. Our CIFAR-10 10 model, whose hyper-parameters were tuned for likelihood, results in FID (perceptual quality) score of 7.41. This would have been state-of-the-art until recently, but is worse than recent diffusion models that specifically target FID scores [Nichol and Dhariwal, 2021, Song et al., 2021b, Ho al., 2020]. By instead using a weighted diffusion loss, with the weighting function w (SNR) used by Ho et al. [2020] and described in Appendix K, our FID score improves 4.0. We did not pursue further tuning of the model improve FID instead of likelihood. A random sample of generated images from our model is provided in Figure 3. We provide additional samples from this model, as well as our other models for the other datasets, Appendix M.
6.2 Ablations
Next, we investigate the relative importance of our contributions. In Table we compare our discrete- time and continuous-time specifications of the diffusion model: When evaluating our model with small number of steps, our discretely trained models perform better by learning the diffusion schedule to optimize the VL.B. However, as argued theoretically in Section 4.1, we find experimentally that more steps indeed gives better likelihood. When grows large, our continuously trained model performs best, helped by training its diffusion schedule to minimize variance instead.
Minimizing the variance also helps the continuous time model train faster, shown in Figure This effect is further examined in Table 4b, where we find dramatic variance reductions compared to our baselines in continuous time. Figure 4a shows how this effect is achieved: Compared the other schedules, our learned schedule spends much more time in the high SNR(t) / low of range.
further show training curves for model including and excluding the Fourier features proposed in Appendix C: with Fourier features enabled our model achieves much better likelihood. For comparison et al., 2017], where we do not see benefit. In addition, we find that learning the SNR is necessary to get the most of including Fourier features: if fix the SNR schedule to that used by Ho et al. [2020], the maximum log-SNR is fixed to approximately 8 (see figure 7), and test set negative likelihood stays above bits per dim. When learning the SNR endpoints, our maximum log-SNV -NR-SNR ends up at 13.3, which, combined with the inclusion of Fourier features, leads to the SOTA test set likelihoods reported in Table
In Figure 5
also implemented Fourier features in PixeICNN+ model [Salimans
6.3 Lossless compression
For a fixed number of evaluation timesteps Tevat, our diffusion model discrete time a hierarchical latent variable model that can be turned into a lossless compression algorithm using bits-back coding [Hinton and Van Camp, 1993]. As a proof of concept of practical lossless compression, Table 2 reports net codelengths on the CIFAR10 test set for various settings of Tevel using BB-

---

10
SNR (f) schedule
Var(BPD)
Learned (ours)
0.53
ogg
log SNR-linear
alpha cosine [2]
8-Liincar [1]
31.6
a-Cosine [2]
31.1
0.0
0.2
0.4
0.6
(a) log SNR time
(b) Variance of VLB estimate
Figure 4: Our learned continuous-time variance- variance-minimizzin minimizing noise schedule SNR(t) for CIFAR- 10, compared to its log-linear initialization and to schedules from the literature: [1] The 3-Linear schedule from Ho al. [2020]. [2] The a-Cosine schedule from Nichol and Dhariwal [2021]. All schedules scaled and shifted the log scale such that the resulting SNRmin SNR were the equal to our learned endpoints, resulting in the same VLB estimate of 2.66. We report the variance of our VLB estimate per data point, computed on the test set, and conditional on the data: This does not include the noise due to sampling minibatches of data.
Bits-Back Net BPD
Tiraén
Tcea!
VDM (ours)
10
10
4,31
Founeer
100
100
2.84
VOM, No Variance Minimization
250
250
2.73
PbaiCCN- -4 Foutler features
500
500
2.68
1000
1000
2.67
10000
10000
2.66
10
7.54
7.54
2.91
2.0
250
2.74
2.76
Bis
00
500
2.69
2,72
2.67
2.72
2.6-
10000
2,65
100000 200000
300000
400000
500000
600000
2.68
Table 2:
Discrete versus continuous-time train- ing and evaluation with CIFAR-10, in terms of bits per dimension (BPD).
Figure 5: Test set likelihoods during training, with/without Fourier features, and with/without learning the noise schedule to minimize variance.
ANS [Townsend al., 2018), implementation of bits-back coding based asymmetric numeral systems [Duda, 2009]. Details of our implementation are given in Appendix N. We achieve statc-of- the-art net codelengths, proving our model can used the basis of lossless compression algorithm. However, for large Teval a gap remains with the theoretically optimal codelength corresponding to the negative VLB, and compression becomes computationally expensive due to the large number of neural network forward passes required. Closing this gap with more efficient implementations of bits-back coding suitable for very deep models an interesting avenue for future work.
7 Conclusion
We presented state-of-the-arr results on modeling the density of natural images using a new class of diffusion models that incorporates a learnable diffusion specification, Fourier features for fine-scale modeling, well as other architectural innovations. In addition, we obtained new theoretical insight into fikelihood-basee generative modeling with diffusion models, showing surprising invariance of the VLB the forward time diffusion process in continuous time, well as equivalence between various diffusion processes from the literature previously thought to be different.
Acknowledgments
We thank Yang Song, Kevin Murphy and Mohammad Norouzi for feedback drafts of this paper.

---

References
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Waveg- rad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.007 2020.
Rewon Child. Very deep VAËS generalize autoregressive models and can outperform them images.
arXiv preprini arXiv:201 1.06500 2020.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:7904. 10509, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
Jarek Duda. Asymmetric numeral systems. arXiv preprint arXiv:0902.0271, 2009.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. arXiv preprint arXiv:20/2.08/25, 2020.
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, pages 5-13, 1993.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pages 2722-2730. PMLR, 2019a.
Jonathan Ho, Evan Lohn, and Pieter Abbeel. Compression with flows via local bits-back coding. In Advances in Neural Information Processing Systems, pages 3874-3855 2019b.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 11239, 2020.
Chin-Wei Huang, Jae Hyun Lim, and Aaron Courville. A variational perspective on diffusion generative models and score matching. arXiv preprint arXiv:2106.0288 02808. 2021.
Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever. Distribution augmentation for generative modeling. In International Conference Machine Learning, pages 5006-5019. PMLR, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 1rXiv:1710.10196, 2017.
Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions.
arXiv preprint arXiv:1807 00009,3,,, 2018.
Diederik Kingma and Max Welling, Auto-encoding variational Bayes, International Conference on Learning Representations, 2013.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pages 4743-4751, 2016.
Friso Kingma, Pieter Abbeel, and Jonathan Ho. Bit-swap: Recursive bits-back coding for lossless compression with hierarchical latent variables. In International Conference on Machine Learning, pages 1408-3417. PMLR, 2019.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0977 09766, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.01101, 2017.
10

---

David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling, arXiv preprint arXiv: 2018.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint
2021.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 2016.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018.
Danilo J Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Leaming, pages 1286. 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241, Springer, 2015.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.
Samarth Sinha and Adji B Dieng. Consistency regularization for variational auto-encoders. arXiv preprint wxXiv:2105.14855 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning. pages 1256-2265, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.
In Advances in Neural Information Processing Systems, pages 11895-11907, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florins Balcan, and Aisan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. arXiv e-prints, pages arXiv-2101, 2021a.
Yang Song, Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling Through Stochastic Differential Equations. In International Conference Learning Representations, 2021b.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.377 2014.
Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.1077 2020.
11

---

James Townsend, Thomas Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. In International Conference on Learning Representations, 2018.
James Townsend, Thomas Bird, Julius Kunze, and David Barber. HiLLoC: lossless image com-
pression with hierarchical latent variable models. In International Conference Learning Representations, 2020.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03889, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. arXiv preprint (rxxi:2106.059333 2021.
Aaron Van Oord, Nal Kalchbrenner. and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747-1756, 2016.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 23(7):1661-1674, 2011.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506. 150.03330. 2015.
12


Reviews:
[{"review_id": "z5P3GfVfthT", "reviewer": "Reviewer_bEoD", "summary": "The paper introduce a new insight to diffusion generative models. Mainly, the paper re-write the forward and backward processes in new formulation which incorporate the SNR value into the diffusion process. \n\nPrevious diffusion process like DDPM can be regard as instance of the proposed model. The snr value in the proposed method is learned jointly with the model whereas previous approach use fix snr value. \n\nThe paper showed various theoretical results: (i) More steps is better, (ii) for continues time loss the model is invariance to the noise schedule, (iii) the training is impacted the start and end snr values.   \nThe paper introduce to main novelties: (i) learning noise schedule and (ii) insert Fourier features as additional features\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written as show big progress in understanding diffusion process. However i concern with the following issues:\n\n(1) The paper introduce results only for low-mid resolution datasets (64x64<). Does the proposed model improved the results for larger resolution datasets, like LSUN 256x256?\n\n(2) The results in the paper is given with bits per dim. Since most of the literature in diffusion process use FID score, it would be beneficial if the paper also show the improvement in terms of FID scores. \n\n(3) The paper should add ablation analysis to see the contribution of the learned noise schedule.\n\n(4) In the ablation study, it seems that adding the Fourier features give the major part of the bits per dim improvement. Therefore, it will be beneficial to check what is the performance of other diffusion method when adding the Fourier features, e.g. Improved DDPM with Fourier features\n\n(5) There are some typo in Eq22 (2nd line), v-->v' at the beginning of the line.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitations and the potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "_QCsJkN8bO7", "reviewer": "Reviewer_2LRL", "summary": "The paper introduces a family of diffusion-based generative models for density estimations on image datasets. Here, the forward diffusion is directly defined by a set of conditional densities of diffused random variables for a given data. In particular, the paper proposes that $t$-th density for a given data follows a normal distribution; its mean and variance will be determined by data and a monotonic function of $t$, called $SNR(t)$. The generative models aim at learning diffusion processes reversed in time by maximizing its ELBO.\n\nThe paper first draws connections between the proposed models and previous diffusion-based models. For example, the authors show that the proposed method's discrete-time versions include previous finite-length Markov chain models as special cases. The paper then demonstrates that the ELBOs of the proposed continuous-time models exist. In here, the resulting ELBOs include an integration (wrt time) of a reconstruction loss weighted by the time-derivative of $SNR(t)$. Based on this, the paper shows that the losses of the previous continuous-time diffusion-based models follow the same integration form but with different weighting functions.\n\nSecond, the paper presents interesting properties of the proposed method due to $SNR(t)$. In particular, remind that $SNR(t)$ is invertible because of its monotonicity, and the continuous-time loss's integration contains the time-derivative of $SNR(t)$. The paper shows that the integral can be re-written wrt $SNR(t)$'s output by change-of-variable. In this form, the integral only depends on the range of the integration but doesn't depend on the shape of $SNR(t)$ within the range. As a result, acknowledging the invariant of the loss on the $SNR(t)$'s output space, the paper proposes to parameterize the score models conditioning on log-scaled $SNR(t)$ instead of $t$.\n\nThird, the paper proposes to train $SNR(t)$ for the discrete and continuous-time models. For the monotonicity, $SNR(t)$ is defined as a monotonic network wrt $t$. For discrete-time models, the paper proposes to train $SNR(t)$ networks by maximizing the ELBO together with the diffusion models. For continuous-time models, $SNR(t)$ is trained by minimizing the variance of the Monte-Carlo estimation of the continuous-time ELBOs.\n\nFurthermore, the paper introduces various practical techniques to improve training the models. For example, instead of iid sampling of $t$, the authors propose to use a low-discrepancy sampler of $t$ within a mini-batch. The paper also proposes to use the Fourier feature of inputs: feeding the concatenation of inputs and its Fourier feature to diffusion models.\n\nIn the experiments, the paper demonstrates that the proposed models achieve SOTA likelihoods on image datasets, such as CIFAR-10 and ImageNet.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(Originality & Significance)  \nIn my understanding, the paper's contributions are clear, and I also consider that the results are essential for several reasons.\n\nFirst, it introduces a novel family of diffusion-based models whose diffusions are characterized by $SNR(t)$. In particular, with the proposed method, the paper demonstrates the SOTA results on density estimations which autoregressive models have dominated.\n\nSecond, the novel $SNR(t)$-based characterization of diffusions helps to understand similar models. For example, the proposed models include some popular diffusion-based models as special cases; thus, one can evaluate the continuous-time ELBOs for those cases. On the other hand, one can apply similar proof techniques for more general cases. Another example includes that for continuous-time cases, the invariance-to-the-noise-schedule property clearly motivates the efficient formulation of score models: models should condition on $SNR(t)$ (or its log-scale), especially when $SNR(t)$ can vary during training.\n\nThird, the paper discusses the benefits of various practical techniques applicable to similar family models with ablation studies. For example, the low-discrepancy sampler of $t$ or Fourier features.\n\n(Quality & Clarity)  \nIn general, the paper has a well-organized structure to motivate the proposed diffusion-based models and other practical techniques to improve training.\n\n\n(Minor comments)  \nIn Lines 209-210 or Appendix B, I consider some previous works on monotonic neural networks should be cited; for example, J. Sill 1997, C-W. Huang et al. 2018, and A. Wehenkel & G. Louppe 2019.  \n\nJ. Sill, '97, Monotonic Networks  \nC-W. Huang et al., '18, Neural Autoregressive Flows  \nA. Wehenkel & G. Louppe, '19, Unconstrained Monotonic Neural Networks  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "(Limitations)  \nN/A\n\n(Societal Impact)  \nN/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": ">12hrs"}, "extra_text": {}}, {"review_id": "RNSMPmqOx4_", "reviewer": "Reviewer_m8xn", "summary": "The paper proposed an improvement to DPM. The main contribution is summarized below.\n\n- Sec 3: The diffusion and reverse processes in DPM are re-written with the proposed SNR function. It can be seen as a high-level framework of the noise schedule. \n- Sec 4: The ELBO (for both discrete and continuous DPMs) is then re-written in terms of SNR. The authors had a few other findings: e.g., a larger T is better, and equivalence between continuous DPMs. \n- Sec 5: The SNR function is parameterized by a neural network and targets at minimizing the estimation variance of ELBO. \n- Sec 6: Additional features computed from Fourier transformation are added to data.\n- Sec 7: Experiments on CIFAR and ImageNet show improvements on BPD. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is original to my knowledge. The research problem is significant, as DPMs are recently found to be very powerful in a number of generative tasks. As for quality and clarity, generally speaking, the theoretical results are correct, the experimental results are reasonable, but the writing is not very clear (detailed comments below).\n\nPros:\n- The SNR function is the most interesting and important contribution of the paper. It provides a high-level view of both discrete and continuous DPMs. Learning SNR is meaningful as most current noise schedules are heuristic and it is unclear how to setup a best schedule.\n- There are some interesting findings (e.g. \"more steps is better\" and \"equivalence between continuous DPMs\") and other useful tools (e.g. unbiased estimators and numerically stable computation).\n- Experimentally, it is shown that the proposed Fusion, with the SNR learned and additional Fourier features, lead to SOTA BPD.\n\nCons / main concerns:\n\nMy first major concern is that neither the theoretical or the empirical contribution is sufficient. Do the authors consider this paper as a theoretical paper or an empirical paper? Currently, it looks only like a somewhat smart algorithm, with limited theoretical understanding and some experimental validation. \n- If the authors want to write a **theoretical** paper, then there should be a **complete** theory for the proposed method and relative findings. However, there is no theorem/proposition in the main text or the appendix. More importantly, many findings are not formally justified, so they are only insights. Below are some suggestions for writing a theoretical paper.\n    - Line 104-105: small SNR(1) -> $q(z_1|x)\\approx$ the standard Gaussian (need theoretical analysis).\n    - Line 109-111: large SNR(0) -> $p(x|z_0)\\approx q(x|z_0)$ (need theoretical analysis).\n    - Line 152-155 and the paragraph after eq(48): $L_{2T} < L_T$. It makes sense that loss tends to be larger because $z_{t}$ is noisier than $z_{t'}$. However, a formal theoretical justification, even under some more assumptions, is necessary. Then, a tighter (non-zero) bound on $L_T - L_{2T}$ in terms of SNR is ideal.\n    - Section 5: The SNR function is learned so that it minimizes the ELBO estimation variance. To make the theory sound, I suggest providing an upper bound on the variance term, which should be relevant to the SNR function or the general noise schedule. Then, provide a theoretical bound on the possible improvement of the variance estimation. \n    - Appendix E: a sensitivity analysis for the numerical stable computation is recommended.\n- If otherwise, the authors want to write an **empirical** paper, then there should be **extensive** experiments. Below are some suggestions for making it an empirical paper.\n    - Evaluation: The BPD improvement is marginal, given that those for prior DPMs are just upper bounds. The FID on CIFAR is worse than prior DPMs. Beating prior methods is not absolutely necessary; however, given these results, it is too early to call the proposed method \"SOTA\". There should be at least some aspect that your method clearly outperforms other models. In addition, in line 267-268, those DPMs are not *specifically* designed for FID, and if you can get much better FID then you should report the setting and numbers.\n    - More datasets should be surveyed, because different datasets have different characteristics. In this paper, both CIFAR and ImageNet are very diverse datasets. How about datasets that are less diverse but have higher quality (resolution), such as CelebA-HQ and LSUN? It would be more interesting to look at other domains, such as audio or 3d point cloud generation. \n    - Figure 4 shows that the learned SNR leads to very small variance. However, it is necessary to directly compare BPD between learned SNR and prior (linear/cosine) schedules. What is the improvement of BPD if we just learn the SNR and keep all other settings the same? In addition, why does reducing the variance leads to improvement of BPD, if that is the case?\n    - I notice that in Figure 3, having \"variance minimization\" very marginally improves BPD, whereas having Fourier features significantly improves BPD. How about applying these features to prior DPMs? It seems this is the main reason for the improvement.\n    - In Table 2, when $T_{\\mathrm{eval}}$ is 100 or 1000, the discrete DPM is better than the continuous DPM. Does that conflict with line 157-158?\n\nAnother major concern is about writing. The whole paper is very hard to follow. \n\n- Intro is too short and lacks some key discussion. Why are we interested in image density estimation benchmarks (rather than e.g. FID)? What are the main drawback of current methods? What are the challenges / difficulties, and how do you get around with it? How is the proposed method meaningful for the community? What can people learn from it?\n- Always define a symbol **before** using it. (Minor concern for equations: remember to add commas or dots after an equation, such as in eq 8 and 9.)\n- Eq(1) seems coming from nowhere. More background is needed. You could start with either diffusion in physics or DDPM by Ho et al 2020. Also, $\\mathcal{N}(z_t;\\alpha_tx,\\sigma_t^2I)$. Why don't you model the covariance matrix in a similar as improved DDPM?\n- Line 80-85: The correct order is defining the parameters first and then proving the diffusion process is Markov. Then, why do we care about $q(z_s|z_t,x)$? To make it clear, the ELBO expansion should be introduced here. And similarly, a proof is needed. To make this part clearer, I suggest start with DDPM and show how to adapt the theory here (by talking about connections and modification). \n- Line 86-90: It is okay to postpone theory to section 4, but more explanation on the SNR is preferred. Briefly talk about its meaning and how it affects the diffusion process.\n- Eq (7): Why do you separate $x$ and $z_0$? In DDPM they are the same thing. What is the benefit here? (Minor concern for notations $s(i)$ and $t(i)$: sometimes the index is ignored, e.g. in eq17 and eq47. I suggest use subscriptions, and also remember to apply it to $t'$ as in eq47.)\n- Line 106-111: this part is confusing. Where did you define $p(x|z_0)$? Why can you factorize it according to dimensions? Why does it approximate the $q(x|z_0)$?\n- Line 112-116: What is the network and what are the re-parameterizations? Why does $\\hat{x}_{\\theta}$ predict $x$? And similarly, it is preferred to connect eq11 and eq12 to DDPM as well as eq4. \n- Eq 13: what is $W_t$?\n- A general problem for section 3 is that too many things are postponed to later sections without any further explanation. This makes it very hard to read for general readers.\n- Section 4.2: Do $L_T$ and $L_{2T}$ use the same SNR, or you are taking the inf over all SNR? And as mentioned before, line 151-154 is not rigorous.\n- Line 221-222: What random variables are the $\\mathbb{E}$ and VAR over? $t$, $\\epsilon$, or both? (Minor concern: in eq 50, it should be $\\frac{\\partial}{\\partial\\eta}$.)\n- Section 5: As mentioned before, why does reducing variance help? A deeper discussion is needed.\n\n\n-----------------------------------------\n\nIn the rebuttal, the authors added insights and preliminary experimental results (or descriptions) to support their methods. These sound reasonable to me so I will raise my score to 6.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and social impact are discussed in the paper. More questions on limitations are in the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "9jRg6nCtGk-", "reviewer": "Reviewer_Mjaj", "summary": "The paper tackles density estimation with denoising diffusion-based generative models and achieves very strong, state-of-the-art likelihoods (bounds) on standard image modeling benchmarks. The formulation largely relies on previous denoising diffusion probabilistic models (DDPMs), but uses the exact ELBO objective without dropping weighting coefficients in the mixture of losses, as was done in DDPM. The paper also proposes to learn the noise schedule, either using the ELBO objective or a variance reduction objective, and adds additional Fourier features into the model, which brings a significant benefit. On the theoretical side, the paper demonstrates how a change of variables in the time variable can lead to an equivalence between models trained with different noise schedules and shows how the important aspect of the noise schedule is only its behaviour at its endpoints. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Overall Impression:__ The *Fusion* model is mostly based on DDPM [1], while introducing additional tricks and architecture modifications (learning the noise schedule, adding Fourier features) that benefit the quantitative performance. The paper primarily stands out by its extraordinarily strong quantitative results, although the computational resources required for reproducing the results seem intense.\n\n__Originality:__ As mentioned, the core model formulation (section 3) is largely based on DDPM [1], while reformulating certain terms and defining the noise schedule in terms of a signal-to-noise-ratio SNR(t), but it is essentially the same model, I think. The fact that more steps is better, leading to a tighter bound (section 4.2), is well-known and has been pointed out already by [2], which proposed this class of models. The originality and conceptual novelty up to this point seems limited. \n\nThe demonstration that in the continuous time formulation the only thing that truly matters about the noise schedule are its endpoints is quite interesting, in particular considering that multiple previous and concurrent works explore different noise schedules. The idea to learn the noise schedule in itself is not novel, as it has been pointed out before [1], but this paper seems to be the first one to actually do it, proposing meaningful objectives for that. Overall, it feels the originality and theoretical/methodological novelty is not impressive, but there are some interesting contributions.\n\n__Clarity:__ The paper is well written and easy to read.\n\n__Specific Questions and Feedback:__\n1. In Section 5, in the discretized setting, the noise schedule is learned using the ELBO objective. In that case, I think the model essentially behaves as a VAE. However, VAEs suffer from the prior-hole problem, explaining their poor sample quality. One major advantage of diffusion models is that they usually avoid this problem, diffusing each individual data point to full noise by construction. But by learning the noise schedule with the ELBO, don't we re-introduce this issue? What if the noise schedule learns to add virtually no noise to the data, so the diffusion objective (which is sort of a bunch of reconstruction objectives, given the noisy data) can be easily minimized, while still encoding data points under the prior, as in a VAE, to also satisfy the prior term? I admit that if we are purely targeting strong likelihood this argument doesn't matter. Nevertheless, it seems we may lose one of the main advantages of diffusion models here and sample quality may suffer. I feel this aspect is discussed insufficiently.\n2. The authors also propose to instead learn the noise schedule with a variance reduction objective. This makes a lot of sense, but I think this is conceptually analogous to importance sampling techniques proposed by previous work: In principle, we can either optimize the noise schedule as proposed here considering uniform sampling from $t$, or alternatively we can use any arbitrary fixed noise schedule (fixing only the endpoints) and instead use importance sampling that oversamples the important $t$, similarly reducing variance. This seems basically equivalent, explained by the change-of-variables discussed in the paper. But this conceptual relation of learning the noise schedule to previously proposed importance sampling schemes is not discussed in the paper at all.\n3. Regarding the theoretical equivalence between different models with different noise schedules: Does this only hold for noise schedules corresponding to a similar underlying SDE? This is, does this mean, for example, that the $\\beta(t)$ in the VPSDE [3] is basically arbitrary? Or does this mean that also the underlying form of the SDE itself does not matter (apart from endpoint behavior and for training objective variance)? For example [3] proposed the SubVPSDE. Would it still be helpful to explore other SDEs, considering the continuous time framework, in light of the results?\n4. The strongest results are obtained using the continuous setting for both training and evaluation (see Tab. 2). How is evaluation done in this case? Via black-box ODE solvers, leveraging the connection of continuous diffusion models to SDEs and ODEs and therefore continuous Normalizing flows [3]? I feel like I am missing some details regarding the model evaluation. I also didn't find this in the appendix.\n\n__Conclusion:__ I think the paper meets the bar for publication, considering the importance of likelihood-based generative modeling and the paper's very strong quantitative results on that. It also has some minor theoretical contributions that help the understanding of diffusion models and the proposed architectural innovations may be adopted by the community.\n\n[1] Ho et al., \"Denoising Diffusion Probabilistic Models\";\n[2] Sohl-Dickstein et al., \"Deep Unsupervised Learning using Nonequilibrium Thermodynamics\";\n[3] Song et al. \"Score-Based Generative Modeling through Stochastic Differential Equations\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "__(Negative) Societal Impact:__ Has been discussed very briefly, but in a satisfactory manner.\n\n__Limitations:__ I think limitations could have been discussed more thoroughly. The proposed *Fusion* model is a typical denoising diffusion model, and as such has well-known and important limitations, I think. For example:\n1. Very slow, iterative synthesis. The model samples shown in the appendix use 1000 synthesis steps, for example.\n2. Other types of likelihood-based generative models, such as VAEs and autoregressive models, can be applied on arbitrary data types (language, graphs, discrete/binary data, etc.). Most current diffusion models, including the one in this paper, are limited to continuous data types where we can easily define meaningful diffusion processes. \n3. While the quantitative results of the paper are without doubt very strong, there seems to be a trade-off between achieving strong FID scores (or perceptual image quality more generally) and NLL. This can be considered a limitation.\n\nI am aware that this paper does not aim to address these aspects, but I think it would not hurt to acknowledge these important limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel perspective on denoising diffusion models. The main contributions of this submission include (i) a new formulation of ELBO using $SNR(t)$ and showing invariance to the noise schedule under this formulation, (ii) training noise schedule, and (iii) several new architecture improvements. The paper executes these ideas extremely well and obtains SOTA likelihood results on several images. Although the majority of the improvements in likelihood results seem to be obtained from architectural improvements (i.e., Fourier features), this paper provides very valuable insights into the diffusion models and can be of great interest to the community. Given the contributions, I am pleased to recommend this paper for acceptance. 
Decision: Accept (Poster)