Improved Denoising Diffusion Probabilistic Models
Authors: Alexander Quinn Nichol
Prafulla Dhariwal
Venue:  (2021)
Keywords: neural networks
generative models
log-likelihood
diffusion models
denoising diffusion probabilistic models
image generation

Abstract: We explore denoising diffusion probabilistic models, a class of generative models which have recently been shown to produce excellent samples in the image and audio domains. While these models produce excellent samples, it has yet to be shown that they can achieve competitive log-likelihoods. We show that, with several small modifications, diffusion models can achieve competitive log-likelihoods in the image domain while maintaining high sample quality. Additionally, our models allow for sampling with an order of magnitude fewer diffusion steps with only a modest difference in sample quality. Finally, we explore how sample quality and log-likelihood scale with the number of diffusion steps and the amount of model capacity. We conclude that denoising diffusion probabilistic models are a promising class of generative models with excellent scaling properties and sample quality.

Under review as a conference paper at ICLR 2021
IMPROVED DENOISING DIFFUSION PROBABILISTIC MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We explore denoising diffusion probabilistic models, class of generative models which have recently been shown to produce excellent samples in the image and audio domains. While these models produce excellent samples, it has yet be shown that they can achieve competitive log-likelihoods. We show that, with sev- eral small modifications, diffusion models can achieve competitive log- likelihoods in the image domain while maintaining high sample quality. Additionally, our models allow for sampling with an order of magnitude fewer diffusion steps with only modest difference in sample quality. Finally, we explore how sample qual- ity and log-likelihood scale with the number of diffusion steps and the amount of model capacity. We conclude that denoising diffusion probabilistic models are a promising class of generative models with excellent scaling properties and sample quality.
INTRODUCTION
Sohl-Dickstein et al. (2015) introduced diffusion probabilistic models ("diffusion models" for brevity), a class of generative models which match data distribution by learning reverse a grad- ual, multi-step noising process. More recently. Ho et al. (2020) showed an equivalence between
these models and score based generative models (Song & Ermon 2019 2020). which learn a gradi-
ent of the log-density of the data distribution using denoising score matching (Hyvärinen) (005)). It
has recently been shown that this class of models can produce high-qualit images (Ho al. 2020 Song & Ermon 2020 Policoeu-Maartnnal et al. 2020) and audio (Chen et al. 2020b) Kong et al., 2020), but it has yet to be shown that diffusion models can achieve competitive log-likelihoods. Furthermore, while Ho et al. (2020) showed extremely good results on the CIFAR-10 (Krizhevsky) 2009) and LSUN (Yu et al., 2015) datasets, it is unclear how well diffusion models scale to datasets with higher diversity such ImageNet. Finally, while Chen et al. (2020b) found that diffusion mod- els can efficiently generate audio using a small number of sampling steps, it has yet to be shown that the same is true for images.
In this paper, we show that diffusion models can achieve competitive log-likelihoods while main- taining good sample quality, even high-diversity datasets like ImageNet. Additionally, we show that our improved models can produce competitive samples an order of magnitude faster than those from Ho et (2020). We achieve these results by combining simple reparameterization of the reverse process variance, a hybrid learning objective that combines the variational lower-boun. with the simplified objective from Ho et al. (2020), and a novel noise schedule which allows the model to better leverage the entire diffusion process.
We find surprisingly that, with our hybrid objective, our models obtain better log likelihoods than those obtained by optimizing the log-likelihood directly, and discover that the latter objective has much more gradient noise during training. We show that simple importance sampling technique reduces this noise and allows Using our trained models, we study how sample quality and log-likelihood change as we adjust the number of diffusion steps used sampling time. We demonstrate that our improved models allow us to use an order of magnitude fewer steps at test time with only a modest change in sample quality and log- weggllielihood, thus speeding up sampling for in practical applications.
achieve better log-likelshoods than with the hybrid objective.
Finally, we evaluate the performance of these models as we increase model size, and observe trends that suggest predictable improvements in performance increase training compute.

---

Under review as a conference paper at ICLR 2021
2 DENOISING DIFFUSION PROBABILISTIC MODELS
We briefly review the formulation of diffusion models from at. 2020). This formulation makes various simplifying assumptions, such fixed noising process g which adds diagonal Gaussian noise at each timestep. For more general derivation, see Sohl-Dickstein et al. (2015).
2.1 DEFINITIONS
Given a data distribution zo ~ ((z0), we define forward noising process g which produces latents 21 through øy by adding Gaussian noise at time & with variance 3r (0, 1) as follows:
q IT xD) = II ((
g(<4 x4-1) = N(25 v 1,TT-11
(2)
Given sufficiently large T and a well behaved schedule of Bi, the latent ør is nearly an isotropic Gaussian distribution. Thus, if know the exact reverse distribution g(#;-1 (ax), we can sample IT ~ (O, I) and run the process in reverse to get sample from ((0o). However, since q( depends on the entire data distribution, we approximate using neural network:
Po (o = N (2 (
(3)
The combination of and is a variational auto-encoder (Kingma & Welling 2013). and we can write the variational lower bound (VLB) as follows:
Lub Lo + Li + + LT-1 LT
Lo = log po
(5)
L-11 DxI(g( (22, (o(
LT = (xI(a(ar(z)) p(xT))
Aside from Lo, each term of Equation and can thus be evaluated in closed form. To evaluate Lo for images, assume that each color component is divided into 256 bins, and we compute the probability po (x)) landing in correct bin (which is tractable using the CDF of the Gaussian distribution). Also note that while Lr does not depend on 0, will be close to zero if the forward noising process adequately destroys the data distribution that (#r(22)) (O, I).
KL divergence between two Gaussian distributions,
useful to define and derive several other quantities which are relevant to the forward noising process, repeat them here from Ho et al. (2020)
3
(8)
(9)
s=0
B1
3
(10)
1 - ōj
/ 04
Var(l a.-1)
(2,,, 20)
20
Dt
(11)
1 - āi
1- δz
g(zx (10)
N N(xi Vanto, - (I-ā:)I)
(12)
g(21-112;, To) (x 100,, 2o), 3rI)
(13)
2.2 TRAINING PRACTICE
Equation 12 provides efficient way jump directly arbitrary step of the forward noising process. This makes it possible to randomly sample during training. Ho et al. (2020) uniformly sample / for each image in each mini-batch.
2

---

Under review as a conference paper at ICLR 2021
There are many different ways to parameterize tt) The most obvious option is to predict He (Zx, t) directly with a neural network; alternatively. the network could predict žo. and this output could then be fed through j( zo); finally, the network could predict the noise added to Zp. and this noise could be used to predict 20 via
(14)
you
VI
di
Ho al. (2020) found that predicting e worked best, especially when combined with reweighted loss function:
Limple = Er,so.s [lle - ea (xt, 1)]
(15)
This objective can be seen a reweighted form of Lvb (without the terms affecting Eg). The authors found that optimizing this reweighted objective resulted in much better sample quality than optimizing Lvb directly, and explain this by drawing a connection generative score matching (Sog & Emmon 2019; 2020).
One subtlety is that simple provides no learning signal for Eg( t). This is irrelevant, however, since Ho et al. (2020) achieved their best results by fixing the variance to of rather than learning it. They found that they achieve similar sample quality using either of 3r or of = Be, which are two extremes given by ((za) being either isotropic Gaussian noise or a delta function, respectively.
3 IMPROVING THE LOG-LIKELIHOOD
While Ho et al. (2020) found that diffusion models can generate high-fidelity samples according to FID (Heusel et al. 2017) and Inception Score (Salimans et al. 2016), they were unable achieve competitive log- likelihhhoos with these models. Log-likelihood is widely used metric in gener- ative modeling, and it is generally believed that optimizing log- log-likelihood forces generative mod- els to capture all of the modes of the data distribution (Razavi et al. 2019). Additionally, recent work (Henighan et al. 2020) has shown that small improvements in log- likelihood can have a dra- matic impact on sample quality and learnt feature representations. Thus, it is important to explore why diffusion models seem to perform poorly on this metric, since this may suggest a fundamental shortcoming such as bad mode coverage. This section explores several modifications the algo- rithm described in Section 2 that, when combined, allow diffusion models to achieve much better logllikelhhoods on image datasets, suggesting that these models enjoy the same benefits as other lkelihood-basee generative models.
To study the effects of different modifications, we train fixed model architectures with fixed hyper- parameters (Appendix A on the ImageNet 64 x 64 (van den Oord et al., (016a) and CIFAR-10 (Krizhvssyy 2209) datasets. While CIFAR-10 has seen more usage for this class of models, we chose to study ImageNet 64 x 64 as well because it provides a good trade-off between diversity and resolution, allowing us to train models quickly without worrying about overfitting. Additionally, ImageNet 64 x 64 has been studied extensively in the context of generative modeling (van den Oord
ct al. 6016b; Menick & Kalchbrenner 2018 Child et al. 2019, Roy et al., 2020), allowing us to compare diffusion models directly to many other generative models.
The setup from Ho et al. 2020) (optimizing sippl while setting of = 3 and = 1000) achieves log-likelihood of 3.99 bits/dim ImageNet 64 x 64 after 200K training iterations. We found in carly experiments that we could get a boost in log-likelihood by increasing T from 1000 to 4000; with this change, the log-likelihood improves to 3.77 bits/dim. For the remainder of this section, we use T = 4000, but we explore this choice in Section
3.1 LEARNING Eg (x.,,)
In Ho et al. (2020), the authors set Eo,,, ILI, where o, is not learned. Oddly, they found that fixing of to 3. yielded roughly the same sample quality as fixing it to 3. Considering that B, and 3, represent two opposite extremes, it is reasonable to ask why this choice doesn't affect samples. One clue is given by Figure la] which shows that 3e and 3 are almost equal except near t = O, Le. where the model is dealing with imperceptible details. Furthermore, we increase the number of diffusion steps, B, and 3, seem remain close to one another for more of the diffusion process.
3

---

Under review as a conference paper at ICLR 2021
107
6X00
10"3
Av.
4x100
30
3x10
1000 xcapa
staps
10-0
10000 steps
10
5.2
diffunton
0.6
3.5
500
diftusion
2000
3000
4000
Figure la: The ratio B. for every diffusson step for diffusion processes of different lengths.
Figure lb: Terms the diffusion step.
The first few terms contribute most to NLL.
Latent samples from linear (top) and cosine (bottom) schedules respectively linearly spaced values of t from O to T. The latents in the last quarter of the linear schedule are almost purely noise, whereas the cosine schedule adds noise more slowly
Figure 2:
This suggests that, in the limit of infinite diffusion steps, the choice of o, might not matter at all for sample quality. In other words, as we add more diffusion steps, the model mean (o( t) determines the distribution much more than (x t)-
While the above argument suggests that fixing de is a reasonable choice for the sake of sample quality, it says nothing about log- ligeliikllhood. In fact, Figure [ shows that the first few steps of the diffusion process contribute the most to the variational lower bound. Thus, it seems likely that we could improve log-likelihood by using a better choice of Ea(x,, (). To achieve this, we must learn Eg (Iz, t) without the instabilities encountered by Ho et al. 2020))
Since Figure La shows that the reasonable range for (2,,, very small, it would be hard for neural network to predict [x t) directly, even in the log domain, as observed by Ho et al. 2020 Instead, we found it better to parameterize the variance an interpolation between 3r and 3, in the log domain. In particular, our model outputs vector u containing one component per dimension, and we turn this output into variances as follows:
Ea(x,, () exp(v log + (1 - v) log B.)
(16)
We did not apply any constraints on theoretically allowing the model to predict variances outside of the interpolated range. However, we did observe the network doing this in practice, suggesting that the bounds for Es(x,, () are indeed expressive enough.
Since Tximple doesn't 't depend on Eg(xe; l), we define new hybrid objective:
Taybria
Lsimple
ALvıb
(17)
For our experiments, we set À = 0.001 to prevent Lvb from overwhelming Lsimple. Along this same line reasoning, we also apply stop-gradient to the j (æ+, t) output for the Toyh term. This way, Tw/b can guide Eg(x,, f) while Tsimple is still the main source of influence over (ate, f).
3.2 IMPROVING THE NOISE SCHEDULE
We found that the noise schedule used in Ho et al. 2020) was sub-optimal for ImageNet 64 x 64. In particular, the end of the forward noising process is too noisy, and doesn't contribute very
4

---

Under review as a conference paper at ICLR 2021
+ ear schedule
Inear
cosine schedule
2
0.4
0.2
0.0
0.0
0.1
0.0
0.2
0.4
0.5
0.6
1.0
fruction of flusion
difusiio
Figure 3a: FID when skipping prefix of the
Figure 3b: de throughout diffiusion the linear schedule and our proposed cosine schedule.
reverse diffusion process on ImageNet 64 x 64.
5.00
4.75
Tuyiel (reammpted)
100
Caineo beg (00atim))
10
Tmm Mm
102
4.00
107
3.75
3.30
325
100
Lu
3.00
200
30"
25
50
75
100
125 (1+3)
150
175
200
Figure 4a: Learning curves comparing the log- likelihoods achieved by different objectives on ImageNet 64 64.
Figure 4b: Gradient noise scales for the Lvm and
Lighei objectives on ImageNet x 64.
much to sample quality. This can be seen visually in Figure 2 The result of this effect is studied in Figure 3a] where we see that model trained with the linear schedule does get much worse (as measured by FID) when we skip up to 20% of the reverse diffusion process.
To address this problem, construct different noise schedule in terms of ā
f(t) a. S(O)
t/T + π
f(t) = cos
8 0.008
(18)
1+8 2
To go from this definition variances 3,, we note that 3r no larger than 0.999 prevent singularities the end of the diffusion process near T..
In practice, we clip 3r to be
Our cosine schedule is designed to have a linear drop-off of ãr in the middle of the process, while changing very little near the extremes of 0 and / T prevent abrupt changes in noise level. Figure shows how āg progresses for both schedules. We can see that the linear schedule from Ho al. (2020) falls towards zero much faster, destroying information more quickly than necessary.
The small offset in our schedule prevents 3r from being too small near t = O, since we found that having tiny amounts of noise the beginning of the process made it hard for the network to predict accurately enough. In particular, we selected such that vBa was slightly smaller than the pixel bin size, 1/127.5. We chose use cos² particular because is a common mathematical function with the shape we were looking for. This choice was arbitrary, and we expect that many other functions with similar shapes would work as well.
3.3 REDUCING GRADIENT NOISE
We expected to achieve the best log- legliikeihoods by optimizing Lvb directly, rather than by optimiz- ing Laybrid However, we were surprised to find that Lyjb was actually quite difficult to optimize in practice, at least the diverse ImageNet 64 x 64 dataset. Figure 4a shows the learning curves for both LyB and Lhybrid- Both curves are noisy, but the hybrid objective clearly achieves better nieliikelssods the training set given the same amount of training time.
5

---

Under review as a conference paper at ICLR 2021
MODEL
TRAIN ITERS.
T
SCHEDULE OBJECTIVE
NLL (bits/dim)
FID
200K
IK
linear
Liimple
3.99
31.0
Baseline
200K
4K
linear
Liimple
3.77
29.7
200K
4K
linear
Lhybrid
3.66
30.4
Improved
200K
4K
cosine
Esimple
3.68
25.6
200K
4K
cosine
Laybrid
3.62
26.6
200K
4K
cosine
Lubb
3.57
54.7
I.5M
4K
cosine
Lhybrid
3.57
18.3
Improved
1.5M
4K
cosine
Lubb
3.53
38.3
Table 1: Comparison of NLL and FID for different diffusion models on ImageNet 64 x 64. Lutb and Lyybrid were trained with learned sigmas using the parameterization from Section 3.1 For Lybb, we used the resampling scheme from Section Using our cosine schedule and Lnybrid improves both log likelihood and FID over the baseline from Ho et (2020). Optimizing Lyth further improves log-likelihood at the cost of higher FID.
MODEL
TRAIN ITERS.
T
SCHEDULE OBJECTIVE
NLL (bits/dim)
FID
500K
IK
linear
Lsimple
3.73
3.29
Baseline
500K
4K
linear
Lrimple
3.37
2.90
500K
4K
linear
Lhybrid
3,26
3.07
Improved
500K
4K
cosine
Lsample
3.26
3.05
500K
4K
cosine
Lhybrid
3.17
3.19
500K
4K
cosine
Lvbb
2.94
11.47
Table 2:
Comparison of NLL and FID for different diffusion models on CIFAR-10. Using our cosine schedule and Lhybria improves log-likelihood with a marginal impact on FID, Optimizing Lvbb further improves log-likelihood at the cost of significantly higher FID.
We hypothesized that the gradient of Lutb much noisier than that of Lhybrid- We confirmed this by evaluating the gradient noise scales (McCandlish et al.] (018)) for models trained with both objectives, shown in Figure 4b] Thus, we sought out way reduce the variance of Lub order to optimize directly for log-likelihood.
Noting that different terms of T.ub have greatly different magnitudes (Figure 1b hypothesized that sampling t uniformly causes unnecessary noise in the Lubb objective. To address this, we employ importance sampling:
L where Pt x / ETL and Σ =
Lubb Exsop
(19)
Since E[[.]] is unknown beforehand and may change throughout training. maintain history of the previous 10 values for each loss term, and update this dynamically during training. At the beginning of training, we sample uniformly until we draw 10 samples for every & 11.
With this importance sampled objective, we are able to achieve our best log-likelihoods by optimiz- ing Lvb [ This can be seen in Figure 4ajas the "Leth (resampled)" curve. The figure also shows that the importance sampled objective is considerably less noisy than the original, uniformly sampled objective.
3.4 RESULTS AND ABLATIONS
In this section, we ablate the changes we have made to achieve better log-likelihoods. Table summarizes the results of ablations ImageNet 64 64, and Table them for CIFAR- 10. We also trained our best ImageNet 64 x 64 models for 1.5M iterations, and report these results as well. Based on the results, recommend always using the cosine schedule, and the Laybrid objective in most cases. If one is only optimizing for likelihood and not sample quality, the importance sampled Lub is the best objective use.
We found that the importance sampling technique helpful when optimizing Taybni directly.

---

Under review as a conference paper at ICLR 2021
MODEL
ImageNet 64 64
CIFAR-10
NLL (bits/dim)
NLL (bits/dim)
Glow (Kingma & Dhariwal 2018)
3.81
3.35
+low++ (Ho et al. 2019)
3.69
3.08
PixelCNN van den Oord et al.
2016b
3,57
3.14
PixelSNAIL (Chen al. 2018
3.52
2.85
SPN (Menick & Kalchbrenner 2018)
3.52
Image Transtormer (Parmar at.
2018
3.48
2.90
Sparse Transformer (Chuld et al.
2019)
3,44
2.80
Routing Transformer (Roy al.
2020)
3,43
Diffusion (Ho et al. 2020)
3.77
3.70
Improved Dittusion (ours)
3.53
2.94
Table 3: Comparison of diffusion models other likelihood-based models CIFAR-I0 and Un- conditional ImageNet 64 x 64. On ImageNet 64 x 64, our model is competitive with the best conventional models, but worse than fully transformer-based architectures.
4 IMPROVING SAMPLING SPEED
All of our models were trained with 4000 diffusion steps, and thus producing a single sample takes several minutes on a modern GPU. this section, we explore how performance scales we reduce the steps used during sampling, and find that our pre-trained models can produce high-quality sam- ples with many fewer diffusion steps than they were trained with without any fine-tuning. Reducing the steps in this way makes it possible to sample from our models in number of seconds rather than minutes, and greatly improves practical applicability image diffusion models.
For model trained with T diffusion steps, we would typically sample using the same set of / values (1, ..., 7) as used during training. However, it is also possible sample using an arbitrary set of t values. We define sequence S of values to use for sampling, such strided schedule like = (1, ..., T - 1). Given the training noise schedule ā,, we can obtain the sampling noise schedule os,, which can be used to obtain corresponding sampling variances
AS. = ūSS.-1 3s.
Bs. = 1
(20)
ās.
We can compute p(as 2 (s))) N ((uo (2 St), Eo(rs, St)). Note that Ea(ss, St) is parame- terized as a range between Ss, and Bs, so will automatically be rescaled for the shorter diffusion process.
To evaluate sample quality for reduced numbers of sampling steps, we use a stride K over timesteps reduce the total number of sampling steps from T to T/K. In Figures 5a and [cc] we evaluate FIDs for an Laybrid model and an Isimpl model that were trained with 4000 diffusion steps, using 30, 50, 100, 150, 200, 400, and 4000 sampling steps. We do this for multiple checkpoints throughout training. We find that the Lsiple model suffers much more in sample quality when using reduced number of sampling steps, whereas our Lnybrid model maintains sample quality. Furthermore, we find that using more sampling steps becomes increasingly beneficial throughout training, However, 100 sampling steps is still sufficient achieve near-optimal FIDs for our fully trained models.
In initial experiments, we found that although constant striding did not significantly affect FID, it drastically reduced likeliiol.oodd. To address this, we use a strided subset of timesteps as for FID (with stride K). but we also include every from 1 to T/K. This requires T/K extra evaluation steps, but greatly improves log-likelihood compared to the uniformly strided schedule. In Figures abland 5d we present log-lik likeliioksoods with this modified strided schedule.
SCALING MODEL. SIZE
In the previous sections, showed algorithmic changes that improved liglliill and FID without changing the amount of training compute. However, a trend in modern machine learning is that larger models and more training time tend improve model performance (Kaplan al. 2020

---

To measure how performance scales with compute, we train four different models on ImageNet 64 64 with the Lyyeri objective described in Section 3.1 To change model capacity, we apply depth multiplier across all layers, such that the first layer has either 64, 96, 128, or 192 channels. Note that our previous experiments used 128 channels layer affects the scale of the initial weights, we scale the Adam learning rate for each model by our Under review as a conference paper at ICLR 2021
Campie 200K iters)
(Camas 200K iters)
Crpons (200K iters)
500K iters)
500K iters)
Lhoons (500K iters)
(Aage. 1500K itens)
1500K
Chobrs (1500K iters)
4.1
4.0-
2
3.9
3.6
16a
107
sampling stops
evoluctioo
Figure 5a: FID versus sampling steps on Ima-
Figure 5b: NLL versus evaluation steps on Ima- geNet 64 x 64.
geNet 64 x 64.
20.0
4.0
17.5
15.0
12.5
3.6-
2 10.0
32-
102
10a
10
sampling stops
sampling
Figure 5c: FID versus sampling steps CIBAR-
Figure 5d: NLL versus evaluation steps on CIFAR-10.
10.
Figure 5: NLL and FID versus number of evaluation/sampling steps, for models trained ImageNet 64 x 64 and 10. All models were trained with 4000 diffusion steps. Models that learn sigmas using our reparametrization and Layond objective (Section 3.1) increase marginally NLL and FID as we reduce evaluation/sampling steps, while using fixed sigmas as in Ho et al. (2020) results in larger increase.
Chen al. 2020a Brown al., 2020). Given this observation, investigate how FID and NLL scale as a function of model size. Our results suggest that diffusion models can achieve better and better performance as training compute increases.
the first layer. Since the depth of each
1/ /channel multiplier, such that the 128 channel model has learning rate of 0.0001 other experiments).
Figure 6aa and 6b show how FID and NLL improve relative to compute. These plots reveal that, to achieve optimal performance for given amount of compute, it often makes sense to train larger model for fewer iterations, rather than training a smaller model to convergence. We note that these models do not achieve optimal li-liieiihoods because they were trained with Layond objective and not directly with Lyb to keep both good log ligeliioohsos and sample quality. The x-axis both figures is the theoretical amount of training compute, assuming full hardware utilization.
8

---

Under review as a conference paper at ICLR 2021
64 (30M params)
params)
60
parems)
96 ch (6SM params)
128 ch 1120M params)
128 ch (12 params)
192 1270M params)
3.75-
1270M params)
50
30
3.60-
20
3.35
10
10
1040
10
compute (FLOP4)
compute (FLLGP))
Figure 6a: FID throughout training on ImageNet
Figure 6b: NLL throughout training on ImageNet 64 64 for different model sizes.
64 64 for different model sizes.
6 RELATED WORK
Chen et al. 2020b) and Kong et al. (2020) are two recent works that use diffusion models to produce high fidelity audio conditioned on mel-spectrograms. Concurrent our work, Chen et (2020b) use combination of improved schedule and L1 loss to allow sampling with fewer steps with very little reduction sample quality. However, compared to our unconditional image generation task, their generative task has strong input conditioning signal provided by the mel-spectrograms, and we hypothesize that this makes easier to sample with fewer diffusion steps.
Jolicocur-Martincau Martineau al. (2020) explored score matching in the image domain, and constructed an adversarial training objective to produce better zo predictions. However, they found that choosing better network architecture removed the need for this adversarial objective, suggesting that the adversarial objective is not necessary for powerful generative modeling.
7
CONCLUSION
We have shown that, with few modifications, diffusion models can sample much faster and achieve better likeeiisssoods with little impact sample quality. Here summarize main findings:
Our cosine noise schedule improves NLL (and sometimes FID) compared the linear schedule from Ho et al. (2020).
Learning Mg using our parameterization and Laybrid objective provides good trade-off be- tween NLL and FID. More importantly, it allows sampling with many fewer steps without decreased sample quality.
One can optimize Lup directly using our importance sampling technique achieve the best possible NLL at the expense of sample quality.
The combination of these results makes diffusion models attractive choice for generative model- ing, since they combine good ligelhik,,,oods high-quality samples, and fast sampling with a well- grounded, stationary training objective. Furthermore, have investigated how diffusion models scale with the amount of available training compute, and found that more training compute trivially leads better sample quality and ligliiili.ood. These results indicate that diffusion models a promising direction for future research, especially as the affordability of compute increases over time.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels, 2020a. URL http::tttt//teett //cdn/ opp./ai..o..........
htom://appess/eeeeratieeeeeeeeee www.eeerrrinnngineieenn wwwinnnnnnnnnineeeeeeeeee Pixpom____eelsseee V2. pdf
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation, 2020b.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Picter Abbeel. Pixelsnail: An improved au- toregressive generative model. In International Conference on Machine Learning, pp. 864-872 PMLR, 2018.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers, 2019.
Terrance Veieis Michal Drozdzal, and Graham W Taylor. Instance selection for gans. arXiv preprint arX1v:2007.15255, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Rad- ford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: based generative models with variational dequantization and architecture design. arXiv preprint 1oXi:::902..02255. 2019.
Improving flow-
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(App):695-709, 2005.
Alexia Jolicoeur-Martineau. Rémi Piché- Taillefer, Rémi Tachet des Combes, and Ioannis Mitliagkas. Adversarial score matching and improved sampling for image generation, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in neural information processing systems, pp. 10215-10224, 2018.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, 2020.
10

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009. URL http://
WWW. CS ..s.teee toronto -ttp://eiiiiiaaaaaaae-e--tteeeee
pdf
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training, 2018.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling, 2018.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv:1802.05733 2018.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2, 2019.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-basee sparse attention with routing transformers, 2020.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications, 2017.
Jascha Dicksiii,,,,,,, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Prucessing Systems, pp. 11918-11930 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.0901,, 2020.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders, 2016a. URL http://
Image-net.org /mtg://mooo/./ phpl
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders, 2016b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans the loop, 2015.
Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen, Feature quantization im- proves gan training. arXiv, pp. arXiv-2004, 2020.

---

Under review as a conference paper at ICLR 2021
A HYPERPARAMETERS
For all of our experiments, we use UNet model architecture similar to that used by Ho et al. (2020). We changed the attention layers to use multi-head attention (Vaswani et al. (017), and opted use four attention heads rather than one (while keeping the same total number of channels). We employed attention not only at the 16x 16 resolution, but also at the 8x8 resolution. Additionally, we changed the way the model conditions on vector e and injecting it into hidden state / as GroupNorm (ooooommmmmmmtmm (b++),, we compute conditioning vectors w and and inject them into the hidden state as GroupNNorm(4) (w+ +1++++++ +6. We found preliminary experiments ImageNet 64 x 64 that these modifications slightly improved FID.
particular, instead of computing conditioning
We used a 120M parameter model for all ImageNet 64 x 64 experiments except in Section 5 where scaled the number of channels in all layers. In this architecture. the downsampling stack performs four steps of downsampling, each with three residual blocks (He et al. (015). The upsampling stack setup mirror image of the downsampling stack. From highest to lowest resolution, the UNet stages use [C, 2C, 3C, 4C| channels, respectively. In all experiments except those in Section 5 we C 128. We estimate that, with C 128, our model requires roughly 39 billion FLOPs the forward pass.
For our CIFAR-10 experiments, used smaller model with three resblocks per downsampling stage and layer widths [C, 2C, 2C, 2C with C' = 128. We swept over dropout values (0.1,0.2, 0.3) and found that 0.1 worked best for the linear schedule while 0.3 worked best for our cosine schedule (Section
3.2
We expand upon this in Appendix
For all of our experiments, we used Adam (Kingma & Ba 2014) with batch size of 128 and an exponential moving average (EMA) over model parameters with a rate of 0.9999. Except in Section 5) we fixed the learning rate 0.0001. For quick comparisons in Section 3 we trained models for 200K iterations. This is not enough to reach convergence, but we believe it is enough to fairly compare different modifications. We then trained the best models for 1.5M iterations to achieve better performance.
When using the linear noise schedule from Ho et al. (2020), we linearly interpolated from 31 = 0.0001/4 to 84000 = 0.02/4 in order to preserve the shape of ã, for the T = 4000 schedule.
When computing FID for CIFAR-10, we produce 50K samples and compare them against the train- ing set for consistency with other work. When computing FID for ImageNet 64 x 64, we produce 10K samples and compute FID against 50K validation images unless otherwise stated. Using only 10K samples biases the FID be worse-than-neeeet necessaay,, but requires much less compute for sam- pling. Since we mainly use FII) for relative comparisons, this bias acceptable.
PIn initial experiments, we found that ResNet-style architecture with no downsampling achieved hetter ligeliikls but FIDs than the UNet architecture.
12

---

Under review as a conference paper at ICLR 2021
B SAMPLES
Figure 7a: 50 sampling steps
Figure 7b: 100 sampling steps
Figure 7c: 200 sampling steps
Figure 7d: 400 sampling steps
Figure 7e: 1000 sampling steps
Figure 7f: 4000 sampling steps
Figure 7: Unconditional ImageNet 64 x 64 samples as we reduce number of sampling steps for Lhybai model with 4K diffusion steps trained for 1 1.5M training iterations.
13

---

Under review as a conference paper at ICLR 2021
Figure 8a: 50 sampling steps
Figure 8b: 100 sampling steps
Figure Sc: 200 sampling steps
Figure 8d: 400 sampling steps
Figure 8e: 1000 sampling steps
Figure 8f: 4000 sampling steps
Figure 8: Unconditional CIFAR-10 samples as we reduce number of sampling steps for Lhybeid
model with 4K diffusion steps trained for 500K training iterations.
14

---

Under review as a conference paper at ICLR 2021
Figure 9a: Samples from Ligteid model
Figure 9b:
Samples from Lum model
Figure 9: Unconditional ImageNet 64 x 64 samples generated from an Laybria and Lith model respec-
tively using the exact same random noise. Both models were trained for 1.5M training iterations.
Figure 10a: Samples from Tnprrid model
Figure 10b: Samples from Lub model
Figure 10: Unconditional CIFAR-10 samples generated from an Lisbrid and Lvbb model respectively using the exact same random noise. Both models were trained for 500K training iterations.
15

---

Under review as a conference paper at ICLR 2021
1,950
kh
1,025
1,000
6 Iyhri
0.975
0.950
ensemble
0.925
0.900
Figure 11: Samples from deb and Osytrid as well as ensemble produced by using Avn for the first and last 100 diffusion steps. For these samples, seed was fixed, allowing a direct comparison between models.
1000
2000
2500
3000
3500
4000
diffusonn
Figure 11a: The ratio between VLB each diffusion step Osytria and Oyb- Values less than 1.0 indicate that Ghyhnd is "hetter" than Byth for that timestep of the diffusion process.
for
Figure 12a: Samples with random noise.
Figure 12b: Samples with same noise in column
Conditional ImageNet 64 x 64 samples generated from an Laybrid model trained for 1.7M training steps. The classes are 9; ostrich, 11: goldfinch, 130: flamingo, 141: redshank, 154: pekinese, 157: papillon, 97: drake and 28: spotted salamander. On right we fix the random noise seed in each column to see how the class label affects the sampling process.
Figure 12:
C COMBINING LIYBRID AND LVLR MODELS
To understand the trade-off between Laybrid and Lvlb, we show in Figure IIa that the model resulting from Lvb (referred to (lvb) is better the start and end of the diffusion process, while the model resulting from Laybrid (referred to as Onybria) is better throughout the middle of the diffusion process. This suggests that Ovbb is focusing more imperceptible details, hence the lower sample quality.
Given the above observation, we performed an experiment on ImageNet 64 x 64 to combine the two models by constructing ensemble that uses Onybrid for & [100, T 100) and 0 elsewhere. We found that this model achieved an FID of 18.9 and an NLL of 3.52 bits/dim. As we see from Table I this is only slightly worse than Onybrid terms of FID, while being better than both models in terms of NLL.
D COMPARING SAMPLE QUALITY TO OTHER GENERATIVE MODELS
While this paper does not focus comparing sample quality to other types of generative models, we were curious how diffusion models compared to modern generative models on ImageNet x 64. Unfortunately, we did not find any literature which computed FID for unconditional ImageNet
16

---

Under review as a conference paper at ICLR 2021
MODEL
FID
FQ-GAN (Zhao et al. 2020)
9.67
Instance Selection GAN (De GAN(DeVrii et al., 2020)
9.07
Improved Diffusion (ours)
8.43
Table 4: Sample quality comparison on class conditional ImageNet 64 x 64.
linear (test)
9
lineer Iraint
costne
3.35
codine ttesti
(trai))
3.25
2
5-
3.35
500
training iters (thourands)
Figure 13b: Negative log-likelihood over course of training.
Figure 13a: FID over the course of training.
Figure 13: Evaluation metries over the course of training for two CIFAR-10 models, both with dropout 0.1. The model trained with the linear schedule learns more slowly, but does not overfit as quickly. When too much overfitting occurs, we observed overfitting artifacts similar to those from Salimans et al. (2017), which is reflected by increasing FID.
64 x 64. We ran an additional experiment where we trained a class-conditional diffusion model for 1.7M iterations using the Tayhri objective. To make the model class-conditional, inject class information through the same pathway as the timestep f. particular, we add a class embedding ey to the timestep embedding ex, and pass this embedding to residual blocks throughout the model. When computing FID for this task, we generated 50K samples (rather than 10K) to be directly comparable other works. We found that using more samples led a decrease in estimated FID of roughly 2 points. This is the only FID we report that was computed using 50K samples. Figure 12 shows our samples, and Table summarizes our results.
E OVERFITTIG ON CIFAR-10
On CIFAR-I0, we noticed that all models overfit, but tended reach similar optimal FID some point during training. Holding dropout constant, we found that models trained with our cosine schedule tended to reach optimal performance (and then overfit) more quickly than those trained with the linear schedule (Figure 13). In our experiments,
corrected for this difference by using more dropout for our cosine models than the linear models. We suspect that the overfitting from the cosine schedule is either due 1) less noise in the cosine schedule providing less regularization, or 2) the cosine schedule making optimization, and thus overfitting, easier.
17


Reviews:
[{"review_id": "kn3xx0MTdv3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper found several methods to improve log likelihood of diffusion models while maintain their sample quality, including cosine instead of linear noise schedule, using a hybrid objective to learn parameters of the covariance function, and using importance sampling to improve the gradient noise. The authors also explore how sample quality and log likelihood scale with the number of diffusion steps and model capacity. Experiments on 64x64 ImageNet dataset show competitive llh while keeping the sample quality.\n\nClarity: as the denoising diffusion model, especially its success in generating high-quality image samples, is still quite new, it would be beneficial if the paper could describe the technical background of it and the existing variational training methods in more details. Section 2 kind of serves this purpose, but it misses many important steps and focuses more on defining terms used in later sections.\n\nSignificance of this work: The necessity for having larger log likelihood for the diffusion model is not very well motivated. If the model is mostly used to generate high-quality samples and we have known how to train it to do so, why does the LLH values still matter?\n\nOriginality: using cosine noise schedule, new parameterization and hybrid objective seems effective for training, but doesn't seem to be very innovative. But I'm not very familiar with the denoising diffusion model and could be wrong. The results on how the model scales with computation seems trivial and may have been known already. Finally, it'll be beneficial if the authors can verify findings got in this paper can apply to other dataset or types of data more broadly.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RWWvPZfKbv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Denoising diffusion probabilistic models have been proved to produce excellent samples in the image and audio domains. However, it has yet to be shown that they can achieve competitive log-likelihoods. This paper shows that with several small modifications, diffusion models can achieve competitive log-likelihoods in the image domain while maintaining high sample quality. This paper is well-written and good-organized. However, I have the following concerns.\n\n1.\tThe authors claim that the noise schedule used in Ho et al. (2020) was, experimentally, sub-optimal for ImageNet $64 \\times 64$, which lacks theoretical guarantees.\n2.\tThis manuscript is mainly based on the previous work Ho et al. (2020). The novelty seems to be too limited.\n3.\tI am not convinced that only one dataset (ImageNet $64 \\times 64$) is sufficient to demonstrate the performance of the proposed strategy.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IdSqd5RX6eC", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThis paper presents rich discussions and various practical techniques to improve the training of probabilistic diffusion models, which include a hybrid objective to learn the variance for improving log-likelihood performance, a different noise schedule tailored for ImageNet 64x64 and importance sampling to reduce gradient noise. Experiments on ImageNet 64x64 and various ablation study provided interesting insights and empirically justified the claims.\n\n**Pros**\nThe paper is well-written and develops some useful practical techniques for improving a recently proposed deep generative model (a modified version of the diffusion probabilistic model in (Jascha, et al 2015)). Specifically, the paper managed to improve the log-likelihood performance by identifying the issue of the simplified objective and proposed to learn the variance using a hybrid objective. I think this technique along with others are useful practical techniques to improve the training of diffusion models.\n\n**Questions & Concerns**\n- No results on CIFAR: Most recent papers in this field considers CIFAR-10 as the standard benchmark to report generative performance, including the original DDPM paper [1] and the score matching paper [2]. Although ImageNet 64x64 is a larger dataset with more complicated structure and diversity, outperforming pervious strong baselines in CIFAR-10 is still challenging and non-trivial. Thus the empirical study will also be more convincing to demonstrate that the proposed method can indeed achieve much better log-likelihood without sacrificing sample quality too much. Otherwise, it's hard to get a sense of how much improvement has been actually achieved by directly looking at the numbers in this paper and the ones in previous papers.\n\n- As the major contribution, the variance parametrization (Eq 16) needs more insightful discussions. For example, why this is the case: \"Since Figure 1a shows that the reasonable range for\u0012$\\Sigma(x_t; t)$ is very small, it is clear that we should not use a neural network to predict\u0012$\\Sigma(x_t; t)$ directly.\". Can we predict the log of the variance with a neural network directly? The proposed one is only an interpolation between $\\beta_t$ and $\\tilde{\\beta}_t$ - is this expressive enough?\n\n- About the noise schedule: is this a generally better noise schedule, or it is only tailored for ImageNet 64x64. In the latter case, I think it is only a trick that overfits a specific dataset. To improve the training of DDPM generally, is there any advice on how to find a good noise schedule?\n\nI will consider raising my score if the above concerns can be addressed.\n\n[1] Denoising Diffusion Probabilistic Models\n\n[2] Generative modeling by estimating gradients of the data distribution", "extra_scores": {}, "extra_text": {}}, {"review_id": "6Hq69CeN4b", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper talks builds upon the recent work from Ho (2020) about generative models that use noise diffusion. The authors suggest that the proposal in Ho can not only be used in good quality sample generation (as already shown by Ho), but also leads to reasonable improvements in likelihood. Overall, some of the ideas presented in the paper are interesting and useful; but the paper overall needs work. \n\nOther questions/concerns:\n1. Firstly, from an application point of view, what does achieving a high log-likelihood mean, if the samples are already good enough or high quality? \n2. How do we interpret the bits/dim metric here? Its rather hard to rationalize that a change in 0.01 makes sense in this metric? And more generally, what are we aiming for in terms of a reasonable change?\n3. In section 3.1; how did we end up needing to tune big-sigma_theta (x_t, t) while arguing that fixing small-sig_t^2 is ok? This is in section 3.1 second paragraph; Either I am missing something of the argument here is that we need to tune noise variance and cannot fix it? \n4. What is the intuition behind expecting the (squared) cosine schedule to work? It is interesting to think that a periodic decay noising schedule is better than a linear one? \n5. And related to that, do not understand this weird value of 0.008 for s? The whole point here is some small non-zero s is ok; why specifically 0.008?! \n6. One of the main conclusions in section 3.4 is kind of confusing --- based on the summary, if we are not interested in sample quality but only interested in maxing of likelihood, then the proposal of this work is not good, and working with L_vlb suffices? Is this correct? Based on the motivation, it seems the opposite was being claimed i.e., L_hybrid is important for maxing of likelihood (third para in introduction)? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper arose a number of questions and concerns among Reviewers that made it get below-average scores (unfortunately, Reviewers did not provide further feedback on the rebuttal). After discussion between the Program Chairs, calibrating decisions across all submissions and, given the drawbacks mentioned below, it is decided that this paper does not meet the bar for this year's ICLR. Therefore, the final decision is to REJECT the paper. As a brief summary, I highlight below some pros and cons that arose during the review and meta-review processes.

Pros:
- Further developing on a simplification of previous approaches (learning diffusion sigmas).
- Proposal of a new noise schedule.
- Improving the log-likelihood of diffusion-based generative models.
- Improving generation time.

Cons:
- Similar FIDs as non-improved approaches (in some cases).
- Focus on log-likelihood may not be of paramount importance for a generative task.
- Dichotomy between better FID and better NLL could be further discussed.
- More comparison with other approaches and further data sets could be done.
- A bit ad-hoc noise schedule.

Decision: Reject