Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
Authors: Zhisheng Xiao
Karsten Kreis
Arash Vahdat
Venue: ICLR 2022 Spotlight (2022)

Abstract: A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously addressing three key requirements including: high sample quality, mode coverage, and fast sampling. We call the challenge imposed by these requirements the generative learning trilemma, as the existing models often trade some of them for others. Particularly, denoising diffusion models have shown impressive sample quality and diversity, but their expensive sampling does not yet allow them to be applied in many real-world applications. In this paper, we argue that slow sampling in these models is fundamentally attributed to the Gaussian assumption in the denoising step which is justified only for small step sizes. To enable denoising with large steps, and hence, to reduce the total number of denoising steps, we propose to model the denoising distribution using a complex multimodal distribution. We introduce denoising diffusion generative adversarial networks (denoising diffusion GANs) that model each denoising step using a multimodal conditional GAN. Through extensive evaluations, we show that denoising diffusion GANs obtain sample quality and diversity competitive with original diffusion models while being 2000$\times$ faster on the CIFAR-10 dataset. Compared to traditional GANs, our model exhibits better mode coverage and sample diversity. To the best of our knowledge, denoising diffusion GAN is the first model that reduces sampling cost in diffusion models to an extent that allows them to be applied to real-world applications inexpensively.

Published as a conference paper at ICLR 2022
TACKLING THE GENERATIVE LEARNING TRILEMMA WITH DENOISING DIFFUSION GANS
Zhisheng Xiao"
Karsten Kreis
Arash Vahdat NVIDIA
The University of Chicago zx aoeuchi cago. edu
NVIDIA Dkreis@nvidiin
ABSTRACT
A wide variety deep generative models has been developed in the past decade. Yet, these models often struggle with simultancously addressing three key require- ments including: high sample quality, mode coverage, and fast sampling. We call the challenge imposed by these requirements the generative learning trilemma, the existing models often trade some of them for others. Particularly, denoising diffusion models have shown impressive sample quality and diversity, but their ex- pensive sampling does not yet allow them to be applied in many real-world appli- cations. In this paper, argue that slow sampling in these models is fundamen- tally attributed to the Gaussian assumption in the denoising step which is justified only for small step sizes. To enable denoising with large steps, and hence, to re- duce the total number denoising steps, we propose to model the denoising distri- bution using complex multimodal distribution. We introduce denoising diffusion generative adversarial networks (denoising diffiusion GANs) that model each de- noising step using multimodal conditional GAN. Through extensive evaluations, we show that denoising diffusion GANs obtain sample quality and diversity com- petitive with original diffusion models while being 2000 faster on the CIFAR-10 dataset. Compared to traditional GANs, our model exhibits better mode coverage and sample diversity.
the best of our knowledge, denoising diffusion GAN is the first model that reduces sampling cost in diffusion models to extent that al- lows them to be applied to real-worde applications inexpensively. Project page and code: ht / nvv abs g thub. o denoi i a sing- di ffus on- gan.
INTRODUCTION
In the past decade, a plethora of deep generative models has been developed for various domains such as images (Kar- ras et al., 2019; Razavi et al., 2019), audio (Oord et al., 20; Kong al., 2021), point clouds (Yang et al., 2019) and graphs (De Cao & Kipf, 2018). 1)). However, current gener- ative learning frameworks cannot yet simultaneously satisfy three key requirements, often needed for their wide adop- tion in real-world problems. These requirements include (i) high-quality sampling, (ii) mode coverage and sample diver- sity, and (iii) fast and computationally inexpensive sampling. For example, most current works in image synthesis focus high-quality generation. However, mode coverage and data Figure 1: diversity are important for better representing minorities and for reducing the negative social im- pacts of generative models. Additionally, applications such as interactive image editing or real-timo speech synthesis require fast sampling. Here, we identify the challenge posed by these requirements as the generative learning trilemma, since existing models usually compromise between them.
High Quality Samples
Generative Adversarial Networks
Denoising Diffusion Modkls
Fast Sampling
Mode Covernge Diversity
Variational Autoencoders, Normalicing Flows
Generative learning trilemma.
Fig. summarizes how mainstream generative frameworks tackle the trilemma. Generative adver- sarial networks (GANs) (Goodfellow et al., 2014; Brock et al., 2018) generate high-quality samples rapidly, but they have poor mode coverage (Salimans et al., 2016; Zhao al., 2018). Conversely,
Work done during internship NVIDIA.

---

Published as a conference paper at ICLR 2022
variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) and normalizing flows (Dinh et al., 2016; Kingma & Dhariwal, 2018) cover data modes faithfully, but they often suffer from low sample quality. Recently, diffusion models -Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021c) have emerged as powerful generative models. They demonstrate surpris- ingly good results in sample quality, beating GANs in image generation (Dhariwal & Nichol, 2021; Ho et al., 2021). They also obtain good mode coverage, indicated by high likelihood (Song et al., 2021b; Kingma et al., 2021; Huang et al., 2021). Although diffusion models have been applied to variety of tasks (Dhariwal & Nichol; Austin et al.; Mittal et al.; Luo & Hu), sampling from them often requires thousands of network evaluations, making their application expensive in practice.
In this paper, we tackle the generative learning trilemma by reformulating denoising diffusion mod- els specifically for fast sampling while maintaining strong mode coverage and sample quality. We investigate the slow sampling issue of diffusion models and we observe that diffusion models com- monly assume that the denoising distribution can be approximated by Gaussian distributions. How- ever, it is known that the Gaussian assumption holds only in the infinitesimal limit of small denoising steps (Sohl-Dickstein et al., 2015; Feller, 1949), which leads to the requirement of a large number of steps in the reverse process. When the reverse process uses larger step sizes (i.e., it has fewer denois- ing steps), we need Intuitively, in image synthesis, the multimodal distribution arises from the fact that multiple plausi- ble clean images may correspond to the same noisy image.
Gaussin multimodal distribution for modeling the denoising distribution.
Inspired by this observation, we propose to parametrize the denoising distribution with expressive multimodal distribution to enable denoising for large steps. In particular, we introduce a novel gen- erative model, termed denoising diffusion GAN, in which the denoising distributions are modeled with conditional GANs. In image generation, we observe that our model obtains sample quality and mode coverage competitive with diffusion models, while taking only as few as two denoising steps, achieving about 2 speed-up in sampling compared to the predictor-correctt et al. (2021c) on CIFAR-10. Compared to traditional GANs, we show that our model significantly outperforms state-of-ther aort----r-tttttttt GANs in sample diversity, while being competitive in sample fidelity.
sampling by Song
In summary, we make the following contributions: i) We attribute the slow sampling of diffusion models to the Gaussian assumption in the denoising distribution and propose to employ complex, multimodal denoising distributions. ii) We propose denoising diffusion GANs, a diffusion model whose reverse process is parametrized by conditional GANs. iii) Through careful evaluations, demonstrate that denoising diffusion GANs achieve several orders of magnitude speed-up up compared to current diffusion models for both image generation and editing. We show that model over- comes the deep generative learning trilemma to a large extent, making diffusion models for the first time applicable interactive, real-world applications at low computational cost.
BACKGROUND
In diffusion models (Sohl-Dickstein et al., 2015; Ho al., 2020), there is a forward process that gradually adds noise to the data Xo ((xo) in T steps with pre-defined variance schedule 3y:
(xx II (x), q(x,/x,-)) (21
((x,;; / 1,8,),
where ((xo) is a data-generating distribution. The reverse denoising process defined by: (x(x = (xx-15 o),
po (x(xT))) p(x)) Il pa px(xx-
where (x, 1) and of are the mean and variance for the denoising model and @ denotes its param-
eters. The goal of training is to maximize the likelihood pe (Xn) Pe (xo:T)dx):T) by maximizing the evidence lower bound (ELBO, L < log pe (xo)). The ELBO can be written as matching the true denoising distribution ( xx)) with the parameterized denoising model po (x+-1 [xx) using:
(x:- Ey(,,) [DRL (g(x;- (x- + C,
(3)
where contains constant terms that are independent of 0 and DKL. denotes the Kullback- (KI.) divergence. The objective above is intractable due to the unavailability of q(x;-1|x,)) Instead,
2

---

Published as a conference paper at ICLR 2022
(()))
((aba)
((2b))
2aa
Marginal Diffused
Data Distributions
True Denoising
Distributions
(()/)s-) (( daizz=X) dz//==-)
Figure 2: Top: The evolution of 1D data distribution ((x)) through the diffusion process. Bottom: The visualization of the true denoising distribution for varying step sizes conditioned
fixed X5- The true denoising distribution for a small step size (i.c., q(x4 X3 X)) close to a Gaussian distribution. However, it becomes more complex and multimodal the step size increases.
Sohl-Dickstein et al. (2015) show that C can be written in an alternative form with tractable distribu- tions (see Appendix A of Ho et al. (2020) for details). Ho et al. (2020) show the equivalence of this form with score-based models trained with denoising score matching (Song & Ermon, 2019; 2020).
Two key assumptions are commonly made in diffusion models: First, the denoising distribution po(x;-1(x,) is modeled with a Gaussian distribution. Second, the number of denoising steps T often assumed be in the order of hundreds to thousands of steps. In this paper, focus discrete-time diffusion models. In continuous-tin diffusion models (Song et al., 2021c), similar assumptions are also made at the sampling time when discretizing time into small timesteps.
3 DENOISING DIFFUSION GANS
We first discuss why reducing the number of denoising steps requires learning a multimodal denois- ing distribution in Sec. 3.1. Then, we present our multimodal denoising model in Sec. 3.2.
3.1 MULTTMODDA DENOISING DISTRIBUTIONS FOR LARGE DENOISING STEPS
As we discussed in Sec. 2, a common assumption in the diffusion model literature is to approximate (xx |x,) with Gaussian distribution. Here, we question when such an approximation is accurate.
The true denoising distribution g(x;11 Xx) can be written as gxx:1 1 x q(x: (xx-1)9(x;-)) us- ing Bayes' rule where q(x, x;-1) is the forward Gaussian diffusion shown in Eq. and ((x,-1) is the marginal data distribution at step t. It can be shown that in two situations the true denoising distribution takes a Gaussian form. First, in the limit of infinitesimal step size 8e, the product in the Bayes' rule is dominated by ((x+ and the reversal of the diffusion process takes an identical functional form as the forward process (Feller, 1949). Thus, when B3 is sufficiently small, since q(x, xx-1) .) is Gaussian, the denoising distribution gxx;-1 is also Gaussian, and the approx- imation used by current diffusion models can be accurate. To satisfy this, diffusion models often have thousands of steps with small 3,. Second, if data marginal ((x)) is Gaussian, the denoising dis- tribution q( x(x,11 |x,) is also a Gaussian distribution. The idea of bringing data distribution g(xo) and consequently ((x,) closer to Gaussian using VAE encoder
recently explored in LSGM (Vah- dat et al., 2021). ).. However, the problem of transforming the data to Gaussian itself is challenging and VAE encoders cannot solve it perfectly. That why LSGM still requires tens hundreds of steps on complex datasets.
In this paper, we argue that when neither of the conditions are met, i.c., when the denoising step is large and the data distribution is non-Gaussian there are no guarantees that the Gaussian assumption on the denoising distribution holds. To illustrate this, in Fig. we visualize the true denoising distribution for different denoising step sizes for multimodal data distribution. We see that as the denoising step gets larger, the true denoising distribution becomes more complex and multimodal.
3.2 MODELING DENOISING DISTRIBUTIONS WITH CONITIONAL GANS
Our goal is to reduce the number of denoising diffusion steps T required in the reverse process of diffusion models. Inspired by the observation above, propose to model the denoising distribution
3

---

Published as a conference paper at ICLR 2022
with an expressive multimodal distribution. Since conditional GANs have been shown to model complex conditional distributions in the image domain (Mirza & Osindero, 2014; Ledig et al., 2017; Isola et al., 2017), adopt them to approximate the true denoising distribution ((x;-1|xx,)
Specifically, our forward diffusion is set up similarly to the diffusion models in Eq. 1 with the main assumption that 7 is assumed to be small (T < 8) and each diffusion step has larger (e. Our training is formulated by matching the conditional GAN generator Pg(X, (x,-1 (xx) and g(x;-1 (x,) using an adversarial loss that minimizes a divergence Dadv per denoising step:
min
Rotx)) Paad
Ipo (x [x,]]]
(4)
where Dadv can be Wasserstein distance, the adversarial training setup (Arjovsky et al., 2017: Goodfellow et al., 2014; Nowozin al., 2016). In this paper, we rely on non-saturating GANs (Goodfellow et al., 2014), that are widely used in successful GAN frameworks such as StyleGANs (Karras al., 2019; 2020b). In this case, Dady takes a special instance of f-divergence called softened reverse KL (Shannon et al., 2020), which is different from the forward KL divergence used in the original diffusion model training in Eq.31.
divergence, or f-divergence depending on
To set up the adversarial training, we denote the time-dependent discriminator as Do(xe-1, Xx, t): RN x RN x R > [0, 1]. with parameters Φ. It takes the N N-dimensional x;-1 and x; as inputs, and decides whether x a plausible denoised version of x,. The discriminator trained by:
min
Eg(xs) -xxx (oogg &D ( (x,-1, Xx, t)] + 1xx(xx1xxxx11 -og(olloooo x,, t)]]], (5)
where fake samples from po (x (x)) are contrasted against real samples from (xx)) The first expectation requires sampling from ((x:-1 (xx) which is unknown. However, we use the identity
x(x,,x,-1) = (xxx) Xx- Xx) (xog(xx)g(xx- xx))(xx(xxx-1) to rewrite the first
expectation in Eq. 5 as:
[o(xx)((x, 1|x, log( Dx t)] = (x(x))(((x 11x0)((x11x,, .)|- log( Do (x,,
Given the discriminator, we train the generator by max Eσ>1 P (x (x.) Dog (D X,, ())] which updates the generator with the non-saturating GAN objective (Goodfellow et al., 2014).
Parametrizing the implicit denoising model: Instead of directly predicting x,-1 in the denoising step, diffusion models (Ho al., 2020) can be interpreted as parameterizing the denoising model by po (xx-1 |xx) == q(x;-1 Xx,, Xg = fo(x,,)) ()) in which first Xo is predicted using the denoising model [x,; 1), and then, x;-1 is sampled using the posterior distribution
Xo) given x, and the predicted xo (See Appendix B for details). The distribution g(x:-1 Xo, xx) is intuitively the distribution over X when denoising from x; towards Xo, and always has a Gaussian form for the diffusion process in Eq. 1, independent of the step size and complexity of the data distribution (see Appendix A for the expression of g(x;-1/X0) Xo. xt)). Similarly, we define (x xx) by:
(x(x;- (x.) = / Pa px(xolx;) |x,, x,) )xx = / p(z)q |x, X, Gx( z, t)),z,
(6)
where (x([)|))) Xx) is the implicit distribution imposed by the GAN generator Gx z, 1) R2N x E. N that outputs xo given x, and an L-dimensional latent variable z ~ p(z) := N O.
Our parameterization has several advantages: Firstly, our Pe (x lxx)) is formulated similar to DDPM (Ho et al., 2020). Thus, we can borrow some inductive bias such as the network struc- ture design from DDPM. The main difference is that, in DDPM, xo is predicted as a deterministic mapping of x,, while in our case X, is produced by the generator with random latent variable z. This is the key difference that allows our denoising distribution po (x(x,-10 (x,- xx) ) to become multimodal and complex in contrast to the unimodal denoising model in DDPM. Secondly, note that for different f's, X₁ has different levels of perturbation, and hence using single network to predict x,-1 directly at different t may be difficult. However, in our case the generator only needs to predict unperturbed Xo and then add back perturbation using q x x Fig. 3 visualizes our training pipeline.
examined training conditional VAB as (x by minimizing ET, Eytt) [DKL (g(x+-1ixx) po (x.-(xxeaa) [x+)]] for small T'. However, conditional VAEs consistently resulted poor generative performance in our early experiments. In this paper, we focus on conditional GANs and we leave the exploration of other expressive conditional generators for po (x;-1 (x.) to future work.
At carly stages,
4

---

Published as a conference paper at ICLR 2022
Advantage over one-shot generator: One natural question for our model is, why not just train GAN that gen- erate samples in one shot using a tra- ditional setup, in contrast our model that generates samples by denoising it- eratively? Our model has several advan- tages over traditional GANs. GANs are known to suffer from training instability and mode collapse (Kodali et al., 2017;
Forward diffusion
at
2
x,
Z0
8:
Real Fake!
D(2 1.
Conditianing
Posterior sampling
20---
Salimans et al., 2016), and some possi- ble reasons include the difficulty of di- rectly generating samples from a com- plex distribution in one-shot, and the
Gi 2,
overfitting issue when the discriminator Figure 3: The training process of denoising diffusion GAN. only looks at clean samples. In contrast, our model breaks the generation process into several con- ditional denoising diffusion steps in which each step is relatively simple to model, due to the strong conditioning on X,, Moreover, the diffusion process smoothens the data distribution (Lyu, 2012), making the discriminator less likely to overfit. Thus, we expect our model to exhibit better training stability and mode coverage. We empirically verify the advantages over traditional GANs in Sec. 5.
RELATED WORK
Diffusion-basee models (Sohl-Dickstein et al., 2015; Ho et al., 2020) learn the finite-time reversal of a diffusion process, sharing the idea of learning transition operators of Markov chains with Goyal al. (2017); Alain et al. (2016); Bordes al. (2017). Since then, there have been number of im- provements and alternatives to diffusion models. Song et al. (2021c) generalize diffusion processes continuous time, and provide a unified view of diffusion models and denoising score matching (Vincent, 2011; I; Song & Ermon, 2019). Jolicoeur-Martincauu et al. (2021b) add an auxiliary adversar- ial loss to the main objective. This is fundamentally different from ours, as their auxiliary adversarial loss only acts as an image enhancer, and they do not use latent variables; therefore, the denoising distribution is still a unimodal Gaussian. Other explorations include introducing alternative noise distributions in the forward process (Nachmani et al., 2021), jointly optimizing the model and noise schedule (Kingma et al., 2021) and applying the model in latent spaces (Vahdat al., 2021).
One major drawback of diffusion or score-based models is the slow sampling speed due to large number of iterative sampling steps. To alleviate this issue, multiple methods have been proposed, including knowledge distillation (Luhman & Luhman, 2021), learning an adaptive noise schedule -aanRooman et al., 2021). introducing non-Markovian diffusion processes (Song et al., 2021a; Kong & Ping, 2021), and using better SDE solvers for continuous-timg models (Jolicoeur- Bealiceer--Martinauu 2021a). In particular, Song et al. (2021a) uses Xn sampling as a crucial ingredient to their method, but their denoising distribution is still Gaussian. These methods either suffer from significant degradation in sample quality, or still require many sampling steps as we demonstrate in Sec. 5.
al.,
Among variants of diffusion models, Gao al. (2021) have the closest connection with our method. They propose to model the single-step denoising distribution by a conditional energy-based model (EBM), sharing the high-level idea of using expressive denoising distributions with us. However, they motivate their method from the perspective of facilitating the training of EBMs. More impor- tantly, although only a few denoising steps are needed, expensive MCMC has to be used to sample from each denoising step, making the sampling process slow with 1880 network evaluations. Im- ageBART (Esser et al., 2021a) explores modeling the denoising distribution of diffusion process on discrete latent space with auto-regressive model per step in few denoising steps. However, the auto-regressive structure of their denoising distribution still makes sampling slow.
Since model is trained with adversarial loss, our work is related to recent advances in improving the sample quality and diversity of GANs, including data augmentation (Zhao et al., 2020; Karras et al., 2020a), consistency regularization (Zhang et al., 2020; Zhao et al., 2021) and entropy regu- larization (Dieng et al., 2019). In addition, the idea of training generative models with smoothed distributions is also discussed in Meng et al. (2021a) for auto-regressive models.
5

---

Published as a conference paper at ICLR 2022
5 EXPERIMENTS
In this section, we evaluate our proposed denoising diffusion GAN for the image synthesis problem. We begin with briefly introducing the network architecture design, while additional implementation details are presented in Appendix C. For our GAN generator, we adopt the NCSN+ architecture from Song et al. (2021c) which has a U-net structure (Ronneberger et al., 2015). The conditioning X, is the input of the network, and time embedding is used to ensure conditioning on t. We let the latent variable control the normalization layers. In particular, we replace all group normalization layers (Wu & He, 2018) in NCSN++ with adaptive group normalization layers in the generator, similar to Karras al. (2019); Huang & Belongie (2017), where the shift and scale parameters in group normalization are predicted from using a simple multi-layer fully-connected network.
5.1 OVERCOMING THE GENERATIVE LEARNING TRILEMMA
One major highlight of our model is that it excels at all three criteria in the generative learning trilemma. Here, carefully evaluate our model's ' performances on sample fidelity, sample diversity and sampling time, and benchmark against comprehensive list of models on the CIFAR- 10 dataset.
Evaluation criteria: We adopt the commonly used Fréchet inception distance (FID) (Heusel al., 2017) and Inception Score (IS) (Salimans et al., 2016) for evaluating sample fidelity. We use the training set as a reference to compute the FID, following common practice in the literature (see Ho et al. (2020); Karras et al. (2019) an example). For sample diversity, we use the improved recall score from Kynkäänniemi et al. (2019), which an improved version of the original precision and recall metric proposed by Sajjadi et al. (2018). It is shown that an improved recall score reflects how the variation in the generated samples matches that in the training set (Kynkäänniemí et al., 2019). For sampling time, we use the number of function evaluations (NFE) and the clock time when generating a batch of 100 images on a V100 GPU.
We present our quantitative results in Table 1. We observe that sample quality is com- petitive among the best diffusion models and GANs. Although some variants of diffusion models obtain better IS and FID, they require large number of function evaluations to generate samples (while we use only 4 denoising steps). For example, our sampling time is about 2000 faster than the predictor-corrector sampling by Song et al. (2021c) and ~20x faster than FastDDPM (Kong & Ping, 2021). I). Note that diffusion models can produce samples in fewer steps while trading off the sample quality. To better benchmark our method against existing diffusion models, we plot the FID score versus sampling time of diffusion models by varying the number of denoising steps (or the er- ror tolerance for continuous-time models) in Figure 4. The figure clearly shows the advantage of our model compared to previous diffusion models. When comparing our model to GANs, we observe that only StyleGAN2 with adaptive data augmentation has slightly better sample quality than ours. However, from Table 1, we see that GANs have limited sample diversity, their recall scores below 0.5. In contrast, our model obtains a significantly better recall score, even higher than several advanced iikeihhodd-based models, and competitive among diffusion models. We show qualitative samples of CIFAR- 10 in Figure 5. In summary, our model simultaneously excels at sample quality, sample diversity, and sampling speed and tackles the generative learning trilemma by a large extent.
Results:
11
Benwweing
(pand)
PaelIDPM Kong &e ling)
RDIM (Sing al)
RDPM (13a dtal)
Trodubility Nlow (Seng et
Scon SDD, PC (tag
SNGAN-DGOW (homii StyAGAA
StyleGANN wi (eera
'2'
Sampling Time (6)
Figure 4: Sample quality vs sampling time trade-off.
Figure CIFAR-10 qualitative samples.

---

Published as a conference paper at ICLR 2022
Table I: Results for unconditional generation on CIFAR-I0.
Model
ISt
FID
Recallf
NFE
Time
Denoising Diffusion GAN (ours), T=4
9.63
3.75
0.57
4
0.21
DDPM (Ho al., 2020)
9,46
3.21
0.57
1000
80.5
NCSN (Song & Emonn 2019)
2019)
8.87
25.3
1000
107.9
Adversarial DSS Doliiceur-Marttnennn et al,, 2021b)
6.10
1000
Likelihood SDE (Song et al., 2021b)
Score SDE (VE) (Song 2021c)
9.89
2.20
0.59
2000
423.2
Score SDE (VP) (Song et ul., 2021c)
9.68
2.41
0.59
2000
421.5
Probability Flow (VP) (Song et al., 2021c)
0.57
140
50.9
1.SGM (Vahdat
2021)
9.87
2.10
0.61
147
44.5
DDIM, T=50 (Song, et al., 202
8,78
4.67
0.53
50
4.01
FastDDMM, T=50 (Kong & Ping. 2021)
3,41
0,56
50
4.01
Recovery EBM (Gao
2021)
8.30
Improved DDPM (Nichol & Dhariwal, 2021)
2.90
4000
VDM (Kingma et al., 2021)
4,00
1000
UDM (Kim al., 2021)
2000
D3PMs (Austin et al., 2021)
8.56
7.34
1000
Gotta Go Fast (Jolicouu--arttnn Martineau et al., 202
2.44
180
DDPM Distillation (Luhman & Luhmun, 2021)
9.36
0.51
SNGAN (Miyato et al., 2018)
8.22
21.7
0.44
SNGAN+DGiter (Ansari
2021)
9.35
9.62
0.48
25
1.98
AutoGAA (Gong et al., 2019)
8.60
12.4
0.46
TransGAN (Jiang et al., 2021)
9.02
9.26
StyleGAN2 w/o ADA (Karras at., 2020s)
0.41
0.04
StyleGAN2 w/ ADA (Karras et al. 2020a)
9.83
2.92
0.49
0.04
StyleGAN2 w/ Diffaug (Zhao et ul., 2020)
9.40
5.79
0.42
0.04
Glow (Kingma & Dhariwal, 2018)
3.92
48.9
PixeICNN (Oord et al., 2016b)
4.60
65.9
1024
NVAE (Vahdat Kautz, 2020)
23.5
0.51
0.36
IGEBM (Du & Mordatch, 2019)
6.112
40.6
60
VAEBM (Xiao al., 2021)
8.43
12.2
0.53
8.79
5.2 ABLATION STUDIES
Here, we provide additional insights into our model by performing ablation studies.
Number of denoising steps: In the first part of Table 2, we study the effect of using a different number of denoising steps (T). Note that T = corresponds to training an unconditional GAN, as the conditioning x, contains almost no information about Xg- We observe that T leads to significantly worse results with low sample diversity, indicated by the low recall score. This confirms the benefits of breaking generation into several denoising steps, especially for improving the sample diversity. When varying T'> 1, observe that 7 gives the best results, whereas there is a slight degrade in performance for larger T. We hypothesize that we may require a significantly need conditional GAN for each denoising step.
higher capacity accommodate larger T,
Diffusion as data augmentation: Our model shares some similarities with recent work on applying data augmentation to GANs (Karras et al., 2020a; Zhao et al., 2020). To study the effect of perturbing inputs, we train a one-shot GAN with our network structure following the protocol in (Zhao et al., 2020) with the forward diffusion process data augmentation. The result, presented in the second group of Table 2. is significantly worse than our model, indicating that our model is not equivalent augmenting data before applying the discriminator.
Parametrization for pe (xx)= We study two alternative ways to parametrize the denoising distribution for the same T setting. Instead of letting the generator produce estimated samples of we set the generator to directly output denoised samples x,-1 without posterior sampling (direct denoising), or output the noise e, that perturbs a clean image to produce x; (noise generation). Note that the latter case is closely related to most diffusion models where the network deterministically predicts the perturbation noise. Table 2, we show that although these alternative parametrizations work reasonably well, our main parametrization outperforms them by large margin.
Importance of latent variable: Removing latent variables z converts our denoising model to a unimodal distribution. In the last line of Table 2, study our model's performance without any latent variables z. We see that the sample quality is significantly worse, suggesting the importance of multimodal denoising distributions. In Figure 8, visualize the effect of latent variables by showing samples of Po (Xn x1), where x, is a fixed noisy observation. We see that while the majority of information in the conditioning x, preserved, the samples are diverse due to the latent variables.

---

Published as a conference paper at ICLR 2022
Table 2: Ablation studies on CIFAR-10.
Table 3: Mode coverage on StackedMNIST.
Model Variants
1St
FID
Recullf
Model
Modest
KL,
T=1 1
8,93
14.6
0.19
VEEGAN (Srivastava et al.)
762
2.173
9,80
4.08
0.54
PacGAN (Lin et al.)
992
0.277
T=4
9.63
3.75
0.57
PresGAN (Dieng et al.)
1000
0.115
T=8
9.43
4.36
0.56
InclusiveGAN (Yu ct al.)
997
0.200
One-shot w/ aug
8,96
13.2
0.25
StyleGAN2 (Karras et al.)
940
0.424
Direct denoising
9.10
6.03
0,53
Adv. DSM JolicoeurMaattneau Martineau al))
1000
1.49
Noise generation
8.79
8.04
0.52
VAEBM (Xiao et al.)
1000
0.087
No latent variable
8.37
20.6
0.42
Denoising Diffusion GAN (ours)
1000
0.071
5.3 ADDITIONAL STUDIES
Besides the recall score in Table 1, we also evaluate the mode coverage of our model on the popular 25-Gaussians and StackedMNISTT. The 25-Gaussians dataset is a 2-D toy dataset, generated by mixture of 25 two-dimensional Gaussian distributions, arranged in grid. We train our denoising diffusion GAN with 4 denoising steps and compare it to other models in Figure 6. We observe that the vanilla GAN suffers severely from mode collapse, and while techniques like WGAN-GP (Gulrajani et al., 2017) improve mode coverage, the sample quality is still limited. In contrast, our model covers all the modes while maintaining high sample quality. We also train a diffusion model and plot the samples generated by 100 and 500 denoising steps. We see that diffusion models require large number of steps to maintain high sample quality.
Mode Coverage:
Real samples
GAN
WGAN-GP
Ours, T-4
DDPM, T-500
DDPM, T-100 L.
$ g
2 *0 @
2 x
iM A 5 40 4 vg:
Figure 6: Qualitative results on the 25 15-Gaussians datasct.
StackMNIST contains images generated by randomly choosing 3 MNIST images and stacking them along the RGB channels. Hence, the data distribution has 1000 modes. Following the setting of Lin al. (2018), we report the number of covered modes and the KI. divergence from the categorical distribution over 1000 categories of generated samples to true data in Table 3. We observe that model covers all modes faithfully and achieves the lowest KL compared to GANs that are specifi- cally designed for better mode coverage StyleGAN2 that is known to have the best sample quality.
Training Stability: We discuss the training stability of our model in Appendix D.
High Resolution Images: We train our model on datasets with larger images, including CelebA- HQ (Karras al., 2018) and LSUN Church (Yu al., 2015) at 256 256px resolution. We report FID on these two datasets in Table 4 and Similar to CIFAR-10, our model obtains competitive sample quality among the best diffusion models and GANs. In particular, in LSUN Church, our model outperforms DDPM and ImageBART (see Figure and Appendix E for samples). Although, reflected by the FID score.
some GANs perform better on this dataset, their mode coverage
Stroke-based image synthesis: Recently, Meng et al. (2021b) propose an interesting application of diffusion models stroke-based generation. Specifically, they perturb a stroke painting by the for- ward diffusion process, and denoise it with a diffusion model. The method is particularly promising because only requires training an unconditional generative model on the target dataset and does not require training images paired with stroke paintings like GAN-based methods (Sangkloy et al., 2017; Park et al., 2019). We apply our model to stroke-based image synthesis and show qualita- tive results in Figure 9. The generated samples are realistic and diverse, while the conditioning in the stroke paintings is faithfully preserved. Compared to Meng et al. (2021b), our model enjoys a 1100x speedup in generation, as it takes only 0.16s to generate one image at 256 resolution 181s for Meng et al. (2021b). This experiment confirms that our proposed model enables the application of diffusion models interactive applications such image editing.
8

---

Published as a conference paper at ICLR 2022
Table Generative results on Cel:bA-HQ-256
Table 5: Generative results on LSUN Church 256
Model
FID_
Model
FID4
Denoising Diffusion GAN (ours)
7.64
Denoising Diffusion GAN (ours)
5.25
Score SDE (Song et al., 2021c)
7.23
LSGM (Vahdat et al., 2021)
7.22
DDPM (Ho al., 2020)
7.89
UDM (Kim al., 2021)
7.16
ImageBART (Esser et al., 2021a)
7.32
NVAE (Vahdat Kautz, 2020)
29.7
Gotta Go Fast (Jolicocur- Martineau et al.)
25.67
VAEBM (Xiao et al., 2021)
20.4
NCP-VAR (Aneja al,, 2021)
PGGAN (Karras et al., 2018))
6.42
PGGAN (Karras al., 2018)
8.03
StyleGAN (Karras et al., 2019)
4.21
Adv. LAE (Pidhorskyi et al., 2020)
19.2
StyleGAN2 (Karras et al., 2020b)
3.86
VQ-GAN (Esser et al., 2021b)
10.2
CIPS (Anokhin al., 2021)
2.92
DC-AE al., 2021)
15.8
GRRBT
Figure 7: Qualitative results on CelebA-HQ 256 and LSUN Church Outdoor 256.
Additional results: Additional qualitative visualizations are provided in Appendix E, F, and G.
Figure 8: Multi-modality of denoising dis- tribution given the same noisy observation. Left: clean image Xy and perturbed image X1. Right: Three samples from po (x)
Figure 9: Qualitative results stroke-based synthesis. Top row: stroke paintings. Bottom two rows: generated samples corresponding the stroke painting (best seen when zoomed in).
CONCLUSIONS
Deep generative learning frameworks still struggle with addressing the generative learning trilemma. Diffusion models achieve particularly high-quality and diverse sampling. However, their slow sam- pling and high computational cost do not yet allow them be widely applied in real-world appli- cations. In this paper, we argued that one of the main sources of slow sampling in diffusson models the Gaussian assumption in the denoising distribution, which is justified only for very small de- noising steps. To remedy this, we proposed denoising diffusion GANs that model each denoising step using a complex multimodal distribution, enabling us take large denoising steps. In extensive experiments, we showed that denoising diffusion GANs achieve high sample quality and diversity competitive to the original diffusion models, while being orders of magnitude faster at sampling. Compared to traditional GANs, our proposed model enjoys better mode coverage and sample diver- sity. Our denoising diffusion GAN overcomes the generative learning trilemma to large extent, allowing diffusion models to be applied real-world problems with low computational cost.

---

Published as a conference paper at ICLR 2022
7 ETHICS AND REPRODUCIBILITY STATEMENT
Generating high-quality samples while representing the diversity in the training data faithfully has been a daunting challenge in generative learning. Mode coverage and high diversity are key require- ments for reducing biases in generative models and improving the representation of minorities in population. While diffusion models achieve both high sample quality and diversity, their expensive sampling limits their application in many real-world problems. Our proposed denoising diffusion GAN reduces the computational complexity of diffusion models to an extent that allows these mod- els to be applied in practical applications our model can help with reducing the negative social impacts of existing generative models that fall short in capturing the data diversity.
low cost. Thus, foresee that in the long term,
We evaluate our model using public datasets, and therefore this work does not involve any human subject evaluation or new data collection.
We currently provide experimental details in the appendices with a detailed list of hyperparameters and training settings. To aid with reproducibility further, will release our source code publicly in the future with instructions to reproduce our results.
Reproducibility Statement:
REFERENCES
Guillaume Alain, Yoshua Bengio, Li Yao, Jason Yosinski, Eric Thibodeau-Laufer, Saizheng Zhang, and Pascal Vincent. Gsns: generative stochastic networks. Information and Inference: A Journal of the IMA, 5(2):210-249, 2016.
Jyoti Aneja, Alexander Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. In Neural Information Processing Systems, 2021.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Ko- rzhenkov. Image generators with condtiionlly-indeeeeneent pixel synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Abdul Fatir Ansari, Ming Liang Ang, and Harold Soh. Refining deep generative models via dis- criminator gradient flow. In International Conference on Learning Representations, 2021.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, 2017
Jacob Austin, Daniel Johnson, Jonathan Ho. Danny Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, arXiv preprint arXiv:2107.0300 2021.
Florian Bordes, Sina Honari, and Pascal Vincent. Learning to generate samples from noise through infusion training. arXiv preprint arXiv:1703.0669: 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rXiv:1809.11099 2018.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint 100. 2018.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2/05.05233, 2021.
Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias. Prescribed generative adversarial networks. arXiv preprint arXiv:1910.04300 2019.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations, 2016.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, 2019.
10

---

Published as a conference paper at ICLR 2022
Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjöm Ommer. Imagebart: Bidirec- tional context with multinomial diffusion for autoregressive image synthesis. arXiv preprint TrXii:2/08808827, 2021a.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis, Proceedings of the IEEE conference on computer vision and pattern recognition, 2021b.
W Feller. On the theory of stochastic processes, with particular reference to applications. In Proceed- ings of the [First] Berkeley Symposium on Mathematical Statistics and Probability, pp. 403-432. University of California Press, 1949.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representa- tions, 2021.
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, 2014.
Anirudh Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Variational walkback: Learning a transition operator as a stochastic recurrent net. arXiv preprint 1rXiv:1711.0228. 2017.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans, arXiv preprint arXiv:1704.0000 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. In Advances in neural information processing systems, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106.15282. 2021.
Chin-Wei Huang, Jae Hyun Lim, and Aaron Courville. A variational perspective on diffusion-based generative models and score matching. arXiv preprint arXiv:2106.02866, 2021.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers can make one strong gan. arXiv preprint arXiv:2102.07074 2021.
Alexia Jolicoeur-Martineau Ke Li, Rémi Piché-Tailllfer, Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2115. /4080, 2021a.
Alexia Jolicoeur-Martineuu Martineau, Rémi Fiihlefail,eee, Ioannis Mitliagkas, and Remi Tachet des Combes. Adversarial score matching and improved sampling for image generation. In Inter- national Conference Learning Representations, 2021b.

---

Published as a conference paper at ICLR 2022
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in neural information processing systems, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Ana- lyzing and improving the image quality of stylegan. Proceedings of the IEEE conference on computer vision and pattern recognition, 2020b.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. Score matching model for unbounded data score. arXiv preprint arXiv:2106.05527, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In The International Con- ference on Learning Representations, 2014.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint urXiv:2107.006:: 2021.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in neural information processing systems, 2018.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. convergence and stability of gans. arXiv preprint arXiv: arXiv: 1705.07255, 2017.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference Learning Representations, 2021.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. arXiv preprint arXiv: (ttv:/904.006 009.06691, 2019.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. Advances in neural information processing systems, 2018.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 16088:1608.0399 2016.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.022 023888 2021.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2021.
Siwei Lyu. Interpretation and generalization of score matching. arXiv preprint 1rXiv:1205.263 2012.
12

---

Published as a conference paper at ICLR 2022
Chenlin Meng, Jiaming Song, Yang Song, Shengjia Zhao, and Stefano Ermon. Improved autore- gressive modeling with distribution smoothing. arXiv preprint arXiv:2103.15089 2021a.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Im- age synthesis and editing with stochastic differential equations. arXiv preprint urXiv:2108.0107. 2021b.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on Machine Learning, 2018.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint orXiv:1411. 141441814,, 2014.
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103.66091, 2021.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 2018.
Eliya Nachmani, Robin San Roman, and Lior Wolf. Non gaussian denoising diffusion models. arXiv preprint arXiv:2106.075:: 075822 2021.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint 0rX1v:2102.09672, 2021.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan:
Training generative neural samplers using variational divergence minimization. In Advances neural information processing systems, 2016.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXív: 1009.0399, 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. International Conference Machine Learning, 2016b.
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with gpatially-adaptive normalization. In Proceedings of the IEEE conference computer vision and pattern recognition, 2019.
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2021.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2020.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high- images with vq-vae-2. In Advances in neural information processing systems, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference deep generative models. In International Conference on Machine Learn- ing, 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention. Springer, 2015.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, Assessing generative models via precision and recall. arXiv preprint arXiv:1806.000 80680003... 2018.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 2016.
13

---

Published as a conference paper at ICLR 2022
Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion mod- els. arXiv preprint arXiv:2 020000 2021.
Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Scribbler: Controlling deep image synthesis with sketch and color. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2017.
Matt Shannon, Ben Poole, Soroosh Mariooryad, Tom Bagby, Eric Battenberg, David Kao, Daisy Stanton, and RJ Skerry- Ryan. Non-saturating gan training divergence minimization. arXiv preprint arXiv:2010.08029 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn ing, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in neural information processing systems, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.09011 2020.
Yang Song, Conor Durkan, lain Murray, and S. Ermon. Maximum likelihood training of score-based diffusion models. arXiv preprint 1xXiv:2001.0925:: 2021b.
Yang Song, Jascha Sohl-Dicksteii Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- fional Conference on Learning Representations, 2021c.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Infor- mation Processing Systems, 2017.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in neural information processing systems, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in neural information processing systems, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1674 2011.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision, 2018
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between varia- tional autoencoders and energy-based models. In International Conference on Learning Repre- sentations, 2021.
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 506 003660 2015.
14

---

Published as a conference paper at ICLR 2022
Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz. Inclusive gan: Improving data and minority coverage in generative models. arXiv preprint arXiv:2004.03355, 2020.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In International Conference on Learning Representations, 2020.
Richard Zhang. Making convolutional networks shift-invariant again. In International conference on Machine Learning. 2019.
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. Bias and generalization deep generative models: An empirical study. In Advances in neural information processing systems, 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Advances in neural information processing systems, 2020.
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Im- proved consistency regularization for gans, In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
15

---

Published as a conference paper at ICLR 2022
A DERIVATION FOR THE GAUSSIAN POSTERIOR
Ho et al. (2020) provide a derivation for the Gaussian posterior distribution. We include here for completeness. Consider the forward diffusion process in Eq. which we repeat here:
T
II ((xx(xx-1), (=1
g(xx(xx-1)
N(x; / 1,
Due to the Markov property of the forward process, we have the marginal distribution of x given the initial clean data Xo:
(xx x0) = (xx;; /,x0, (1 Tt)
(8)
where denote oz 1- 3r and 0r II Os- Applying Bayes' rule, can obtain the forward process posterior when conditioned on Xo:
((x-1 (2, xo) q(,, X,- Xo (x,))(X,11110)) (z)
q(x, X
(x
(9)
(xxe|xo)
where the second equation follows from the Markov property of the forward process. Since three terms in Eq. 9 are Gaussians, the posterior q Xp) is also Gaussian distribution, and it can be written as
(xx X,, xo) N (x (x,, xo); 3.1)
(10)
with mean (Xe, x6) and variance Be. Plugging the expressions Eq. and Eq. 8 into Eq. 10, we obtain
18- (x, xo)
Var(1 ā(-1)
1 01-1 Br
Xo
X, and B ==
(11)
αe
at
ae
after minor simplifications.
B PARAMETRIZATION OF DDPM
In Sec. 3.2, mention that the parametrization of the denoising distribution for current diffusion models such as Ho et al. (2020) can be interpreted as pe (x x,) == qxx,11 Xo fo fx( t)). However, such parametrization
not explicitly stated in Ho al. (2020) but discussed by Song et al. (2021a). To avoid possible confusion, here we show that the parametrization of Ho et al. (2020) is equivalent to what we describe See 3.2.
Ho et al. (2020) train noise prediction network co t) which predicts the noise that perturbs data Xg to Xx, and a sample from pe (x- (x:) is obtained as (see Algorithm of Ho et al. (2020))
1 Var
1 aq
Xi
V1 ā Eq (x(:
o.Z,
(12)
where (O, except for the last denoising step where 0, and o, / is the standard deviation of the Gaussian posterior distribution in Eq. 11.
Firstly, notice that predicting the perturbation noise (x(x;, I) is equivalent to predicting Xp- We know that x, is generated by adding .N(0, I) noise as:
X
ax50 + v1 ārE,
Hence, after predicting the noise with cx t) we can obtain a prediction of xy using:
1
Xa
(x+ vI (x.o
(13)
Vo
16

---

Published as a conference paper at ICLR 2022
Next, we can plug the expression for x in Eq. 13 into the mean of the Gaussian posterior distribution in Eq. 11, and we have
1
(x,, xo)
fit
Xt,
(x, -vI
(xc, (a)
(14)
Vās
1 Otz (x(: v Cit
X1
(15)
V 0xg
after simplifications. Comparing this with Eq. 12, we observe that Eq. 12 simply corresponds to sampling from the Gaussian posterior distribution. Therefore, although Ho al. (2020) use an alternative re-parametrization, their denoising distribution can still be equivalently interpreted as pe (x |(x := g(x;-1|xx X Xo fx(,,, t)), i.e, first predicting X using the time-dependent denoising model. and then sampling x,-1 using the posterior distribution g(x;-1 Xx, Xo) given X² and the predicted Xg-
C EXPERIMENTAL DETAILS
In this section, we present our experimental settings in detail.
C.I NETWORK STRUCTURE
Generator: Our generator structure largely follows the U-net structure (Ronneberger et al., 2015) used in NCSN+ (Song et al., 2021c), which consists of multiple ResNet blocks (He et al., 2016) and Attention blocks (Vaswani et al., 2017). Hyper-parameters for the network design, such as the number of blocks and number of channels, are reported in Table 6. We follow the default settings in Song et al. (2021c) for other network configurations not mentioned in the table, including Swish activation function, upsampling and downsampling with anti anti-aliasing based Finite Impulse Response (FIR) (Zhang, 2019), re- re-scaling all skip connections by V using residual block design from BigGAN (Brock et al., 2018) and incorporating progressive growing architectures (Karras al., 2020b). See Appendix H of Song et al. (2021c) for more details on these configurations.
We follow Ho et al. (2020) and use sinusoidal positional embeddings for conditioning on integer time steps. The dimension for the time embedding is 4 x the number of initial channels presented in Table 6. Contrary to previous works, we did not find the use of Dropout helpful in our case.
The fundamental difference between our generator network and the networks of previous diffusion models is that our generator takes an extra latent variable z as input. Inspired by the success of Style- GANs, we provide z-conditioning to the NCSN++ architecture using mapping networks, introduced in StyleGAN (Karras et al., 2019). We group normalization (GN) layers in the network with adaptive group normalization (AdaGN) layers to allow the input of latent variables. The latent variable z is first transformed by fully- fully-connected network (called mapping network). and then the resulting embedding vector, denoted by w, is sent every AdaGN layer. Each AdaGN layer contains one fully-connected layer that takes w as input, and outputs the per-channel shift and scale parameters for the group normalization. The network's s feature maps are then subject to affine transformations using these shift and scale parameters of the AdaGN layers. The mapping network and the fully-connected layer in AdaGN are independent of time steps t. as we found no extra benefit in incorporating time embeddings in these layers. Details about latent variables are also presented in Table 6.
(O for all experiments. We replace all the
Discriminator: We design our time-dependent discriminator with a convolutional network with ResNet blocks, where the design of the ResNet blocks is similar to that of the generator. The dis- criminator tries to discriminate real and fake x;-1, conditioned on x, and t. The time conditioning is enforced by the same sinusoidal positional embedding as in the generator. The conditioning is enforced by concatenating X2 and X;-1 1 as the input to the discriminator. We use LeakyReLU activations with negative slope 0.2 for all layers. Similar to Karras et al. (2020b), we mini- batch standard deviation layer after all the ResNet blocks. We present the exact architecture of discriminators in Table 7.
17

---

Published as a conference paper at ICLR 2022
Table 6: Hyper-parameter for the generator network.
CIFARIO
CelebA-HQ
I.SUN Church
# of ResNet blocks per scale
Initial # of channels
128
64
64
Channel multiplier for each scale
(1, 2, 2. 2)
(1, 1, 2, 2, 4,4)
(1, 2, 2,,4,,
Scale of attention block Latent Dimension # of latent mapping layers
16
16
16
100
100
100
3
Latent embedding dimension
256
256
256
Table 7: Network structures for the discriminator. The number on the right indicates the number of channels in each residual block.
CelebA-HQ and LSUN Church
CIFAR-10
conv2d, 128
1 conv2d, 128
ResBlock down, 256
ResBlock, 128
ResBlock down, 512
ResBlock down, 256
ResBlock down, 512
ResBlock down, 512
ResBlock down, 512
ResBlock down, 512
ResBlock down, 512
minibatch std layer
ResBlock down, 512
Global Sum Pooling
minibatch std layer
FC layer scalar
Global Sum Pooling
FC layer scalar
C.2 TRAINING
Diffusion Process: For all datasets, set the number of diffusion steps to be 4. In order to com- pute 3r per step, we use the discretization of the continuous- extension of the process described in Eq. I, which is called the Variance Preserving (VP) SDE by Song al. (2021c). We compute 3e based on the continuous-time diffusion model formulation, as it allows us to ensure that variance schedule stays the same independent of the number of diffusion steps. Let's define the normalized time variable by 1' = which normalizes E (1,2..... T) to [0, 1]. The variance function of VP SDE is given by:
0²((') = BBnnt 0.518mm Pmn)r"
with the constants 9max 20 and Bmin 0.1. Recall that sampling from hth step in the forward diffusion process can be done with g(xe|Xo) N N(xx; Vaxxo. (1-ā,)I). We compute 3 by solving ā₁ =²)
0² ( 1 02 =
31 = - 04 =
=
-mil -
Bute) 1 72
(16)
for t E f1,2.... This choice of 3g values corresponds to equidistant steps in time according to VP SDE. Other choices are possible, but we did not explore them.
Objective: We train our denoising diffusion GAN with the following adversarial objective:
min
1xx (o + Bpoo logg - Dap (xt-1, Xt, 1))]
max
By (xxx))g(xx (x)](xx(xx=.)))
( (log(
[))[]
where the outer expectation denotes ancestral sampling from q(Xo, X4- xx) and po (xx (x,) is our implicit GAN denoising distribution.
18

---

Published as a conference paper at ICLR 2022
Table 8: Optimization hyper-parameters.
CIFAR10
CelebA-HQ
LSUN Church
Initial learning rate for discriminator
4
10-4
10-4
Initial learning rate for generator
1.6
10-4
1.6 10-4
10
Adam optimizer 11
0.5
0.5
0.5
Adam optimizer [3
0.9
0.9
0.9
EMA
0.9999
0.999
0,999
Batch size
256
32
64
# of training iterations # of GPUs
400k
750k
600k
4
8
8
Similar to Ho et al. (2020), during training we randomly sample an integer time step / E 11,2,3,4 for each datapoint batch. Besides the main objective, we also add an R₁ regularization term (Mescheder et al., 2018) to the objective for the discriminator. The Ri term is defined as
R1 (Ø)
2 We -
||| x, Do |
2
where y is the coefficient for the regularization. We use = 0.05 for CIFAR-10, and y 1 for CelebA-HQ and LSUN Church. Note that the R regularization a gradient penalty that encourages the discriminator stay smooth and improves the convergence of GAN training (Mescheder et al., 2018).
Optimization: We train models using the Adam optimizer (Kingma & Ba, 2015). We cosine learning rate decay (Loshchilov & Hutter, 2016) for training both the generator and discriminator. Similar to Ho al. (2020); Song al. (2021c); Karras et al. (2020a), observe that applying exponential moving average (EMA) on the generator is crucial to achieve high performance. We summarize the optimization prper-parameter in Table 8.
We train our models on CIFAR-10 using V100 GPUs. On CelebA-HQ and LSUN Church we 8 V100 GPUs. The training takes approximately 48 hours CIFAR-10, and 180 hours CelebA-HQ and LSUN Church.
C.3 EVALUATION
When evaluating IS, FID and recall score, we use 50k generated samples for CIFAR-10 and LSUN Church, and 30k samples for CelebA-HQ (since the CelebA HQ dataset contains only 30k samples).
When evaluating sampling time, we use models trained on CIFAR-10 and generate batch of 100 samples. We benchmark the sampling time machine with single V100 GPU. We use Pytorch 1.9.0 and CUDA 11.0.
C.4 ABLATION STUDIES
Here introduce the settings for the ablation study Sec. 5.2. We observe that training requires larger number of training iterations when T' is larger. As a result, we train the model for each 7' until the FID score does increase any further. The number training iteration is 200k for T 1 and T = 2, 400k for T = 4 and 600k for T 8. We use the same network structures and optimization settings as in the main experiments.
For the data augmentation baseline, we follow the differenniablee data augmentation pipeline in Zhao al. (2020). particular, for every (real fake) image in the batch, we perturbed by sampling from random timestep at the diffusion process (except the last diffusion step where the informa- tion of data is completely destroyed). We find the results insensitive to the number of possible perturbation levels (i.e, the number of steps in the diffusion process), and report the result using a diffusion process with 4 steps. Since the perturbation by the diffusion process is differentiable due to the re- parametrization trick (Kingma & Welling, 2014), and generator with the perturbed samples. See Zhao et al. (2020) for a detailed explanation for the training pipeline.
train both the discriminator
19

---

Published as a conference paper at ICLR 2022
For the experiments alternative parametrizations, use T for the diffusion process and keep other settings the same as in the main experiments.
For the experiment on training model without latent variables, similar to the main experiments, the generator takes the conditioning X, as its input, and the time conditioning is still enforced by the time embedding. However, the AdaGN layers replaced by plain GN layers, such that latent variable needed, and the mapping network for z removed. Other settings follow the main experiments.
C.5 TOY DATA AND STACKEDMNIST
For the 25- Gaussian toy dataset, both our generator and discriminator have 3 fully-connected layers each with 512 hidden units and LeakyReLU activations (negative slope of 0.2). We enforce both and by concatenation with the input. We use the Adam optimizer with 10-4 the conditioning 4 for both the generator and discriminator. The batch size 512, and we train learning rate of 10
the model for 50k iterations.
Our experimental settings for StackedMNIST are the same as those for CIFAR-10, 10, except that we train the model for only 150k iterations.
D TRAINING STABILITY
In Fig. 10, plot the discriminator loss for different time steps in the diffusion process when 4. We observe that the training of our denoising diffusion GAN is stable and we do not see any explosion in loss values, is sometimes reported for other GAN methods such as Brock et al. (2018). The stability might be attributed to two reasons: First, the conditioning on x for both generator and discriminator provides strong signal. The generator is required to generate few plausible samples given X; and the discriminator requires classifying them. The x, conditioning keeps the discriminator and generator smooth distributions, distributions of fake and real samples closer to each other (Lyu, 2012). As we can see from Fig- 10, the discriminator loss for > O) is higher than t = 0 (the last denoising step). Note that > 0 corresponds to training the discriminator on noisy images, and in this case true and generator distributions are closer to each other, making the discrimination harder and hence resulting in higher discriminator loss. We believe that such a property prevents the discriminator from overfitting, which leads to better training stability.
balance. Second, we are training the GAN on relatively the diffusion process is known as a smoothening process that brings the
E ADDITIONAL QUALITATIVE RESULTS
We show additional qualitative samples of CIFAR-10, CelebA-HQ and LSUN Church Outdoor in Figure 11, Figure 12 and Figure 13, respectively.
F ADDITIONAL VISUALIZATION FOR po (xooXx))
In Figure 14 and Figure 15, we show visualizations of samples from (x for different t. Note that except for po(Xo ). the samples from po (Xx xx) do not need to be sharp, they only intermediate outputs of the sampling process. The conditioning is less preserved as the perturbation in x, increases, and in particular XT (x4 in our example) contains almost information of clean data Xg-
G NEAREST NEIGHBOR RESULTS
In Figure 16 and Figure 17, we show the nearest neighbors in the training dataset, corresponding to few generated samples, where the nearest neighbors are computed using the feature distance of pre-trained VGG network (Simonyan & Zisserman, 2014). We observe that the nearest neighbors are significantly different from the samples, suggesting that our models generalize well.
20

---

Published as a conference paper at ICLR 2022
1.4
12
D loss
0.8
0.6
100000
200000
300000
400000
Training iterations
Figure 10: The discriminator loss per denoising step during training.
21

---

Published as a conference paper at ICLR 2022
Figure 11: Additional qualitative samples on CIFAR-10.
22

---

Published as a conference paper at ICLR 2022
IT
CC
EOU
O
VO
Figure 12: Additional qualitative samples on CelebA-HQ.
23

---

Published as a conference paper at ICLR 2022
shu tsck
Figure 13: Additional qualitative samples on LSUN Church Outdoor.
24

---

Published as a conference paper at ICLR 2022
X X1 X2
X3
(xolx)))
Po (Xolx2)
Po (Xolx3)
(x(xx)
X0
X
X2
X3
(xolxx)
Po (xolx2)
Pe (x))
Pe (xolx4)
Figure 14: Visualization of samples from (xo(xx))) for different on CelebA-HQ. For each exam- ple, the top row contains x, from diffusion process steps, where xo sample from the dataset. The bottom rows contain 3 samples from (x x.) for different t's.
25

---

Published as a conference paper at ICLR 2022
X
X1
X2
X3
(x(x(()))
Po (xolx)))
Po (Xolx3)
po (xolx
X
X1
X2
X3
X4
(xol((
Po (xoox()))
(xooxx))
Pe (x)(xx))
Figure 15: Visualization of samples from po (x.) for different on LSUN Church. For each example, the top row contains X, from diffusion process steps, where Xo sample from the dataset. The bottom rows contain 3 samples from pe (xo xx) for different l's.
26

---

Published as a conference paper at ICLR 2022
Figure 16:
CIFAR-10 nearest neighbors in VGG feature space. Generated samples in the leftmost column, and training set nearest neighbors are in the remaining columns.
27

---

Published as a conference paper at ICLR 2022
2OH
Figure 17:
CelebA-HQ nearest neighbors in the VGG feature space. Generated samples are in the leftmost column, and training set nearest neighbors are in the remaining columns.
28


Reviews:
[{"review_id": "ke5CCKykMDQ", "reviewer": "Reviewer_FKLx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors start by showing with a toy example that the denoising conditional\ndistribution can be heavily multi-modal for large diffusion steps, highlighting\nthe limitations of the typical Gaussian model, which can only be assumed for\nvery small steps, thus leading to an expensive multi-step sampling process. The\nauthors overcome this by using a conditional GAN to model the denoising\ndistribution under large steps, and experimentally show that this can indeed\nmodel very multimodal distributions. This allows the reversal of the diffusion\nprocess using much fewer steps than in competing methods, achieving efficient\nsampling speeds while maintaining the high sample quality and mode coverage of\ndiffusion models. While taking this idea to the extreme would lead to the\nstandard one-step GAN, interestingly the authors show that their method\noutperforms standard GANs, and the provided justification is that in the\nmulti-step approach the distribution mapping in intermediate steps is less\naggressive than direct Gaussian-to-data-distribution, and thus easier to learn.\n\nTo the best of my understanding I couldn't identify any serious weakness in \nthis work.", "summary_of_the_paper": "This paper proposes to use a conditional GAN as a more expressive model of the\ndenoising step in a denoising diffusion model, allowing to reverse the diffusion\n(sampling) much more efficiently, while maintaining the high quality of diffusion\nmodels.", "summary_of_the_review": "I think the proposed idea is clever and very well executed. Experimental results\nvalidate the method in terms of the triple sample quality / mode coverage /\nsampling efficiency trade-off. Experiments demonstrate their conditional\ngenerator exhibits the strong multi-modality required for the large reverse\ndiffusion steps.  I expect this hybrid between conditional GANs and diffusion\nmodels to become an important player in generative modeling.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "KzE_dfiqxcx", "reviewer": "Reviewer_27CK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths** \n\nI think this work and the proposed combination of adversarial and likelihood-based training is a natural and novel extension of DDPMs towards faster sampling times. The main idea of modeling the conditional transition probabilities with more informative models is simple (which is a good thing). Most design decisions are justified by ablation experiments (e.g., type of parameterization, number of diffusion steps, deterministic vs. stochastic generators, etc.). Experiments with the 25-Gaussian and StackedMNIST datasets show the improved mode coverage over GAN models.\n\nThe results on CIFAR-10, CelebA-HQ and LSUN Churches are competitive with previous state-of-the-art methods. Furthermore, I really like the underlying motivation of the work that interactive image processing or speech synthesis require very fast sampling, and I agree that this is a pressing issue when dealing with diffusion models. \n\n**Weaknesses**\n\n- Fig. 1 is a good motivator, but a bit imprecise: (non-reweighted) diffusion models also belong to the class of likelihood-based models, as do autoregressive generative models. The latter are slow, but cover the modes and provide high quality.\n- How important is the choice of variance schedule? If I understand correctly, the work relies exclusively on the schedule given in equation (16). How does the schedule affect controlled synthesis, e.g., stroke-based image synthesis?\n- The paper should address previous work that also attempts to solve the trilemma by reducing the number of steps required for a trained DDPM model, e.g., DDIM [1] or GGF [2], and compare with them.\n- It should also be shown that the coverage of modes on more complex, high-resolution data (such as LSUN Churches) is still as good as the results on the Toy data suggest.\n- Do the results presented in Tab. 2 (ablation studies on CIFAR-10) also hold for high-resolution (i.e. $256 \\times 256$) datasets?\n\n**Mixed Comments:**\n\n- Is there evidence that diffusion models actually provide (approximate) SOTA diversity? My impression is that they can be tuned either to very good likelihoods (as in [3]) or to high quality samples (as in [4]), but currently cannot do both at the same time. The best performing diffusion models in terms of _quality_ are based on the reweighted objective introduced in [5], which sacrifices likelihood interpretation for better sample quality.\n- Figure 9 nicely demonstrates controlled image synthesis/modification. However, compared to [6] only 4 steps provide less fine-grained control over how much content is obtained.\n- How does the model perform on other typical but more complex synthesis tasks such as LSUN Cats or class-conditional generation of ImageNet?\n- Just out of pure interest: Have you tried using the UNet as a discriminator as well (similar to [7])? How important is the exact implementation of the discriminator architecture?\n\n__References__\n\n- [1]: Song, J., Meng, C., Ermon, S.: Denoising Diffusion Implicit Models\n- [2]: Jolicoeur-Martineau, Alexia, et al. \"Gotta Go Fast When Generating Data with Score-Based Models.\"\n- [3]: Kingma, Diederik P., et al. \"Variational diffusion models.\" \n- [4]: Dhariwal, Prafulla, and Alex Nichol. \"Diffusion models beat gans on image synthesis.\" \n- [5]: Ho, Jonathan, Ajay Jain, and Pieter Abbeel. \"Denoising diffusion probabilistic models.\"\n- [6]: Meng, Chenlin, et al. \"Sdedit: Image synthesis and editing with stochastic differential equations.\" \n- [7]:  Schonfeld, Edgar, Bernt Schiele, and Anna Khoreva. \"A u-net based discriminator for generative adversarial networks.\" \n", "summary_of_the_paper": "This paper introduces _Denoising Diffusion GANs (DDGANs)_ to solve the so-called \"generative learning trilemma\" by training a generative model that achieves high-quality sampling, fast sampling times, and at the same time, high mode coverage. The main intuition of the paper is to combine GANs, which have been shown to achieve both high-quality synthesis and very fast sampling times, and DDPMs, which provide high-quality sampling and good mode coverage. In contrast to previous work on diffusion models that rely on Gaussian transition probabilities of the reverse forward process, conditional GANs are used in this work to allow flexible, multimodal denoising distributions in the reverse process and thus much shorter sequence lengths. Experiments show that the proposed approach indeed achieves higher mode coverage than conventional GAN models while maintaining high sample quality within 4 to 8 diffusion steps. The model can also be used for stroke-based image synthesis, with significant improvements in sampling times compared to previous approaches.", "summary_of_the_review": "In summary, the paper provides a nice combination of adversarial and likelihood-based training and, in my opinion, represents a natural and novel extension of DDPMs towards faster sampling times. The experiments validate the claim that denoising diffusion GANs can indeed help overcome the \"generative learning trilemma\", and most of the ablation studies justify the design choices made in the paper. I think this is a good paper worth publishing at ICLR 2022, and I am willing to increase my score if my remaining questions and concerns are adequately answered.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GElhwquO7Oj", "reviewer": "Reviewer_XcPr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, this paper proposes a good idea to solve the trilemma in generative learning, which accelerating the sampling in denoising diffusion models by using conditional GANs. The idea is clean with the experimental support. There are several minor issues in the required addressing as listed below.\n\n1) It is desirable if the conditional GANs in the proposed model can be tested with different settings. It is currently non-saturating GANs. Does the proposed approach also work better with advanced conditional GANs? \n2) The results of FID should be specified with the evaluation dataset or both (FID (train) and FID (test)).\n3) It is desirable to make the diffusion model continuously improve as the iteration increases. However, when T is greater than 4, the performance decreases.\n4) The format of references should be consistent (e.g., with or without abbreviations).", "summary_of_the_paper": "The diffusion models or score matching-based models have achieved great success in generating high fidelity samples with diverse modes. However, the sampling requires tremendous iterations, which hinders the practical usages. This paper proposes to break the Gaussian assumption in denoising steps by parametrizing it with a multimodal conditional GAN. The time for sampling can be 2000 times faster than the original diffusion model while keeping competitive quality and diversity.", "summary_of_the_review": "This paper solves an important problem in denoising diffusion models by parametrizing denoising with a multimodal conditional GAN. Experimental results show that the proposed model achieves good sample quality, mode coverage and speed without struggling in the trilemma. The minor issues are in the experiments. It is encouraged to test and discuss different setting. Therefore, I recommend to accept this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cb9-59gImPc", "reviewer": "Reviewer_6zYL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Presentation\n- The paper is easy to read and understand. It contains many helpful figures, qualitative examples and quantitative comparisons summarized in tables. I especially like Fig. 14 which is very useful in understanding the amount of information within each of the diffusion scales, since this is sometimes very difficult to judge from a visualization of the noisy states themselves.\n- The paper motivates the problem it is addressing well through the mental image of a trilemma.\n\nLiterature/Background\n- The related work section covers many relevant works and provides a good classification of the current work within the existing literature.\n- It contains the necessary background material on diffusion models to be sufficiently self-contained.\n\nApproach and evaluation\n- The approach is relatively simple and provides good results. Experiments convincingly demonstrate that the approach indeed strikes a good balance of the three desired goals.\n- A number of ablations analyze different aspects of the model in more detail.\n- Besides helping with mode coverage, the approach brings other advantages over a pure GAN training such as discriminator regularization helping against overfitting on small datasets, and potentially more control over the generation process (as also evidenced by the stroke-based synthesis).\n\nAdditional Comments\n- Loss weighting: For diffusion models, the weight applied to the objective from different steps of the diffusion process can be critical. In fact, a change in this weighting has been one of the break-throughs in [Ho, 2020] that lead to the current success of this class of models. The choice of this weight in the presented approach is not discussed. Is it simply the same for all timesteps? Did you experiment with this?\n- Choice of generator architecture: While it is nice to see that a similar model as used for diffusion models also works as the conditional generator in the presented setting, Tab. 2 suggests that it is actually a bad choice for an unconditional/one-step generator. Have you experimented with different choices there? Does it help to use a different architecture for the first (unconditional) step compared to the other steps?", "summary_of_the_paper": "The goal of the paper is to develop a generative model which achieves three desirable goals at the same time: High-quality samples, data/mode coverage and fast sampling times. While GANs have been associated with high-quality samples and fast sampling times, they are also known for their tendency to ignore modes of the data. On the other hand, diffusion models have recently been demonstrated to achieve high-quality samples and good likelihood estimation, which can indicate good mode coverage. However, diffusion models tend to be slow to generate samples due to their iterative nature. Thus, the paper proposes to combine diffusion models with GANs to combine their complementary advantages regarding mode coverage and sampling times (together with their shared ability to produce high-quality samples).\n\nThe main reason why diffusion models require a lot of iterations is because they model each of the steps of the reverse diffusion process with a Gaussian distribution, which is only valid for very small timesteps/a large number of steps. Thus, instead of using a network to estimate parameters of a Gaussian reverse process, the authors propose to model steps of the reverse process with a GAN. In theory, a GAN could directly model the data distribution and therefore solve the task in a single step, where the approach would be equivalent to a GAN and therefore suffer again from mode collapse. The main hope is therefore that by using a few diffusion steps, such an approach is still relatively fast compared to pure diffusion models while also having a good mode coverage.\n\nExperiments evaluate sample quality, sampling speed and mode coverage. The main evaluation on CIFAR-10 indeed demonstrates a favorable combination of sample quality according to FID and IS, sampling speed and data coverage as measured by Recall. Additional experiments analyze the effect of various ablations and demonstrate an application in stroke-based image synthesis.", "summary_of_the_review": "Diffusion models come with a lot of desired benefits such as high quality samples, data coverage and stable training. However, their slow sampling speeds limit their applicability and the problem of improving this aspect with a new generative model is very relevant. The idea of combining diffusion models with the benefits of GANs by modeling the reverse process with conditional GANs is plausible. The main question is whether such an approach can really combine the advantages of the different models. To that end, the experiments are quite convincing and I think the paper provides a positive answer to that question, which will make it significant for future research. Thus, I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper modifies DPMs by replacing the denoising L2 losses with GANs to learn the iterative denoising process. This leads to excellent results using a small number of refinement steps. In some sense, this also takes away one of the key advantages of DPMs over GANs, which is DPMs minimize a well-defined objective function. Nevertheless, the results are convincing, but not spectacular. I am not convinced that we should continue to report training FID on CIFAR-10. I would have like to see class-conditional ImageNet results. Also, it is not clear whether the proposed technique provides additional gains on top of SoTA GANs. Overall, I recommend acceptance as a spotlight.
Decision: Accept (Spotlight)