Year: 2021

Abstract:
A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously addressing three key requirements including: high sample quality, mode coverage, and fast sampling. We call the challenge imposed by these requirements the generative learning trilemma, as the existing models often trade some of them for others. Particularly, denoising diffusion models have shown impressive sample quality and diversity, but their expensive sampling does not yet allow them to be applied in many real-world applications. In this paper, we argue that slow sampling in these models is fundamentally attributed to the Gaussian assumption in the denoising step which is justified only for small step sizes. To enable denoising with large steps, and hence, to reduce the total number of denoising steps, we propose to model the denoising distribution using a complex multimodal distribution. We introduce denoising diffusion generative adversarial networks (denoising diffusion GANs) that model each denoising step using a multimodal conditional GAN. Through extensive evaluations, we show that denoising diffusion GANs obtain sample quality and diversity competitive with original diffusion models while being 2000× faster on the CIFAR-10 dataset. Compared to traditional GANs, our model exhibits better mode coverage and sample diversity. To the best of our knowledge, denoising diffusion GAN is the first model that reduces sampling cost in diffusion models to an extent that allows them to be applied to real-world applications inexpensively.

text: 1 INTRODUCTION
1
In the past decade, a plethora of deep generative models has been developed for various domains such as images (Karras et al., 2019; Razavi et al., 2019), audio (Oord et al., 2016a; Kong et al., 2021), point clouds (Yang et al., 2019) and graphs (De Cao & Kipf, 2018). However, current generative learning frameworks cannot yet simultaneously satisfy three key requirements, often needed for their wide adoption in real-world problems. These requirements include (i) high-quality sampling, (ii) mode coverage and sample diversity, and (iii) fast and computationally inexpensive sampling. For example, most current works in image synthesis focus on high-quality generation. However, mode coverage and data
diversity are important for better representing minorities and for reducing the negative social impacts of generative models. Additionally, applications such as interactive image editing or real-time speech synthesis require fast sampling. Here, we identify the challenge posed by these requirements as the generative learning trilemma, since existing models usually compromise between them.
Fig. 1 summarizes how mainstream generative frameworks tackle the trilemma. Generative adversarial networks (GANs) (Goodfellow et al., 2014; Brock et al., 2018) generate high-quality samples rapidly, but they are known for poor mode coverage (Salimans et al., 2016; Zhao et al., 2018). Conversely, variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) and normalizing flows (Dinh et al., 2016; Kingma & Dhariwal, 2018) cover data modes faithfully, but they often suffer from low sample quality.
Recently, diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021c) have emerged as powerful generative models. Diffusion models define a forward diffusion process that maps data to noise by gradually perturbing the input data. Data generation is achieved using a parametrized reverse process that performs iterative denoising, starting from random noise. Diffusion models demonstrate surprisingly good results in sample quality, beating GANs in image generation (Dhariwal & Nichol, 2021; Ho et al., 2021). They also demonstrate good mode coverage, indicated by high likelihood (Song et al., 2021b; Kingma et al., 2021; Huang et al., 2021). Diffusion models have already been applied to a variety of generation tasks (Dhariwal & Nichol, 2021; Austin et al., 2021; Mittal et al., 2021; Luo & Hu, 2021). However, sampling from them often requires thousands of network evaluations, making their application to real-world problems expensive.
In this paper, we tackle the generative learning trilemma by reformulating denoising diffusion models specifically for fast sampling while maintaining strong mode coverage and sample quality. We investigate the slow sampling issue of diffusion models and we observe that diffusion models commonly assume that the denoising distribution can be approximated by Gaussian distributions. However, it is known that the Gaussian assumption holds only in the infinitesimal limit of small denoising steps (Sohl-Dickstein et al., 2015; Feller, 1949), which leads to the requirement of a large number of steps in the reverse process. When the reverse process uses larger step sizes (i.e., it has fewer denoising steps), we need a non-Gaussian multimodal distribution for modeling the denoising distribution. Intuitively, in image synthesis, the multimodal distribution arises from the fact that multiple plausible clean images may correspond to the same noisy image.
Inspired by this observation, we propose to parametrize the denoising distribution with an expressive multimodal distribution to enable denoising for large steps. In particular, we introduce a novel generative model, termed as denoising diffusion GAN, in which the denoising distributions are modeled with conditional GANs. In image generation, we observe that our model obtains sample quality and mode coverage competitive with diffusion models, while taking only as few as two denoising steps, achieving about 2000× speed-up in sampling compared to the predictor-corrector sampling by Song et al. (2021c) on CIFAR-10. Compared to traditional GANs, we show that our model significantly outperforms state-of-the-art GANs in sample diversity, while being competitive in sample fidelity.
In summary, we make the following contributions: i) We attribute the slow sampling of diffusion models to the Gaussian assumption in the denoising distribution and propose to employ complex, multimodal denoising distributions. ii) We propose denoising diffusion GANs, a diffusion model whose reverse process is parametrized by conditional GANs. iii) Through careful evaluations, we demonstrate that denoising diffusion GANs achieve several orders of magnitude speed-up compared to current diffusion models for both image generation and editing. We show that our model overcomes the deep generative learning trilemma to a large extent, making diffusion models for the first time applicable to interactive, real-world applications at a low computational cost.

heading: 2 BACKGROUND
text: In diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020), there is a forward process that gradually adds noise to the data x0 ∼ q(x0) in T steps with pre-defined variance schedule βt:
q(x1:T |x0) = ∏ t≥1 q(xt|xt−1), q(xt|xt−1) = N (xt; √ 1− βtxt−1, βtI), (1)
where q(x0) is a data-generating distribution. The reverse denoising process is defined by: pθ(x0:T ) = p(xT ) ∏ t≥1 pθ(xt−1|xt), pθ(xt−1|xt) = N (xt−1;µθ(xt, t), σ2t I), (2)
where µθ(xt, t) and σ2t are the mean and variance for the denoising model and θ denotes its parameters. The goal of training is to maximize the likelihood pθ(x0) = ∫ pθ(x0:T )dx1:T , by maximizing the evidence lower bound (ELBO, L ≤ log pθ(x0)). The ELBO can be written as matching the true denoising distribution q(xt−1|xt) with the parameterized denoising model pθ(xt−1|xt) using:
L = − ∑ t≥1 Eq(xt) [DKL (q(xt−1|xt)‖pθ(xt−1|xt))] + C, (3)
where C contains constant terms that are independent of θ and DKL denotes the Kullback-Leibler (KL) divergence. The objective above is intractable due to the unavailability of q(xt−1|xt). Instead,
2
Sohl-Dickstein et al. (2015) show that L can be written in an alternative form with tractable distributions (see Appendix A of Ho et al. (2020) for details). Ho et al. (2020) show the equivalence of this form with score-based models trained with denoising score matching (Song & Ermon, 2019; 2020).
Two key assumptions are commonly made in diffusion models: First, the denoising distribution pθ(xt−1|xt) is modeled with a Gaussian distribution. Second, the number of denoising steps T is often assumed to be in the order of hundreds to thousands of steps. In this paper, we focus on discrete-time diffusion models. In continuous-time diffusion models (Song et al., 2021c), similar assumptions are also made at the sampling time when discretizing time into small timesteps.

heading: 3 DENOISING DIFFUSION GANS
text: We first discuss why reducing the number of denoising steps requires learning a multimodal denoising distribution in Sec. 3.1. Then, we present our multimodal denoising model in Sec. 3.2.

heading: 3.1 MULTIMODAL DENOISING DISTRIBUTIONS FOR LARGE DENOISING STEPS
text: As we discussed in Sec. 2, a common assumption in the diffusion model literature is to approximate q(xt−1|xt) with a Gaussian distribution. Here, we question when such an approximation is accurate. The true denoising distribution q(xt−1|xt) can be written as q(xt−1|xt) ∝ q(xt|xt−1)q(xt−1) using Bayes’ rule where q(xt|xt−1) is the forward Gaussian diffusion shown in Eq. 1 and q(xt−1) is the marginal data distribution at step t. It can be shown that in two situations the true denoising distribution takes a Gaussian form. First, in the limit of infinitesimal step size βt, the product in the Bayes’ rule is dominated by q(xt|xt−1) and the reversal of the diffusion process takes an identical functional form as the forward process (Feller, 1949). Thus, when βt is sufficiently small, since q(xt|xt−1) is a Gaussian, the denoising distribution q(xt−1|xt) is also Gaussian, and the approximation used by current diffusion models can be accurate. To satisfy this, diffusion models often have thousands of steps with small βt. Second, if data marginal q(xt) is Gaussian, the denoising distribution q(xt−1|xt) is also a Gaussian distribution. The idea of bringing data distribution q(x0) and consequently q(xt) closer to Gaussian using a VAE encoder was recently explored in LSGM (Vahdat et al., 2021). However, the problem of transforming the data to Gaussian itself is challenging and VAE encoders cannot solve it perfectly. That is why LSGM still requires tens to hundreds of steps on complex datasets.
In this paper, we argue that when neither of the conditions are met, i.e., when the denoising step is large and the data distribution is non-Gaussian, there are no guarantees that the Gaussian assumption on the denoising distribution holds. To illustrate this, in Fig. 2, we visualize the true denoising distribution for different denoising step sizes for a multimodal data distribution. We see that as the denoising step gets larger, the true denoising distribution becomes more complex and multimodal.

heading: 3.2 MODELING DENOISING DISTRIBUTIONS WITH CONITIONAL GANS
text: Our goal is to reduce the number of denoising diffusion steps T required in the reverse process of diffusion models. Inspired by the observation above, we propose to model the denoising distribution
with an expressive multimodal distribution. Since conditional GANs have been shown to model complex conditional distributions in the image domain (Mirza & Osindero, 2014; Ledig et al., 2017; Isola et al., 2017), we adopt them to approximate the true denoising distribution q(xt−1|xt). Specifically, our forward diffusion is set up similarly to the diffusion models in Eq. 1 with the main assumption that T is assumed to be small (T ≤ 8) and each diffusion step has larger βt. Our training is formulated by matching the conditional GAN generator pθ(xt−1|xt) and q(xt−1|xt) using an adversarial loss that minimizes a divergence Dadv per denoising step:
min θ ∑ t≥1 Eq(t) [Dadv(q(xt−1|xt)‖pθ(xt−1|xt))] , (4)
whereDadv can be Wasserstein distance, Jenson-Shannon divergence, or f-divergence depending on the adversarial training setup (Arjovsky et al., 2017; Goodfellow et al., 2014; Nowozin et al., 2016). In this paper, we rely on non-saturating GANs (Goodfellow et al., 2014), that are widely used in successful GAN frameworks such as StyleGANs (Karras et al., 2019; 2020b). In this case, Dadv takes a special instance of f-divergence called softened reverse KL (Shannon et al., 2020), which is different from the forward KL divergence used in the original diffusion model training in Eq. 31.
To set up the adversarial training, we denote the time-dependent discriminator as Dφ(xt−1,xt, t) : RN × RN × R→ [0, 1], with parameters φ. It takes the N -dimensional xt−1 and xt as inputs, and decides whether xt−1 is a plausible denoised version of xt. The discriminator is trained by:
min φ ∑ t≥1 Eq(xt) [ Eq(xt−1|xt)[−log(Dφ(xt−1,xt, t)] + Epθ(xt−1|xt)[−log(1−Dφ(xt−1,xt, t))] ] , (5)
where fake samples from pθ(xt−1|xt) are contrasted against real samples from q(xt−1|xt). The first expectation requires sampling from q(xt−1|xt) which is unknown. However, we use the identity q(xt,xt−1) = ∫ dx0q(x0)q(xt,xt−1|x0) = ∫ dx0q(x0)q(xt−1|x0)q(xt|xt−1) to rewrite the first expectation in Eq. 5 as:
Eq(xt)q(xt−1|xt)[− log(Dφ(xt−1,xt, t))] = Eq(x0)q(xt−1|x0)q(xt|xt−1)[− log(Dφ(xt−1,xt, t))]. Given the discriminator, we train the generator by maxθ ∑ t≥1 Eq(xt)Epθ(xt−1|xt)[log(Dφ(xt−1,xt, t))], which updates the generator with the non-saturating GAN objective (Goodfellow et al., 2014).
Parametrizing the implicit denoising model: Instead of directly predicting xt−1 in the denoising step, diffusion models (Ho et al., 2020) can be interpreted as parameterizing the denoising model by pθ(xt−1|xt) := q(xt−1|xt,x0 = fθ(xt, t)) in which first x0 is predicted using the denoising model fθ(xt, t), and then, xt−1 is sampled using the posterior distribution q(xt−1|xt,x0) given xt and the predicted x0 (See Appendix B for details). The distribution q(xt−1|x0,xt) is intuitively the distribution over xt−1 when denoising from xt towards x0, and it always has a Gaussian form for the diffusion process in Eq. 1, independent of the step size and complexity of the data distribution (see Appendix A for the expression of q(xt−1|x0,xt)). Similarly, we define pθ(xt−1|xt) by:
pθ(xt−1|xt) := ∫ pθ(x0|xt)q(xt−1|xt,x0)dx0 = ∫ p(z)q(xt−1|xt,x0 =Gθ(xt, z, t))dz, (6)
where pθ(x0|xt) is the implicit distribution imposed by the GAN generatorGθ(xt, z, t) : RN×RL× R→ RN that outputs x0 given xt and an L-dimensional latent variable z ∼ p(z) := N (z;0, I). Our parameterization has several advantages: Firstly, our pθ(xt−1|xt) is formulated similar to DDPM (Ho et al., 2020). Thus, we can borrow some inductive bias such as the network structure design from DDPM. The main difference is that, in DDPM, x0 is predicted as a deterministic mapping of xt, while in our case x0 is produced by the generator with random latent variable z. This is the key difference that allows our denoising distribution pθ(xt−1|xt) to become multimodal and complex in contrast to the unimodal denoising model in DDPM. Secondly, note that for different t’s, xt has different levels of perturbation, and hence using a single network to predict xt−1 directly at different t may be difficult. However, in our case the generator only needs to predict unperturbed x0 and then add back perturbation using q(xt−1|xt,x0). Fig. 3 visualizes our training pipeline. 1At early stages, we examined training a conditional VAE as pθ(xt−1|xt) by minimizing∑T t=1 Eq(t) [DKL (q(xt−1|xt)‖pθ(xt−1|xt))] for small T . However, conditional VAEs consistently resulted in poor generative performance in our early experiments. In this paper, we focus on conditional GANs and we leave the exploration of other expressive conditional generators for pθ(xt−1|xt) to future work.
Advantage over one-shot generator: One natural question for our model is, why not just train a GAN that can generate samples in one shot using a traditional setup, in contrast to our model that generates samples by denoising iteratively? Our model has several advantages over traditional GANs. GANs are known to suffer from training instability and mode collapse (Kodali et al., 2017; Salimans et al., 2016), and some possible reasons include the difficulty of directly generating samples from a complex distribution in one-shot, and the overfitting issue when the discriminator
only looks at clean samples. In contrast, our model breaks the generation process into several conditional denoising diffusion steps in which each step is relatively simple to model, due to the strong conditioning on xt. Moreover, the diffusion process smoothens the data distribution (Lyu, 2012), making the discriminator less likely to overfit. Thus, we expect our model to exhibit better training stability and mode coverage. We empirically verify the advantages over traditional GANs in Sec. 5.

heading: 4 RELATED WORK
text: Diffusion-based models (Sohl-Dickstein et al., 2015; Ho et al., 2020) learn the finite-time reversal of a diffusion process, sharing the idea of learning transition operators of Markov chains with Goyal et al. (2017); Alain et al. (2016); Bordes et al. (2017). Since then, there have been a number of improvements and alternatives to diffusion models. Song et al. (2021c) generalize diffusion processes to continuous time, and provide a unified view of diffusion models and denoising score matching (Vincent, 2011; Song & Ermon, 2019). Jolicoeur-Martineau et al. (2021b) add an auxiliary adversarial loss to the main objective. This is fundamentally different from ours, as their auxiliary adversarial loss only acts as an image enhancer, and they do not use latent variables; therefore, the denoising distribution is still a unimodal Gaussian. Other explorations include introducing alternative noise distributions in the forward process (Nachmani et al., 2021), jointly optimizing the model and noise schedule (Kingma et al., 2021) and applying the model in latent spaces (Vahdat et al., 2021).
One major drawback of diffusion or score-based models is the slow sampling speed due to a large number of iterative sampling steps. To alleviate this issue, multiple methods have been proposed, including knowledge distillation (Luhman & Luhman, 2021), learning an adaptive noise schedule (San-Roman et al., 2021), introducing non-Markovian diffusion processes (Song et al., 2021a; Kong & Ping, 2021), and using better SDE solvers for continuous-time models (Jolicoeur-Martineau et al., 2021a). In particular, Song et al. (2021a) uses x0 sampling as a crucial ingredient to their method, but their denoising distribution is still a Gaussian. These methods either suffer from significant degradation in sample quality, or still require many sampling steps as we demonstrate in Sec. 5.
Among variants of diffusion models, Gao et al. (2021) have the closest connection with our method. They propose to model the single-step denoising distribution by a conditional energy-based model (EBM), sharing the high-level idea of using expressive denoising distributions with us. However, they motivate their method from the perspective of facilitating the training of EBMs. More importantly, although only a few denoising steps are needed, expensive MCMC has to be used to sample from each denoising step, making the sampling process slow with ∼180 network evaluations. ImageBART (Esser et al., 2021a) explores modeling the denoising distribution of a diffusion process on discrete latent space with an auto-regressive model per step in a few denoising steps. However, the auto-regressive structure of their denoising distribution still makes sampling slow.
Since our model is trained with adversarial loss, our work is related to recent advances in improving the sample quality and diversity of GANs, including data augmentation (Zhao et al., 2020; Karras et al., 2020a), consistency regularization (Zhang et al., 2020; Zhao et al., 2021) and entropy regularization (Dieng et al., 2019). In addition, the idea of training generative models with smoothed distributions is also discussed in Meng et al. (2021a) for auto-regressive models.

heading: 5 EXPERIMENTS
text: In this section, we evaluate our proposed denoising diffusion GAN for the image synthesis problem. We begin with briefly introducing the network architecture design, while additional implementation details are presented in Appendix C. For our GAN generator, we adopt the NCSN++ architecture from Song et al. (2021c) which has a U-net structure (Ronneberger et al., 2015). The conditioning xt is the input of the network, and time embedding is used to ensure conditioning on t. We let the latent variable z control the normalization layers. In particular, we replace all group normalization layers (Wu & He, 2018) in NCSN++ with adaptive group normalization layers in the generator, similar to Karras et al. (2019); Huang & Belongie (2017), where the shift and scale parameters in group normalization are predicted from z using a simple multi-layer fully-connected network.

heading: 5.1 OVERCOMING THE GENERATIVE LEARNING TRILEMMA
text: One major highlight of our model is that it excels at all three criteria in the generative learning trilemma. Here, we carefully evaluate our model’s performances on sample fidelity, sample diversity and sampling time, and benchmark against a comprehensive list of models on the CIFAR-10 dataset.
Evaluation criteria: We adopt the commonly used Fréchet inception distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) for evaluating sample fidelity. We use the training set as a reference to compute the FID, following common practice in the literature (see Ho et al. (2020); Karras et al. (2019) as an example). For sample diversity, we use the improved recall score from Kynkäänniemi et al. (2019), which is an improved version of the original precision and recall metric proposed by Sajjadi et al. (2018). It is shown that an improved recall score reflects how the variation in the generated samples matches that in the training set (Kynkäänniemi et al., 2019). For sampling time, we use the number of function evaluations (NFE) and the clock time when generating a batch of 100 images on a V100 GPU.
Results: We present our quantitative results in Table 1. We observe that our sample quality is competitive among the best diffusion models and GANs. Although some variants of diffusion models obtain better IS and FID, they require a large number of function evaluations to generate samples (while we use only 4 denoising steps). For example, our sampling time is about 2000× faster than the predictor-corrector sampling by Song et al. (2021c) and ∼20× faster than FastDDPM (Kong & Ping, 2021). Note that diffusion models can produce samples in fewer steps while trading off the sample quality. To better benchmark our method against existing diffusion models, we plot the FID score versus sampling time of diffusion models by varying the number of denoising steps (or the error tolerance for continuous-time models) in Figure 4. The figure clearly shows the advantage of our model compared to previous diffusion models. When comparing our model to GANs, we observe that only StyleGAN2 with adaptive data augmentation has slightly better sample quality than ours. However, from Table 1, we see that GANs have limited sample diversity, as their recall scores are below 0.5. In contrast, our model obtains a significantly better recall score, even higher than several advanced likelihood-based models, and competitive among diffusion models. We show qualitative samples of CIFAR-10 in Figure 5. In summary, our model simultaneously excels at sample quality, sample diversity, and sampling speed and tackles the generative learning trilemma by a large extent.

heading: 5.2 ABLATION STUDIES
text: Here, we provide additional insights into our model by performing ablation studies.
Number of denoising steps: In the first part of Table 2, we study the effect of using a different number of denoising steps (T ). Note that T = 1 corresponds to training an unconditional GAN, as the conditioning xt contains almost no information about x0. We observe that T = 1 leads to significantly worse results with low sample diversity, indicated by the low recall score. This confirms the benefits of breaking generation into several denoising steps, especially for improving the sample diversity. When varying T >1, we observe that T =4 gives the best results, whereas there is a slight degrade in performance for larger T . We hypothesize that we may require a significantly higher capacity to accommodate larger T , as we need a conditional GAN for each denoising step.
Diffusion as data augmentation: Our model shares some similarities with recent work on applying data augmentation to GANs (Karras et al., 2020a; Zhao et al., 2020). To study the effect of perturbing inputs, we train a one-shot GAN with our network structure following the protocol in (Zhao et al., 2020) with the forward diffusion process as data augmentation. The result, presented in the second group of Table 2, is significantly worse than our model, indicating that our model is not equivalent to augmenting data before applying the discriminator.
Parametrization for pθ(xt−1|xt): We study two alternative ways to parametrize the denoising distribution for the same T = 4 setting. Instead of letting the generator produce estimated samples of x0, we set the generator to directly output denoised samples xt−1 without posterior sampling (direct denoising), or output the noise t that perturbs a clean image to produce xt (noise generation). Note that the latter case is closely related to most diffusion models where the network deterministically predicts the perturbation noise. In Table 2, we show that although these alternative parametrizations work reasonably well, our main parametrization outperforms them by a large margin.
Importance of latent variable: Removing latent variables z converts our denoising model to a unimodal distribution. In the last line of Table 2, we study our model’s performance without any latent variables z. We see that the sample quality is significantly worse, suggesting the importance of multimodal denoising distributions. In Figure 8, we visualize the effect of latent variables by showing samples of pθ(x0|x1), where x1 is a fixed noisy observation. We see that while the majority of information in the conditioning x1 is preserved, the samples are diverse due to the latent variables.
Table 2: Ablation studies on CIFAR-10.
Model Variants IS↑ FID↓ Recall↑ T = 1 8.93 14.6 0.19 T = 2 9.80 4.08 0.54 T = 4 9.63 3.75 0.57 T = 8 9.43 4.36 0.56
One-shot w/ aug 8.96 13.2 0.25
Direct denoising 9.10 6.03 0.53 Noise generation 8.79 8.04 0.52
No latent variable 8.37 20.6 0.42

heading: 5.3 ADDITIONAL STUDIES
text: Mode Coverage: Besides the recall score in Table 1, we also evaluate the mode coverage of our model on the popular 25-Gaussians and StackedMNIST. The 25-Gaussians dataset is a 2-D toy dataset, generated by a mixture of 25 two-dimensional Gaussian distributions, arranged in a grid. We train our denoising diffusion GAN with 4 denoising steps and compare it to other models in Figure 6. We observe that the vanilla GAN suffers severely from mode collapse, and while techniques like WGAN-GP (Gulrajani et al., 2017) improve mode coverage, the sample quality is still limited. In contrast, our model covers all the modes while maintaining high sample quality. We also train a diffusion model and plot the samples generated by 100 and 500 denoising steps. We see that diffusion models require a large number of steps to maintain high sample quality.
StackMNIST contains images generated by randomly choosing 3 MNIST images and stacking them along the RGB channels. Hence, the data distribution has 1000 modes. Following the setting of Lin et al. (2018), we report the number of covered modes and the KL divergence from the categorical distribution over 1000 categories of generated samples to true data in Table 3. We observe that our model covers all modes faithfully and achieves the lowest KL compared to GANs that are specifically designed for better mode coverage or StyleGAN2 that is known to have the best sample quality.
Training Stability: We discuss the training stability of our model in Appendix D.
High Resolution Images: We train our model on datasets with larger images, including CelebAHQ (Karras et al., 2018) and LSUN Church (Yu et al., 2015) at 256 × 256px resolution. We report FID on these two datasets in Table 4 and 5. Similar to CIFAR-10, our model obtains competitive sample quality among the best diffusion models and GANs. In particular, in LSUN Church, our model outperforms DDPM and ImageBART (see Figure 7 and Appendix E for samples). Although, some GANs perform better on this dataset, their mode coverage is not reflected by the FID score.
Stroke-based image synthesis: Recently, Meng et al. (2021b) propose an interesting application of diffusion models to stroke-based generation. Specifically, they perturb a stroke painting by the forward diffusion process, and denoise it with a diffusion model. The method is particularly promising because it only requires training an unconditional generative model on the target dataset and does not require training images paired with stroke paintings like GAN-based methods (Sangkloy et al., 2017; Park et al., 2019). We apply our model to stroke-based image synthesis and show qualitative results in Figure 9. The generated samples are realistic and diverse, while the conditioning in the stroke paintings is faithfully preserved. Compared to Meng et al. (2021b), our model enjoys a 1100× speedup in generation, as it takes only 0.16s to generate one image at 256 resolution vs. 181s for Meng et al. (2021b). This experiment confirms that our proposed model enables the application of diffusion models to interactive applications such as image editing.
Table 4: Generative results on CelebA-HQ-256
Model FID↓ Denoising Diffusion GAN (ours) 7.64
Score SDE (Song et al., 2021c) 7.23 LSGM (Vahdat et al., 2021) 7.22 UDM (Kim et al., 2021) 7.16
NVAE (Vahdat & Kautz, 2020) 29.7 VAEBM (Xiao et al., 2021) 20.4 NCP-VAE (Aneja et al., 2020) 24.8
PGGAN (Karras et al., 2018) 8.03 Adv. LAE (Pidhorskyi et al., 2020) 19.2 VQ-GAN (Esser et al., 2021b) 10.2 DC-AE (Parmar et al., 2021) 15.8
Table 5: Generative results on LSUN Church 256
Model FID↓ Denoising Diffusion GAN (ours) 5.25
DDPM (Ho et al., 2020) 7.89 ImageBART (Esser et al., 2021a) 7.32 Gotta Go Fast (Jolicoeur-Martineau et al.) 25.67
PGGAN (Karras et al., 2018)) 6.42 StyleGAN (Karras et al., 2019) 4.21 StyleGAN2 (Karras et al., 2020b) 3.86 CIPS (Anokhin et al., 2021) 2.92
Figure 7: Qualitative results on CelebA-HQ 256 and LSUN Church Outdoor 256.
Additional results: Additional qualitative visualizations are provided in Appendix E, F, and G.

heading: 6 CONCLUSIONS
text: Deep generative learning frameworks still struggle with addressing the generative learning trilemma. Diffusion models achieve particularly high-quality and diverse sampling. However, their slow sampling and high computational cost do not yet allow them to be widely applied in real-world applications. In this paper, we argued that one of the main sources of slow sampling in diffusion models is the Gaussian assumption in the denoising distribution, which is justified only for very small denoising steps. To remedy this, we proposed denoising diffusion GANs that model each denoising step using a complex multimodal distribution, enabling us to take large denoising steps. In extensive experiments, we showed that denoising diffusion GANs achieve high sample quality and diversity competitive to the original diffusion models, while being orders of magnitude faster at sampling. Compared to traditional GANs, our proposed model enjoys better mode coverage and sample diversity. Our denoising diffusion GAN overcomes the generative learning trilemma to a large extent, allowing diffusion models to be applied to real-world problems with low computational cost.

heading: 7 ETHICS AND REPRODUCIBILITY STATEMENT
text: Generating high-quality samples while representing the diversity in the training data faithfully has been a daunting challenge in generative learning. Mode coverage and high diversity are key requirements for reducing biases in generative models and improving the representation of minorities in a population. While diffusion models achieve both high sample quality and diversity, their expensive sampling limits their application in many real-world problems. Our proposed denoising diffusion GAN reduces the computational complexity of diffusion models to an extent that allows these models to be applied in practical applications at a low cost. Thus, we foresee that in the long term, our model can help with reducing the negative social impacts of existing generative models that fall short in capturing the data diversity.
We evaluate our model using public datasets, and therefore this work does not involve any human subject evaluation or new data collection.
Reproducibility Statement: We currently provide experimental details in the appendices with a detailed list of hyperparameters and training settings. To aid with reproducibility further, we will release our source code publicly in the future with instructions to reproduce our results.

heading: A DERIVATION FOR THE GAUSSIAN POSTERIOR
text: Ho et al. (2020) provide a derivation for the Gaussian posterior distribution. We include it here for completeness. Consider the forward diffusion process in Eq. 1, which we repeat here:
q(x1:T |x0) = T∏ t=1 q(xt|xt−1), q(xt|xt−1) = N (xt; √ 1− βtxt−1, βtI). (7)
Due to the Markov property of the forward process, we have the marginal distribution of xt given the initial clean data x0:
q(xt|x0) = N (xt; √ ᾱtx0, (1− ᾱt)I), (8)
where we denote αt := 1−βt and ᾱt := ∏t s=1 αs. Applying Bayes’ rule, we can obtain the forward process posterior when conditioned on x0:
q(xt−1|xt,x0) = q(xt|xt−1,x0)q(xt−1|x0)
q(xt|x0)
= q(xt|xt−1)q(xt−1|x0)
q(xt|x0) , (9)
where the second equation follows from the Markov property of the forward process. Since all three terms in Eq. 9 are Gaussians, the posterior q(xt−1|xt,x0) is also a Gaussian distribution, and it can be written as
q(xt−1|xt,x0) = N (xt−1; µ̃t(xt,x0), β̃tI) (10)
with mean µ̃t(xt,x0) and variance β̃t. Plugging the expressions in Eq. 7 and Eq. 8 into Eq. 10, we obtain
µ̃t(xt,x0) :=
√ ᾱt−1βt
1− ᾱt x0 +
√ αt(1− ᾱt−1)
1− ᾱt xt and β̃t := 1− ᾱt−1 1− ᾱt βt (11)
after minor simplifications.

heading: B PARAMETRIZATION OF DDPM
text: In Sec. 3.2, we mention that the parametrization of the denoising distribution for current diffusion models such as Ho et al. (2020) can be interpreted as pθ(xt−1|xt) := q(xt−1|xt,x0 = fθ(xt, t)). However, such a parametrization is not explicitly stated in Ho et al. (2020) but it is discussed by Song et al. (2021a). To avoid possible confusion, here we show that the parametrization of Ho et al. (2020) is equivalent to what we describe in Sec 3.2.
Ho et al. (2020) train a noise prediction network θ(xt, t) which predicts the noise that perturbs data x0 to xt, and a sample from pθ(xt−1|xt) is obtained as (see Algorithm 2 of Ho et al. (2020))
xt−1 = 1 √ αt
( xt −
1− αt√ 1− ᾱt θ (xt, t)
) + σtz, (12)
where z ∼ N (0, I) except for the last denoising step where z = 0, and σt = √ β̃t is the standard deviation of the Gaussian posterior distribution in Eq. 11.
Firstly, notice that predicting the perturbation noise θ(xt, t) is equivalent to predicting x0. We know that xt is generated by adding ∼ N (0, I) noise as:
xt = √ ᾱtx0 + √ 1− ᾱt ,
Hence, after predicting the noise with θ(xt, t) we can obtain a prediction of x0 using:
x0 = 1√ ᾱt
( xt − √ 1− ᾱt θ(xt, t) ) . (13)
Next, we can plug the expression for x0 in Eq. 13 into the mean of the Gaussian posterior distribution in Eq. 11, and we have
µ̃t(xt,x0) = µ̃t ( xt,
1√ ᾱt
( xt − √ 1− ᾱt θ(xt, t) )) (14)
= 1 √ αt
( xt −
1− αt√ 1− ᾱt θ(xt, t)
) (15)
after simplifications. Comparing this with Eq. 12, we observe that Eq. 12 simply corresponds to sampling from the Gaussian posterior distribution. Therefore, although Ho et al. (2020) use an alternative re-parametrization, their denoising distribution can still be equivalently interpreted as pθ(xt−1|xt) := q(xt−1|xt,x0 = fθ(xt, t)), i.e, first predicting x0 using the time-dependent denoising model, and then sampling xt−1 using the posterior distribution q(xt−1|xt,x0) given xt and the predicted x0.

heading: C EXPERIMENTAL DETAILS
text: In this section, we present our experimental settings in detail.
C.1 NETWORK STRUCTURE
Generator: Our generator structure largely follows the U-net structure (Ronneberger et al., 2015) used in NCSN++ (Song et al., 2021c), which consists of multiple ResNet blocks (He et al., 2016) and Attention blocks (Vaswani et al., 2017). Hyper-parameters for the network design, such as the number of blocks and number of channels, are reported in Table 6. We follow the default settings in Song et al. (2021c) for other network configurations not mentioned in the table, including Swish activation function, upsampling and downsampling with anti-aliasing based on Finite Impulse Response (FIR) (Zhang, 2019), re-scaling all skip connections by 1√
2 , using residual block design
from BigGAN (Brock et al., 2018) and incorporating progressive growing architectures (Karras et al., 2020b). See Appendix H of Song et al. (2021c) for more details on these configurations.
We follow Ho et al. (2020) and use sinusoidal positional embeddings for conditioning on integer time steps. The dimension for the time embedding is 4× the number of initial channels presented in Table 6. Contrary to previous works, we did not find the use of Dropout helpful in our case.
The fundamental difference between our generator network and the networks of previous diffusion models is that our generator takes an extra latent variable z as input. Inspired by the success of StyleGANs, we provide z-conditioning to the NCSN++ architecture using mapping networks, introduced in StyleGAN (Karras et al., 2019). We use z ∼ N (0, I) for all experiments. We replace all the group normalization (GN) layers in the network with adaptive group normalization (AdaGN) layers to allow the input of latent variables. The latent variable z is first transformed by a fully-connected network (called mapping network), and then the resulting embedding vector, denoted by w, is sent to every AdaGN layer. Each AdaGN layer contains one fully-connected layer that takes w as input, and outputs the per-channel shift and scale parameters for the group normalization. The network’s feature maps are then subject to affine transformations using these shift and scale parameters of the AdaGN layers. The mapping network and the fully-connected layer in AdaGN are independent of time steps t, as we found no extra benefit in incorporating time embeddings in these layers. Details about latent variables are also presented in Table 6.
Discriminator: We design our time-dependent discriminator with a convolutional network with ResNet blocks, where the design of the ResNet blocks is similar to that of the generator. The discriminator tries to discriminate real and fake xt−1, conditioned on xt and t. The time conditioning is enforced by the same sinusoidal positional embedding as in the generator. The xt conditioning is enforced by concatenating xt and xt−1 as the input to the discriminator. We use LeakyReLU activations with a negative slope 0.2 for all layers. Similar to Karras et al. (2020b), we use a minibatch standard deviation layer after all the ResNet blocks. We present the exact architecture of discriminators in Table 7.
Diffusion Process: For all datasets, we set the number of diffusion steps to be 4. In order to compute βt per step, we use the discretization of the continuous-time extension of the process described in Eq. 1, which is called the Variance Preserving (VP) SDE by Song et al. (2021c). We compute βt based on the continuous-time diffusion model formulation, as it allows us to ensure that variance schedule stays the same independent of the number of diffusion steps. Let’s define the normalized time variable by t′ := tT which normalizes t ∈ {1, 2, . . . , T} to [0, 1]. The variance function of VP SDE is given by:
σ2(t′) = 1− e−βmint ′−0.5(βmax−βmin)t′2 ,
with the constants βmax = 20 and βmin = 0.1. Recall that sampling from tth step in the forward diffusion process can be done with q(xt|x0) = N (xt; √ ᾱtx0, (1−ᾱt)I). We compute βt by solving 1− ᾱt = σ2( tT ):
βt = 1− αt = 1− ᾱt ᾱt−1
= 1− 1− σ2( tT )
1− σ2( t−1T ) = 1− e−βmin( 1 T )−0.5(βmax−βmin) 2t−1 T2 , (16)
for t ∈ {1, 2, . . . T}. This choice of βt values corresponds to equidistant steps in time according to VP SDE. Other choices are possible, but we did not explore them.
Objective: We train our denoising diffusion GAN with the following adversarial objective:
min φ T∑ t=1 Eq(x0)q(xt−1|x0)q(xt|xt−1) [ − log(Dφ(xt−1,xt, t) + Epθ(xt−1|xt)[− log(1−Dφ(xt−1,xt, t))] ] max θ T∑ t=1 Eq(x0)q(xt−1|x0)q(xt|xt−1) [ Epθ(xt−1|xt)[log(Dφ(xt−1,xt, t))]
] where the outer expectation denotes ancestral sampling from q(x0,xt−1,xt) and pθ(xt−1|xt) is our implicit GAN denoising distribution.
Similar to Ho et al. (2020), during training we randomly sample an integer time step t ∈ [1, 2, 3, 4] for each datapoint in a batch. Besides the main objective, we also add an R1 regularization term (Mescheder et al., 2018) to the objective for the discriminator. The R1 term is defined as
R1(φ) = γ
2 Eq(x0)q(xt−1|x0)q(xt|xt−1) [∥∥∇xt−1Dφ(xt−1,xt, t)∥∥2] , where γ is the coefficient for the regularization. We use γ = 0.05 for CIFAR-10, and γ = 1 for CelebA-HQ and LSUN Church. Note that theR1 regularization is a gradient penalty that encourages the discriminator to stay smooth and improves the convergence of GAN training (Mescheder et al., 2018).
Optimization: We train our models using the Adam optimizer (Kingma & Ba, 2015). We use cosine learning rate decay (Loshchilov & Hutter, 2016) for training both the generator and discriminator. Similar to Ho et al. (2020); Song et al. (2021c); Karras et al. (2020a), we observe that applying an exponential moving average (EMA) on the generator is crucial to achieve high performance. We summarize the optimization hyper-parameters in Table 8.
We train our models on CIFAR-10 using 4 V100 GPUs. On CelebA-HQ and LSUN Church we use 8 V100 GPUs. The training takes approximately 48 hours on CIFAR-10, and 180 hours on CelebA-HQ and LSUN Church.
C.3 EVALUATION
When evaluating IS, FID and recall score, we use 50k generated samples for CIFAR-10 and LSUN Church, and 30k samples for CelebA-HQ (since the CelebA HQ dataset contains only 30k samples).
When evaluating sampling time, we use models trained on CIFAR-10 and generate a batch of 100 samples. We benchmark the sampling time on a machine with a single V100 GPU. We use Pytorch 1.9.0 and CUDA 11.0.
C.4 ABLATION STUDIES
Here we introduce the settings for the ablation study in Sec. 5.2. We observe that training requires a larger number of training iterations when T is larger. As a result, we train the model for each T until the FID score does not increase any further. The number of training iteration is 200k for T = 1 and T = 2, 400k for T = 4 and 600k for T = 8. We use the same network structures and optimization settings as in the main experiments.
For the data augmentation baseline, we follow the differentiable data augmentation pipeline in Zhao et al. (2020). In particular, for every (real or fake) image in the batch, we perturbed it by sampling from a random timestep at the diffusion process (except the last diffusion step where the information of data is completely destroyed). We find the results insensitive to the number of possible perturbation levels (i.e, the number of steps in the diffusion process), and we report the result using a diffusion process with 4 steps. Since the perturbation by the diffusion process is differentiable due to the re-parametrization trick (Kingma & Welling, 2014), we can train both the discriminator and generator with the perturbed samples. See Zhao et al. (2020) for a detailed explanation for the training pipeline.
For the experiments on alternative parametrizations, we use T = 4 for the diffusion process and keep other settings the same as in the main experiments.
For the experiment on training a model without latent variables, similar to the main experiments, the generator takes the conditioning xt as its input, and the time conditioning is still enforced by the time embedding. However, the AdaGN layers are replaced by plain GN layers, such that no latent variable is needed, and the mapping network for z is removed. Other settings follow the main experiments.
C.5 TOY DATA AND STACKEDMNIST
For the 25-Gaussian toy dataset, both our generator and discriminator have 3 fully-connected layers each with 512 hidden units and LeakyReLU activations (negative slope of 0.2). We enforce both the conditioning on xt and t by concatenation with the input. We use the Adam optimizer with a learning rate of 10−4 for both the generator and discriminator. The batch size is 512, and we train the model for 50k iterations.
Our experimental settings for StackedMNIST are the same as those for CIFAR-10, except that we train the model for only 150k iterations.

heading: D TRAINING STABILITY
text: In Fig. 10, we plot the discriminator loss for different time steps in the diffusion process when T = 4. We observe that the training of our denoising diffusion GAN is stable and we do not see any explosion in loss values, as is sometimes reported for other GAN methods such as Brock et al. (2018). The stability might be attributed to two reasons: First, the conditioning on xt for both generator and discriminator provides a strong signal. The generator is required to generate a few plausible samples given xt and the discriminator requires classifying them. The xt conditioning keeps the discriminator and generator in a balance. Second, we are training the GAN on relatively smooth distributions, as the diffusion process is known as a smoothening process that brings the distributions of fake and real samples closer to each other (Lyu, 2012). As we can see from Fig. 10, the discriminator loss for t > 0 is higher than t = 0 (the last denoising step). Note that t > 0 corresponds to training the discriminator on noisy images, and in this case the true and generator distributions are closer to each other, making the discrimination harder and hence resulting in higher discriminator loss. We believe that such a property prevents the discriminator from overfitting, which leads to better training stability.

heading: E ADDITIONAL QUALITATIVE RESULTS
text: We show additional qualitative samples of CIFAR-10, CelebA-HQ and LSUN Church Outdoor in Figure 11, Figure 12 and Figure 13, respectively.
F ADDITIONAL VISUALIZATION FOR pθ(x0|xt)
In Figure 14 and Figure 15, we show visualizations of samples from pθ(x0|xt) for different t. Note that except for pθ(x0|x1), the samples from pθ(x0|xt) do not need to be sharp, as they are only intermediate outputs of the sampling process. The conditioning is less preserved as the perturbation in xt increases, and in particular xT (x4 in our example) contains almost no information of clean data x0.

heading: G NEAREST NEIGHBOR RESULTS
text: In Figure 16 and Figure 17, we show the nearest neighbors in the training dataset, corresponding to a few generated samples, where the nearest neighbors are computed using the feature distance of a pre-trained VGG network (Simonyan & Zisserman, 2014). We observe that the nearest neighbors are significantly different from the samples, suggesting that our models generalize well.

Review summaries:
This paper introduces _Denoising Diffusion GANs (DDGANs)_ to solve the so-called "generative learning trilemma" by training a generative model that achieves high-quality sampling, fast sampling times, and at the same time, high mode coverage. The main intuition of the paper is to combine GANs, which have been shown to achieve both high-quality synthesis and very fast sampling times, and DDPMs, which provide high-quality sampling and good mode coverage. In contrast to previous work on diffusion models that rely on Gaussian transition probabilities of the reverse forward process, conditional GANs are used in this work to allow flexible, multimodal denoising distributions in the reverse process and thus much shorter sequence lengths. Experiments show that the proposed approach indeed achieves higher mode coverage than conventional GAN models while maintaining high sample quality within 4 to 8 diffusion steps. The model can also be used for stroke-based image synthesis, with significant improvements in sampling times compared to previous approaches.

The diffusion models or score matching-based models have achieved great success in generating high fidelity samples with diverse modes. However, the sampling requires tremendous iterations, which hinders the practical usages. This paper proposes to break the Gaussian assumption in denoising steps by parametrizing it with a multimodal conditional GAN. The time for sampling can be 2000 times faster than the original diffusion model while keeping competitive quality and diversity.

The goal of the paper is to develop a generative model which achieves three desirable goals at the same time: High-quality samples, data/mode coverage and fast sampling times. While GANs have been associated with high-quality samples and fast sampling times, they are also known for their tendency to ignore modes of the data. On the other hand, diffusion models have recently been demonstrated to achieve high-quality samples and good likelihood estimation, which can indicate good mode coverage. However, diffusion models tend to be slow to generate samples due to their iterative nature. Thus, the paper proposes to combine diffusion models with GANs to combine their complementary advantages regarding mode coverage and sampling times (together with their shared ability to produce high-quality samples).  The main reason why diffusion models require a lot of iterations is because they model each of the steps of the reverse diffusion process with a Gaussian distribution, which is only valid for very small timesteps/a large number of steps. Thus, instead of using a network to estimate parameters of a Gaussian reverse process, the authors propose to model steps of the reverse process with a GAN. In theory, a GAN could directly model the data distribution and therefore solve the task in a single step, where the approach would be equivalent to a GAN and therefore suffer again from mode collapse. The main hope is therefore that by using a few diffusion steps, such an approach is still relatively fast compared to pure diffusion models while also having a good mode coverage.  Experiments evaluate sample quality, sampling speed and mode coverage. The main evaluation on CIFAR-10 indeed demonstrates a favorable combination of sample quality according to FID and IS, sampling speed and data coverage as measured by Recall. Additional experiments analyze the effect of various ablations and demonstrate an application in stroke-based image synthesis.