Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

1 INTRODUCTION 1 In the past decade , a plethora of deep generative models has been developed for various domains such as images ( Karras et al. , 2019 ; Razavi et al. , 2019 ) , audio ( Oord et al. , 2016a ; Kong et al. , 2021 ) , point clouds ( Yang et al. , 2019 ) and graphs ( De Cao & Kipf , 2018 ) . However , current generative learning frameworks can not yet simultaneously satisfy three key requirements , often needed for their wide adoption in real-world problems . These requirements include ( i ) high-quality sampling , ( ii ) mode coverage and sample diversity , and ( iii ) fast and computationally inexpensive sampling . For example , most current works in image synthesis focus on high-quality generation . However , mode coverage and data diversity are important for better representing minorities and for reducing the negative social impacts of generative models . Additionally , applications such as interactive image editing or real-time speech synthesis require fast sampling . Here , we identify the challenge posed by these requirements as the generative learning trilemma , since existing models usually compromise between them . Fig . 1 summarizes how mainstream generative frameworks tackle the trilemma . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ; Brock et al. , 2018 ) generate high-quality samples rapidly , but they are known for poor mode coverage ( Salimans et al. , 2016 ; Zhao et al. , 2018 ) . Conversely , variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) and normalizing flows ( Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ) cover data modes faithfully , but they often suffer from low sample quality . Recently , diffusion models ( Sohl-Dickstein et al. , 2015 ; Ho et al. , 2020 ; Song et al. , 2021c ) have emerged as powerful generative models . Diffusion models define a forward diffusion process that maps data to noise by gradually perturbing the input data . Data generation is achieved using a parametrized reverse process that performs iterative denoising , starting from random noise . Diffusion models demonstrate surprisingly good results in sample quality , beating GANs in image generation ( Dhariwal & Nichol , 2021 ; Ho et al. , 2021 ) . They also demonstrate good mode coverage , indicated by high likelihood ( Song et al. , 2021b ; Kingma et al. , 2021 ; Huang et al. , 2021 ) . Diffusion models have already been applied to a variety of generation tasks ( Dhariwal & Nichol , 2021 ; Austin et al. , 2021 ; Mittal et al. , 2021 ; Luo & Hu , 2021 ) . However , sampling from them often requires thousands of network evaluations , making their application to real-world problems expensive . In this paper , we tackle the generative learning trilemma by reformulating denoising diffusion models specifically for fast sampling while maintaining strong mode coverage and sample quality . We investigate the slow sampling issue of diffusion models and we observe that diffusion models commonly assume that the denoising distribution can be approximated by Gaussian distributions . However , it is known that the Gaussian assumption holds only in the infinitesimal limit of small denoising steps ( Sohl-Dickstein et al. , 2015 ; Feller , 1949 ) , which leads to the requirement of a large number of steps in the reverse process . When the reverse process uses larger step sizes ( i.e. , it has fewer denoising steps ) , we need a non-Gaussian multimodal distribution for modeling the denoising distribution . Intuitively , in image synthesis , the multimodal distribution arises from the fact that multiple plausible clean images may correspond to the same noisy image . Inspired by this observation , we propose to parametrize the denoising distribution with an expressive multimodal distribution to enable denoising for large steps . In particular , we introduce a novel generative model , termed as denoising diffusion GAN , in which the denoising distributions are modeled with conditional GANs . In image generation , we observe that our model obtains sample quality and mode coverage competitive with diffusion models , while taking only as few as two denoising steps , achieving about 2000× speed-up in sampling compared to the predictor-corrector sampling by Song et al . ( 2021c ) on CIFAR-10 . Compared to traditional GANs , we show that our model significantly outperforms state-of-the-art GANs in sample diversity , while being competitive in sample fidelity . In summary , we make the following contributions : i ) We attribute the slow sampling of diffusion models to the Gaussian assumption in the denoising distribution and propose to employ complex , multimodal denoising distributions . ii ) We propose denoising diffusion GANs , a diffusion model whose reverse process is parametrized by conditional GANs . iii ) Through careful evaluations , we demonstrate that denoising diffusion GANs achieve several orders of magnitude speed-up compared to current diffusion models for both image generation and editing . We show that our model overcomes the deep generative learning trilemma to a large extent , making diffusion models for the first time applicable to interactive , real-world applications at a low computational cost . 2 BACKGROUND . In diffusion models ( Sohl-Dickstein et al. , 2015 ; Ho et al. , 2020 ) , there is a forward process that gradually adds noise to the data x0 ∼ q ( x0 ) in T steps with pre-defined variance schedule βt : q ( x1 : T |x0 ) = ∏ t≥1 q ( xt|xt−1 ) , q ( xt|xt−1 ) = N ( xt ; √ 1− βtxt−1 , βtI ) , ( 1 ) where q ( x0 ) is a data-generating distribution . The reverse denoising process is defined by : pθ ( x0 : T ) = p ( xT ) ∏ t≥1 pθ ( xt−1|xt ) , pθ ( xt−1|xt ) = N ( xt−1 ; µθ ( xt , t ) , σ2t I ) , ( 2 ) where µθ ( xt , t ) and σ2t are the mean and variance for the denoising model and θ denotes its parameters . The goal of training is to maximize the likelihood pθ ( x0 ) = ∫ pθ ( x0 : T ) dx1 : T , by maximizing the evidence lower bound ( ELBO , L ≤ log pθ ( x0 ) ) . The ELBO can be written as matching the true denoising distribution q ( xt−1|xt ) with the parameterized denoising model pθ ( xt−1|xt ) using : L = − ∑ t≥1 Eq ( xt ) [ DKL ( q ( xt−1|xt ) ‖pθ ( xt−1|xt ) ) ] + C , ( 3 ) where C contains constant terms that are independent of θ and DKL denotes the Kullback-Leibler ( KL ) divergence . The objective above is intractable due to the unavailability of q ( xt−1|xt ) . Instead , 2 Sohl-Dickstein et al . ( 2015 ) show that L can be written in an alternative form with tractable distributions ( see Appendix A of Ho et al . ( 2020 ) for details ) . Ho et al . ( 2020 ) show the equivalence of this form with score-based models trained with denoising score matching ( Song & Ermon , 2019 ; 2020 ) . Two key assumptions are commonly made in diffusion models : First , the denoising distribution pθ ( xt−1|xt ) is modeled with a Gaussian distribution . Second , the number of denoising steps T is often assumed to be in the order of hundreds to thousands of steps . In this paper , we focus on discrete-time diffusion models . In continuous-time diffusion models ( Song et al. , 2021c ) , similar assumptions are also made at the sampling time when discretizing time into small timesteps . 3 DENOISING DIFFUSION GANS . We first discuss why reducing the number of denoising steps requires learning a multimodal denoising distribution in Sec . 3.1 . Then , we present our multimodal denoising model in Sec . 3.2 . 3.1 MULTIMODAL DENOISING DISTRIBUTIONS FOR LARGE DENOISING STEPS . As we discussed in Sec . 2 , a common assumption in the diffusion model literature is to approximate q ( xt−1|xt ) with a Gaussian distribution . Here , we question when such an approximation is accurate . The true denoising distribution q ( xt−1|xt ) can be written as q ( xt−1|xt ) ∝ q ( xt|xt−1 ) q ( xt−1 ) using Bayes ’ rule where q ( xt|xt−1 ) is the forward Gaussian diffusion shown in Eq . 1 and q ( xt−1 ) is the marginal data distribution at step t. It can be shown that in two situations the true denoising distribution takes a Gaussian form . First , in the limit of infinitesimal step size βt , the product in the Bayes ’ rule is dominated by q ( xt|xt−1 ) and the reversal of the diffusion process takes an identical functional form as the forward process ( Feller , 1949 ) . Thus , when βt is sufficiently small , since q ( xt|xt−1 ) is a Gaussian , the denoising distribution q ( xt−1|xt ) is also Gaussian , and the approximation used by current diffusion models can be accurate . To satisfy this , diffusion models often have thousands of steps with small βt . Second , if data marginal q ( xt ) is Gaussian , the denoising distribution q ( xt−1|xt ) is also a Gaussian distribution . The idea of bringing data distribution q ( x0 ) and consequently q ( xt ) closer to Gaussian using a VAE encoder was recently explored in LSGM ( Vahdat et al. , 2021 ) . However , the problem of transforming the data to Gaussian itself is challenging and VAE encoders can not solve it perfectly . That is why LSGM still requires tens to hundreds of steps on complex datasets . In this paper , we argue that when neither of the conditions are met , i.e. , when the denoising step is large and the data distribution is non-Gaussian , there are no guarantees that the Gaussian assumption on the denoising distribution holds . To illustrate this , in Fig . 2 , we visualize the true denoising distribution for different denoising step sizes for a multimodal data distribution . We see that as the denoising step gets larger , the true denoising distribution becomes more complex and multimodal . 3.2 MODELING DENOISING DISTRIBUTIONS WITH CONITIONAL GANS . Our goal is to reduce the number of denoising diffusion steps T required in the reverse process of diffusion models . Inspired by the observation above , we propose to model the denoising distribution with an expressive multimodal distribution . Since conditional GANs have been shown to model complex conditional distributions in the image domain ( Mirza & Osindero , 2014 ; Ledig et al. , 2017 ; Isola et al. , 2017 ) , we adopt them to approximate the true denoising distribution q ( xt−1|xt ) . Specifically , our forward diffusion is set up similarly to the diffusion models in Eq . 1 with the main assumption that T is assumed to be small ( T ≤ 8 ) and each diffusion step has larger βt . Our training is formulated by matching the conditional GAN generator pθ ( xt−1|xt ) and q ( xt−1|xt ) using an adversarial loss that minimizes a divergence Dadv per denoising step : min θ ∑ t≥1 Eq ( t ) [ Dadv ( q ( xt−1|xt ) ‖pθ ( xt−1|xt ) ) ] , ( 4 ) whereDadv can be Wasserstein distance , Jenson-Shannon divergence , or f-divergence depending on the adversarial training setup ( Arjovsky et al. , 2017 ; Goodfellow et al. , 2014 ; Nowozin et al. , 2016 ) . In this paper , we rely on non-saturating GANs ( Goodfellow et al. , 2014 ) , that are widely used in successful GAN frameworks such as StyleGANs ( Karras et al. , 2019 ; 2020b ) . In this case , Dadv takes a special instance of f-divergence called softened reverse KL ( Shannon et al. , 2020 ) , which is different from the forward KL divergence used in the original diffusion model training in Eq . 31 . To set up the adversarial training , we denote the time-dependent discriminator as Dφ ( xt−1 , xt , t ) : RN × RN × R→ [ 0 , 1 ] , with parameters φ . It takes the N -dimensional xt−1 and xt as inputs , and decides whether xt−1 is a plausible denoised version of xt . The discriminator is trained by : min φ ∑ t≥1 Eq ( xt ) [ Eq ( xt−1|xt ) [ −log ( Dφ ( xt−1 , xt , t ) ] + Epθ ( xt−1|xt ) [ −log ( 1−Dφ ( xt−1 , xt , t ) ) ] ] , ( 5 ) where fake samples from pθ ( xt−1|xt ) are contrasted against real samples from q ( xt−1|xt ) . The first expectation requires sampling from q ( xt−1|xt ) which is unknown . However , we use the identity q ( xt , xt−1 ) = ∫ dx0q ( x0 ) q ( xt , xt−1|x0 ) = ∫ dx0q ( x0 ) q ( xt−1|x0 ) q ( xt|xt−1 ) to rewrite the first expectation in Eq . 5 as : Eq ( xt ) q ( xt−1|xt ) [ − log ( Dφ ( xt−1 , xt , t ) ) ] = Eq ( x0 ) q ( xt−1|x0 ) q ( xt|xt−1 ) [ − log ( Dφ ( xt−1 , xt , t ) ) ] . Given the discriminator , we train the generator by maxθ ∑ t≥1 Eq ( xt ) Epθ ( xt−1|xt ) [ log ( Dφ ( xt−1 , xt , t ) ) ] , which updates the generator with the non-saturating GAN objective ( Goodfellow et al. , 2014 ) . Parametrizing the implicit denoising model : Instead of directly predicting xt−1 in the denoising step , diffusion models ( Ho et al. , 2020 ) can be interpreted as parameterizing the denoising model by pθ ( xt−1|xt ) : = q ( xt−1|xt , x0 = fθ ( xt , t ) ) in which first x0 is predicted using the denoising model fθ ( xt , t ) , and then , xt−1 is sampled using the posterior distribution q ( xt−1|xt , x0 ) given xt and the predicted x0 ( See Appendix B for details ) . The distribution q ( xt−1|x0 , xt ) is intuitively the distribution over xt−1 when denoising from xt towards x0 , and it always has a Gaussian form for the diffusion process in Eq . 1 , independent of the step size and complexity of the data distribution ( see Appendix A for the expression of q ( xt−1|x0 , xt ) ) . Similarly , we define pθ ( xt−1|xt ) by : pθ ( xt−1|xt ) : = ∫ pθ ( x0|xt ) q ( xt−1|xt , x0 ) dx0 = ∫ p ( z ) q ( xt−1|xt , x0 =Gθ ( xt , z , t ) ) dz , ( 6 ) where pθ ( x0|xt ) is the implicit distribution imposed by the GAN generatorGθ ( xt , z , t ) : RN×RL× R→ RN that outputs x0 given xt and an L-dimensional latent variable z ∼ p ( z ) : = N ( z ; 0 , I ) . Our parameterization has several advantages : Firstly , our pθ ( xt−1|xt ) is formulated similar to DDPM ( Ho et al. , 2020 ) . Thus , we can borrow some inductive bias such as the network structure design from DDPM . The main difference is that , in DDPM , x0 is predicted as a deterministic mapping of xt , while in our case x0 is produced by the generator with random latent variable z . This is the key difference that allows our denoising distribution pθ ( xt−1|xt ) to become multimodal and complex in contrast to the unimodal denoising model in DDPM . Secondly , note that for different t ’ s , xt has different levels of perturbation , and hence using a single network to predict xt−1 directly at different t may be difficult . However , in our case the generator only needs to predict unperturbed x0 and then add back perturbation using q ( xt−1|xt , x0 ) . Fig . 3 visualizes our training pipeline . 1At early stages , we examined training a conditional VAE as pθ ( xt−1|xt ) by minimizing∑T t=1 Eq ( t ) [ DKL ( q ( xt−1|xt ) ‖pθ ( xt−1|xt ) ) ] for small T . However , conditional VAEs consistently resulted in poor generative performance in our early experiments . In this paper , we focus on conditional GANs and we leave the exploration of other expressive conditional generators for pθ ( xt−1|xt ) to future work . Advantage over one-shot generator : One natural question for our model is , why not just train a GAN that can generate samples in one shot using a traditional setup , in contrast to our model that generates samples by denoising iteratively ? Our model has several advantages over traditional GANs . GANs are known to suffer from training instability and mode collapse ( Kodali et al. , 2017 ; Salimans et al. , 2016 ) , and some possible reasons include the difficulty of directly generating samples from a complex distribution in one-shot , and the overfitting issue when the discriminator only looks at clean samples . In contrast , our model breaks the generation process into several conditional denoising diffusion steps in which each step is relatively simple to model , due to the strong conditioning on xt . Moreover , the diffusion process smoothens the data distribution ( Lyu , 2012 ) , making the discriminator less likely to overfit . Thus , we expect our model to exhibit better training stability and mode coverage . We empirically verify the advantages over traditional GANs in Sec . 5 .

Summary: The goal of the paper is to develop a generative model which achieves three desirable goals at the same time: High-quality samples, data/mode coverage and fast sampling times. While GANs have been associated with high-quality samples and fast sampling times, they are also known for their tendency to ignore modes of the data. On the other hand, diffusion models have recently been demonstrated to achieve high-quality samples and good likelihood estimation, which can indicate good mode coverage. However, diffusion models tend to be slow to generate samples due to their iterative nature. Thus, the paper proposes to combine diffusion models with GANs to combine their complementary advantages regarding mode coverage and sampling times (together with their shared ability to produce high-quality samples).  The main reason why diffusion models require a lot of iterations is because they model each of the steps of the reverse diffusion process with a Gaussian distribution, which is only valid for very small timesteps/a large number of steps. Thus, instead of using a network to estimate parameters of a Gaussian reverse process, the authors propose to model steps of the reverse process with a GAN. In theory, a GAN could directly model the data distribution and therefore solve the task in a single step, where the approach would be equivalent to a GAN and therefore suffer again from mode collapse. The main hope is therefore that by using a few diffusion steps, such an approach is still relatively fast compared to pure diffusion models while also having a good mode coverage.  Experiments evaluate sample quality, sampling speed and mode coverage. The main evaluation on CIFAR-10 indeed demonstrates a favorable combination of sample quality according to FID and IS, sampling speed and data coverage as measured by Recall. Additional experiments analyze the effect of various ablations and demonstrate an application in stroke-based image synthesis.