Denoising Diffusion Implicit Models

1 INTRODUCTION . Deep generative models have demonstrated the ability to produce high quality samples in many domains ( Karras et al. , 2020 ; van den Oord et al. , 2016a ) . In terms of image generation , generative adversarial networks ( GANs , Goodfellow et al . ( 2014 ) ) currently exhibits higher sample quality than likelihood-based methods such as variational autoencoders ( Kingma & Welling , 2013 ) , autoregressive models ( van den Oord et al. , 2016b ) and normalizing flows ( Rezende & Mohamed , 2015 ; Dinh et al. , 2016 ) . However , GANs require very specific choices in optimization and architectures in order to stabilize training ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Karras et al. , 2018 ; Brock et al. , 2018 ) , and could fail to cover modes of the data distribution ( Zhao et al. , 2018 ) . Recent works on iterative generative models ( Bengio et al. , 2014 ) , such as denoising diffusion probabilistic models ( DDPM , Ho et al . ( 2020 ) ) and noise conditional score networks ( NCSN , Song & Ermon ( 2019 ) ) have demonstrated the ability to produce samples comparable to that of GANs , without having to perform adversarial training . To achieve this , many denoising autoencoding models are trained to denoise samples corrupted by various levels of Gaussian noise . Samples are then produced by a Markov chain which , starting from white noise , progressively denoises it into an image . This generative Markov Chain process is either based on Langevin dynamics ( Song & Ermon , 2019 ) or obtained by reversing a forward diffusion process that progressively turns an image into noise ( Sohl-Dickstein et al. , 2015 ) . A critical drawback of these models is that they require many iterations to produce a high quality sample . For DDPMs , this is because that the generative process ( from noise to data ) approximates the reverse of the forward diffusion process ( from data to noise ) , which could have thousands of steps ; iterating over all the steps is required to produce a single sample , which is much slower compared to GANs , which only needs one pass through a network . For example , it takes around 20 hours to sample 50k images of size 32 × 32 from a DDPM , but less than a minute to do so from a GAN on a Nvidia 2080 Ti GPU . This becomes more problematic for larger images as sampling 50k images of size 256× 256 could take nearly 1000 hours on the same GPU . To close this efficiency gap between DDPMs and GANs , we present denoising diffusion implicit models ( DDIMs ) . DDIMs are implicit probabilistic models ( Mohamed & Lakshminarayanan , 2016 ) and are closely related to DDPMs , in the sense that they are trained with the same objective function . In Section 3 , we generalize the forward diffusion process used by DDPMs , which is Markovian , to non-Markovian ones , for which we are still able to design suitable reverse generative Markov chains . We show that the resulting variational training objectives have a shared surrogate objective , which is exactly the objective used to train DDPM . Therefore , we can freely choose from a large family of generative models using the same neural network simply by choosing a different , nonMarkovian diffusion process ( Section 4.1 ) and the corresponding reverse generative Markov Chain . In particular , we are able to use non-Markovian diffusion processes which lead to ” short ” generative Markov chains ( Section 4.2 ) that can be simulated in a small number of steps . This can massively increase sample efficiency only at a minor cost in sample quality . In Section 5 , we demonstrate several empirical benefits of DDIMs over DDPMs . First , DDIMs have superior sample generation quality compared to DDPMs , when we accelerate sampling by 10× to 100× using our proposed method . Second , DDIM samples have the following “ consistency ” property , which does not hold for DDPMs : if we start with the same initial latent variable and generate several samples with Markov chains of various lengths , these samples would have similar high-level features . Third , because of “ consistency ” in DDIMs , we can perform semantically meaningful image interpolation by manipulating the initial latent variable in DDIMs , unlike DDPMs which interpolates near the image space due to the stochastic generative process . 2 BACKGROUND . Given samples from a data distribution q ( x0 ) , we are interested in learning a model distribution pθ ( x0 ) that approximates q ( x0 ) and is easy to sample from . Denoising diffusion probabilistic models ( DDPMs , Sohl-Dickstein et al . ( 2015 ) ; Ho et al . ( 2020 ) ) are latent variable models of the form pθ ( x0 ) = ∫ pθ ( x0 : T ) dx1 : T , where pθ ( x0 : T ) : = pθ ( xT ) T∏ t=1 p ( t ) θ ( xt−1|xt ) ( 1 ) where x1 , . . . , xT are latent variables in the same sample space as x0 ( denoted as X ) . The parameters θ are learned to fit the data distribution q ( x0 ) by maximizing a variational lower bound : max θ Eq ( x0 ) [ log pθ ( x0 ) ] ≤ max θ Eq ( x0 , x1 , ... , xT ) [ log pθ ( x0 : T ) − log q ( x1 : T |x0 ) ] ( 2 ) where q ( x1 : T |x0 ) is some inference distribution over the latent variables . Unlike typical latent variable models ( such as the variational autoencoder ( Rezende et al. , 2014 ) ) , DDPMs are learned with a fixed ( rather than trainable ) inference procedure q ( x1 : T |x0 ) , and latent variables are relatively high dimensional . For example , Ho et al . ( 2020 ) considered the following Markov chain with Gaussian transitions parameterized by a decreasing sequence α1 : T ∈ ( 0 , 1 ] T : q ( x1 : T |x0 ) : = T∏ t=1 q ( xt|xt−1 ) , where q ( xt|xt−1 ) : = N ( √ αt αt−1 xt−1 , ( 1− αt αt−1 ) I ) ( 3 ) where the covariance matrix is ensured to have positive terms on its diagonal . This is called the forward process due to the autoregressive nature of the sampling procedure ( from x0 to xT ) . We call the latent variable model pθ ( x0 : T ) , which is a Markov chain that samples from xT to x0 , the generative process , since it approximates the intractable reverse process q ( xt−1|xt ) . Intuitively , the forward process progressively adds noise to the observation x0 , whereas the generative process progressively denoises a noisy observation ( Figure 1 , left ) . A special property of the forward process is that q ( xt|x0 ) : = ∫ q ( x1 : t|x0 ) dx1 : ( t−1 ) = N ( xt ; √ αtx0 , ( 1− αt ) I ) ; so we can express xt as a linear combination of x0 and a noise variable : xt = √ αtx0 + √ 1− αt , where ∼ N ( 0 , I ) . ( 4 ) When we set αT sufficiently close to 0 , q ( xT |x0 ) converges to a standard Gaussian for all x0 , so it is natural to set pθ ( xT ) : = N ( 0 , I ) . If all the conditionals are modeled as Gaussians with trainable mean functions and fixed variances , the objective in Eq . ( 2 ) can be simplified to1 : Lγ ( θ ) : = T∑ t=1 γtEx0∼q ( x0 ) , t∼N ( 0 , I ) [ ‖ ( t ) θ ( √ αtx0 + √ 1− αt t ) − t‖ 2 2 ] ( 5 ) where θ : = { ( t ) θ } Tt=1 is a set of T functions , each ( t ) θ : X → X ( indexed by t ) is a function with trainable parameters θ ( t ) , and γ : = [ γ1 , . . . , γT ] is a vector of positive coefficients in the objective that depends on α1 : T . In Ho et al . ( 2020 ) , the objective with γ = 1 is optimized instead to maximize generation performance of the trained model ; this is also the same objective used in noise conditional score networks ( Song & Ermon , 2019 ) based on score matching ( Hyvärinen , 2005 ; Vincent , 2011 ) . From a trained model , x0 is sampled by first sampling xT from the prior pθ ( xT ) , and then sampling xt−1 from the generative processes iteratively . The length T of the forward process is an important hyperparameter in DDPMs . From a variational perspective , a large T allows the reverse process to be close to a Gaussian ( Sohl-Dickstein et al. , 2015 ) , so that the generative process modeled with Gaussian conditional distributions becomes a good approximation ; this motivates the choice of large T values , such as T = 1000 in Ho et al . ( 2020 ) . However , as all T iterations have to be performed sequentially , instead of in parallel , to obtain a sample x0 , sampling from DDPMs is much slower than sampling from other deep generative models , which makes them impractical for tasks where compute is limited and latency is critical . 3 VARIATIONAL INFERENCE FOR NON-MARKOVIAN FORWARD PROCESSES . Because the generative model approximates the reverse of the inference process , we need to rethink the inference process in order to reduce the number of iterations required by the generative model . Our key observation is that the DDPM objective in the form of Lγ only depends on the marginals2 q ( xt|x0 ) , but not directly on the joint q ( x1 : T |x0 ) . Since there are many inference distributions ( joints ) with the same marginals , we explore alternative inference processes that are non-Markovian , which leads to new generative processes ( Figure 1 , right ) . These non-Markovian inference process lead to the same surrogate objective function as DDPM , as we will show below . In Appendix A , we show that the non-Markovian perspective also applies beyond the Gaussian case . 3.1 NON-MARKOVIAN FORWARD PROCESSES . Let us consider a family Q of inference distributions , indexed by a real vector σ ∈ RT≥0 : qσ ( x1 : T |x0 ) : = qσ ( xT |x0 ) T∏ t=2 qσ ( xt−1|xt , x0 ) ( 6 ) where qσ ( xT |x0 ) = N ( √ αTx0 , ( 1− αT ) I ) and for all t > 1 , qσ ( xt−1|xt , x0 ) = N ( √ αt−1x0 + √ 1− αt−1 − σ2t · xt − √ αtx0√ 1− αt , σ2t I ) . ( 7 ) The mean function is chosen to order to ensure that qσ ( xt|x0 ) = N ( √ αtx0 , ( 1 − αt ) I ) for all t ( see Lemma 1 of Appendix B ) , so that it defines a joint inference distribution that matches the “ marginals ” as desired . The forward process3 can be derived from Bayes ’ rule : qσ ( xt|xt−1 , x0 ) = qσ ( xt−1|xt , x0 ) qσ ( xt|x0 ) qσ ( xt−1|x0 ) , ( 8 ) 1Please refer to Appendix C.2 for details . 2We slightly abuse this term ( as well as joints ) when only conditioned on x0 . 3We overload the term “ forward process ” for cases where the inference model is not a diffusion . which is also Gaussian ( although we do not use this fact for the remainder of this paper ) . Unlike the diffusion process in Eq . ( 3 ) , the forward process here is no longer Markovian , since each xt could depend on both xt−1 and x0 . The magnitude of σ controls the how stochastic the forward process is ; when σ → 0 , we reach an extreme case where as long as we observe x0 and xt for some t , then xt−1 become known and fixed .

Summary: This paper develops a variant (DDIM) of an existing method (DDPM) with the goal of accelerating it greatly while still maintaining performance. The authors are working in the context of a denoising process that runs in the reverse direction to a sequence of steps that each add a small amount of Gaussian noise to the original data. The proposal is to introduce an auxiliary function that breaks the Markov assumption by leaking some information in a controlled way about the training points x0, and then use this auxiliary function as scaffolding to train the actual Markov chain of denoising functions.