Denoising Diffusion Implicit Models

1 INTRODUCTION . Deep generative models have demonstrated the ability to produce high quality samples in many domains ( Karras et al. , 2020 ; van den Oord et al. , 2016a ) . In terms of image generation , generative adversarial networks ( GANs , Goodfellow et al . ( 2014 ) ) currently exhibits higher sample quality than likelihood-based methods such as variational autoencoders ( Kingma & Welling , 2013 ) , autoregressive models ( van den Oord et al. , 2016b ) and normalizing flows ( Rezende & Mohamed , 2015 ; Dinh et al. , 2016 ) . However , GANs require very specific choices in optimization and architectures in order to stabilize training ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Karras et al. , 2018 ; Brock et al. , 2018 ) , and could fail to cover modes of the data distribution ( Zhao et al. , 2018 ) . Recent works on iterative generative models ( Bengio et al. , 2014 ) , such as denoising diffusion probabilistic models ( DDPM , Ho et al . ( 2020 ) ) and noise conditional score networks ( NCSN , Song & Ermon ( 2019 ) ) have demonstrated the ability to produce samples comparable to that of GANs , without having to perform adversarial training . To achieve this , many denoising autoencoding models are trained to denoise samples corrupted by various levels of Gaussian noise . Samples are then produced by a Markov chain which , starting from white noise , progressively denoises it into an image . This generative Markov Chain process is either based on Langevin dynamics ( Song & Ermon , 2019 ) or obtained by reversing a forward diffusion process that progressively turns an image into noise ( Sohl-Dickstein et al. , 2015 ) . A critical drawback of these models is that they require many iterations to produce a high quality sample . For DDPMs , this is because that the generative process ( from noise to data ) approximates the reverse of the forward diffusion process ( from data to noise ) , which could have thousands of steps ; iterating over all the steps is required to produce a single sample , which is much slower compared to GANs , which only needs one pass through a network . For example , it takes around 20 hours to sample 50k images of size 32 × 32 from a DDPM , but less than a minute to do so from a GAN on a Nvidia 2080 Ti GPU . This becomes more problematic for larger images as sampling 50k images of size 256× 256 could take nearly 1000 hours on the same GPU . To close this efficiency gap between DDPMs and GANs , we present denoising diffusion implicit models ( DDIMs ) . DDIMs are implicit probabilistic models ( Mohamed & Lakshminarayanan , 2016 ) and are closely related to DDPMs , in the sense that they are trained with the same objective function . In Section 3 , we generalize the forward diffusion process used by DDPMs , which is Markovian , to non-Markovian ones , for which we are still able to design suitable reverse generative Markov chains . We show that the resulting variational training objectives have a shared surrogate objective , which is exactly the objective used to train DDPM . Therefore , we can freely choose from a large family of generative models using the same neural network simply by choosing a different , nonMarkovian diffusion process ( Section 4.1 ) and the corresponding reverse generative Markov Chain . In particular , we are able to use non-Markovian diffusion processes which lead to ” short ” generative Markov chains ( Section 4.2 ) that can be simulated in a small number of steps . This can massively increase sample efficiency only at a minor cost in sample quality . In Section 5 , we demonstrate several empirical benefits of DDIMs over DDPMs . First , DDIMs have superior sample generation quality compared to DDPMs , when we accelerate sampling by 10× to 100× using our proposed method . Second , DDIM samples have the following “ consistency ” property , which does not hold for DDPMs : if we start with the same initial latent variable and generate several samples with Markov chains of various lengths , these samples would have similar high-level features . Third , because of “ consistency ” in DDIMs , we can perform semantically meaningful image interpolation by manipulating the initial latent variable in DDIMs , unlike DDPMs which interpolates near the image space due to the stochastic generative process . 2 BACKGROUND . Given samples from a data distribution q ( x0 ) , we are interested in learning a model distribution pθ ( x0 ) that approximates q ( x0 ) and is easy to sample from . Denoising diffusion probabilistic models ( DDPMs , Sohl-Dickstein et al . ( 2015 ) ; Ho et al . ( 2020 ) ) are latent variable models of the form pθ ( x0 ) = ∫ pθ ( x0 : T ) dx1 : T , where pθ ( x0 : T ) : = pθ ( xT ) T∏ t=1 p ( t ) θ ( xt−1|xt ) ( 1 ) where x1 , . . . , xT are latent variables in the same sample space as x0 ( denoted as X ) . The parameters θ are learned to fit the data distribution q ( x0 ) by maximizing a variational lower bound : max θ Eq ( x0 ) [ log pθ ( x0 ) ] ≤ max θ Eq ( x0 , x1 , ... , xT ) [ log pθ ( x0 : T ) − log q ( x1 : T |x0 ) ] ( 2 ) where q ( x1 : T |x0 ) is some inference distribution over the latent variables . Unlike typical latent variable models ( such as the variational autoencoder ( Rezende et al. , 2014 ) ) , DDPMs are learned with a fixed ( rather than trainable ) inference procedure q ( x1 : T |x0 ) , and latent variables are relatively high dimensional . For example , Ho et al . ( 2020 ) considered the following Markov chain with Gaussian transitions parameterized by a decreasing sequence α1 : T ∈ ( 0 , 1 ] T : q ( x1 : T |x0 ) : = T∏ t=1 q ( xt|xt−1 ) , where q ( xt|xt−1 ) : = N ( √ αt αt−1 xt−1 , ( 1− αt αt−1 ) I ) ( 3 ) where the covariance matrix is ensured to have positive terms on its diagonal . This is called the forward process due to the autoregressive nature of the sampling procedure ( from x0 to xT ) . We call the latent variable model pθ ( x0 : T ) , which is a Markov chain that samples from xT to x0 , the generative process , since it approximates the intractable reverse process q ( xt−1|xt ) . Intuitively , the forward process progressively adds noise to the observation x0 , whereas the generative process progressively denoises a noisy observation ( Figure 1 , left ) . A special property of the forward process is that q ( xt|x0 ) : = ∫ q ( x1 : t|x0 ) dx1 : ( t−1 ) = N ( xt ; √ αtx0 , ( 1− αt ) I ) ; so we can express xt as a linear combination of x0 and a noise variable : xt = √ αtx0 + √ 1− αt , where ∼ N ( 0 , I ) . ( 4 ) When we set αT sufficiently close to 0 , q ( xT |x0 ) converges to a standard Gaussian for all x0 , so it is natural to set pθ ( xT ) : = N ( 0 , I ) . If all the conditionals are modeled as Gaussians with trainable mean functions and fixed variances , the objective in Eq . ( 2 ) can be simplified to1 : Lγ ( θ ) : = T∑ t=1 γtEx0∼q ( x0 ) , t∼N ( 0 , I ) [ ‖ ( t ) θ ( √ αtx0 + √ 1− αt t ) − t‖ 2 2 ] ( 5 ) where θ : = { ( t ) θ } Tt=1 is a set of T functions , each ( t ) θ : X → X ( indexed by t ) is a function with trainable parameters θ ( t ) , and γ : = [ γ1 , . . . , γT ] is a vector of positive coefficients in the objective that depends on α1 : T . In Ho et al . ( 2020 ) , the objective with γ = 1 is optimized instead to maximize generation performance of the trained model ; this is also the same objective used in noise conditional score networks ( Song & Ermon , 2019 ) based on score matching ( Hyvärinen , 2005 ; Vincent , 2011 ) . From a trained model , x0 is sampled by first sampling xT from the prior pθ ( xT ) , and then sampling xt−1 from the generative processes iteratively . The length T of the forward process is an important hyperparameter in DDPMs . From a variational perspective , a large T allows the reverse process to be close to a Gaussian ( Sohl-Dickstein et al. , 2015 ) , so that the generative process modeled with Gaussian conditional distributions becomes a good approximation ; this motivates the choice of large T values , such as T = 1000 in Ho et al . ( 2020 ) . However , as all T iterations have to be performed sequentially , instead of in parallel , to obtain a sample x0 , sampling from DDPMs is much slower than sampling from other deep generative models , which makes them impractical for tasks where compute is limited and latency is critical . 3 VARIATIONAL INFERENCE FOR NON-MARKOVIAN FORWARD PROCESSES . Because the generative model approximates the reverse of the inference process , we need to rethink the inference process in order to reduce the number of iterations required by the generative model . Our key observation is that the DDPM objective in the form of Lγ only depends on the marginals2 q ( xt|x0 ) , but not directly on the joint q ( x1 : T |x0 ) . Since there are many inference distributions ( joints ) with the same marginals , we explore alternative inference processes that are non-Markovian , which leads to new generative processes ( Figure 1 , right ) . These non-Markovian inference process lead to the same surrogate objective function as DDPM , as we will show below . In Appendix A , we show that the non-Markovian perspective also applies beyond the Gaussian case . 3.1 NON-MARKOVIAN FORWARD PROCESSES . Let us consider a family Q of inference distributions , indexed by a real vector σ ∈ RT≥0 : qσ ( x1 : T |x0 ) : = qσ ( xT |x0 ) T∏ t=2 qσ ( xt−1|xt , x0 ) ( 6 ) where qσ ( xT |x0 ) = N ( √ αTx0 , ( 1− αT ) I ) and for all t > 1 , qσ ( xt−1|xt , x0 ) = N ( √ αt−1x0 + √ 1− αt−1 − σ2t · xt − √ αtx0√ 1− αt , σ2t I ) . ( 7 ) The mean function is chosen to order to ensure that qσ ( xt|x0 ) = N ( √ αtx0 , ( 1 − αt ) I ) for all t ( see Lemma 1 of Appendix B ) , so that it defines a joint inference distribution that matches the “ marginals ” as desired . The forward process3 can be derived from Bayes ’ rule : qσ ( xt|xt−1 , x0 ) = qσ ( xt−1|xt , x0 ) qσ ( xt|x0 ) qσ ( xt−1|x0 ) , ( 8 ) 1Please refer to Appendix C.2 for details . 2We slightly abuse this term ( as well as joints ) when only conditioned on x0 . 3We overload the term “ forward process ” for cases where the inference model is not a diffusion . which is also Gaussian ( although we do not use this fact for the remainder of this paper ) . Unlike the diffusion process in Eq . ( 3 ) , the forward process here is no longer Markovian , since each xt could depend on both xt−1 and x0 . The magnitude of σ controls the how stochastic the forward process is ; when σ → 0 , we reach an extreme case where as long as we observe x0 and xt for some t , then xt−1 become known and fixed .

Summary: This paper proposes a change to the recently popular diffusion models, motivated by increasing the speed of sampling. This is accomplished by changing the “forward” process which adds noise to the data. In the original diffusion models, this forward process is a Markov process whose marginals and conditionals can be computed efficiently in closed form. This paper proposes to replace this Markov forward process with a non-markovian process that is designed to have the same marginals. The generative model, in this case, changes such that to predict the next step in the process, the model must first predict the “clean” sample at the end of the chain which is then used to give an estimate for the next step in the chain.