Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders
Authors: Huangjie Zheng
Pengcheng He
Weizhu Chen
Mingyuan Zhou
Venue: ICLR 2023 poster (2023)
Keywords: Diffusion model
adversarial autoencoder
implicit prior

Abstract: Employing a forward diffusion chain to gradually map the data to a  noise distribution, diffusion-based generative models learn how to generate the data by inferring a reverse diffusion chain. However, this approach is slow and costly because it needs many forward and reverse steps. We propose a faster and cheaper approach that adds noise not until the data become pure random noise, but until they reach a hidden noisy data distribution that we can confidently learn. Then, we use fewer reverse steps to generate data by starting from this hidden distribution that is made similar to the noisy data. We reveal that the proposed model can be cast as an adversarial auto-encoder empowered by both the diffusion process and a learnable implicit prior. Experimental results show even with a significantly smaller number of reverse diffusion steps, the proposed truncated diffusion probabilistic models can provide consistent improvements over the non-truncated ones in terms of performance in both unconditional and text-guided image generations.

Published as a conference paper at ICLR 2023
TRUNCATED DIFFUSION PROBABILISTIC MODELS AND DIFFUSION-BASED ADVERSARIAL AUTO-ENCODERS
Huangjie Zheng" 1.2 & Pengcheng He² & Weizhu Chen? & Mingyuan Zhou'
The University of Texas at Austin', Microsoft Azure AI2 huangjie zhengeutexas edu, penhe www.eeeccc.ooo..ooooooooooooo..e con, wzchen@microsofte com, mingyuan. zhou@mccombs ut edu
ABSTRACT
Employing forward diffusion chain to gradually map the data to noise distribu- tion, diffusion- based generative models learn how to generate the data by inferring reverse diffusion chain. However, this approach is slow and costly because it needs many forward and reverse steps. We propose a faster and cheaper approach that adds noise not until the data become pure random noise, but until they reach hidden noisy-data distribution that we can confidently learn. Then, we use fewer reverse steps generate data by starting from this hidden distribution that is made similar to the noisy data. We reveal that the proposed model can be cast as an adversarial auto-encoder empowered by both the diffusion process and a learn- able implicit prior. Experimental results show even with significantly smaller number of reverse diffusion steps, the proposed truncated diffusion probabilistic models can provide consistent improvements over the non-truncated ones in terms performance in both unconditional and text- exx--guieed image generations.
INTRODUCTION
Generating photo-realistic images with probabilistic models challenging and important task in machine learning and computer vision, with many potential applications in data augmentation, image editing, style transfer, etc. Recently, new class of image generative models based on diffusion processes (Sohl-Dickstein et al. 2015) has achieved remarkable results on various commonly used
image generation benchmarks (Song Ermon 2019 Ho et al. 2020: Song & Ermon 2020 Song et al. 2021b: Dhariwal & Nichol 2021), surpassing many existing deep generative models, such as autoregressive models (van den Oord et al. 20166). variational auto-encoders (VAEs) (Kingma &
Welling 2013 Rezende et al.) [014 van den Oord et al. 2017 Razavi et al. 2019), and generative
adversanal networks (GANs) Goodfellow et al. 2014 Radford et al. 2015 Arjovsky et al. 2017
Miyato et al. 2018 Brock al. 2019 Karras et al. 2019 (0200))
This new modeling class, which includes both score-based and diffusion based generative models, noise injection gradually corrupt the data distribution into simple noise distribution that can be easily sampled from, and then uses a denoising network to reverse the noise injection to generate photo-realistic images. From the perspective of score matching (Hyvärinen Dayan 2005) Vincent 2011) and Langevin dynamics (Neal 2011 Welling & Tch 2011). the denoising network trained by matching the score function, which the gradient of the log-density of the data, of the corrupted data distribution and that of the generator distribution at different noise levels (Song Ermon 2019). This training objective can also be formulated under diffusion iiffusion-based generative models (Sohl-Dickstein et al.) 2015 Ho et al.) 2020). These two types of models have been further unified by Song et al. (202Tb) under the framework of discretized stochastic differential equations.
Despite their impressive performance, diffusion-bart based (or score-based) generative models suffer from high computational costs, both in training and sampling. This is because they need to perform large number of diffusion steps, typically hundreds thousands, to ensure that the noise injection is small enough at each step to make the assumption that both the diffusion and denoising processes have the Gaussian form hold in the limit of small diffusion rate (Feller 1949 Sohl- Dicksten et al. 2015). In other words, when the number of diffusion steps is small or when the rate is large, the Gaussian assumption may not hold well, and the model may not be able capture the true score function

---

Published as a conference paper at ICLR 2023
Diffuse step: g/z x..1) ---1=+1 =
Denoise step: data reconstruction loss (o()--186)) ) N
Diffuse step: dx =
Matching prior to aggregated posterior Nia.
Denolse step: data reconstruction loss (o(x--1(6)) N
generato,, (Best viewed in color) An illustrative depiction of diffusion models and our truncated diffusion models. Top: The conventional denoising diffusion models add Gaussian noise gradually with a large number of time steps, where the true posterior (score-matching) loss (marked in a solid blue box). Bottom: Truncated diffusion models truncate the diffusion chain keep its first few steps and small diffusion segment (marked in the dashed blue box). This truncated diffusion chain can still be learned with previous denoising methods, Mcanwhile, as the left part is truncated, the Gaussian prior p(xr) will have large gap to the truneated point g(xx Xu). which is bridged with an implicit generative distribution (xT) s po (xx (z)((z)z (marked in dashed red box).
Truncated diffusion chain
Figure 1:
be kept close Gaussian and hence easy to fit with denoising
of the data. Therefore, previous works have tried to reduce the number of diffusion steps by using
non-Markovian reverse processes (Song et al. 2020 Kong & Ping 2021). adaptive noise scheduling
(San-Roman et al., [202]] Kingma al. 202 knowledge distillation (Luhman & Luhman 2202
Salimans & Ho| (022), diffusing in a power-dimension latent space (Rombach et al. (0022), etc., but
they still cannot achieve significant speedup without sacrificing the generation quality.
In this paper, we propose a novel way to shorten the diffusion trajectory by learning an implicit distribution start the reverse diffusion process, instead of relying on tractable noise distribution. We call our method truncated diffusion probabilistic modeling (TDPM), which is based on the idea of truncating the forward diffusion chain of an existing diffusion model, such as the denoising diffusion probabilistic model (DDPM) of Ho et al. (2020). To significantly accelerate diffusion- text-to-image generation, we also introduce the truncated latent diffusion model (TLDM), which truncates the diffusion chain of the latent diffusion model (LDM) of Rombach et al. LDM is the latent -an-te- diffusion model behind Stable e Di fiffusi i on
2022). We note an open-source project that provides state-of of-the-art performance in generating photo-realistic images given text input. By truncating the chain, we can reduce the number of diffusion steps to an arbitrary level, but at the same time, also lose the tractability of the distribution at the end of the chain. Therefore, we need to learn an implicit generative distribution that can approximate this distribution and provide the initial samples for the reverse diffusion process. We show that this implicit generative distribution can be implemented in different ways, such as using a separate generator network or reusing the denoising network. The former option has more fiexibility and can improve the generation quality, while the latter option has no additional parameters and can achieve comparable results.
We reveal that DDPM and VAE have a similar relationship as TDPM and adversarial auto-encoder (AAE, Makhzani et al. (2015). Specifically, DDPM is like VAE with fixed encoder and learnable decoder that use a diffusion process, and predefined prior. TDPM is like an AAE with a fixed encoder and learnable decoder that use truncated diffusion process, and learnable implicit prior.
Our truncation method has several advantages when we use modify DDPM for generating images without text guidance or LDM for generating images with text guidance. First, it can generate samples much faster by using fewer diffusion steps, without sacrificing or even enhancing the generation quality. Second, it can exploit the cooperation between the implicit model and the diffusion model, as the diffusion model helps the implicit model train by providing noisy data samples, and the implicit model helps the diffusion model reverse by providing better initial samples. Third, it can adapt the truncation level to balance the generation quality and efficiency, depending on the data complexity and the computational resources. For generating images with text guidance, our method can speed up the generation significantly and make it suitable for real-time processing: while LDM takes the time to generate one photo-realistic image, our TLDD can generate more than 50 such images.
2

---

Published as a conference paper at ICLR 2023
The main contributions of our paper follows:
We introduce TDPM, a new diffusion- based generative model that can shorten the diffusion trajectory by learning implicit distribution start the reverse diffusion process, and demonstrate that the learning of the implicit distribution can be achieved in various ways. We further introduce TLDM to significantly accelerate diffusion-based text-to-image generation.
We show TDPM can be formulated as diffusion cuffussio-bbasd AAE.
We show that the implicit distribution can be realized by reusing the denoising network for the reverse diffusion process, which can reduce the reverse diffusion steps by orders of magnitude without adding any extra parameters and with comparable generation quality.
We reveal the synergy between the implicit model and the diffusion model, as the diffusion process can simplify the training of the implicit model like GANs, and the implicit model can speed up the reverse diffusion process of the diffusion model.
We show that both TDPM and TLDM can adapt the truncation level, according to the data complexity and the computational resources, to achieve a good balance between the generation quality and the efficiency.
PRELIMINARIES ON DIFFUSION MODELS
In Gaussian diffusion models (Sohl- Dickktist et al. 2015 Ho et al. 2020), starting from the data distribution Xo ~ (xn), a pre-defined forward diffusion process qe produces auxiliary variables
Xt=1:T by gradually adding Gaussian noise, with variance 3r (0, 1) at time f, follows:
qx11 xr xb) I[ (xx (x-1),, q(x, xx-1) = N(xx /1 -1x (1)
With the limit of small diffusion rate (i.e., Br is kept sufficiently small). the reverse distribution q x2) also follows a Gaussian distribution (Feller 1949
et al. 2015) and can
be approximated using neural network parameterized Gaussian distribution Po as:
(x xx) = N (x (x. f), (x,(,, t)).
(2)
Moreover, with sufficiently large T', the outcome of the diffusion chain xy will follow an isotropic Gaussian distribution. Thus, with the pre-defined forward (inference) diffusion process and the learned reverse (generative) diffusion process, we can sample from Xy NN I) and run the
diffusion process reverse to get sample from the data distribution (xx)).
Under the variational inference (Kingma & Welling 2013 Blei et al. 2017) framework, viewing
((x,...., XT Xn) in 1 as the inference network, we can use the evidence lower bound (ELBO) as our learning objective. Following previous works Dicht-Dickstein et al. 2015 Ho et al., 2020). the
negative ELBO a diffusion probabilistic model, parameterized by 0, can be expressed as
CELBO (0) (0) + ET-2 (0) + Cr, Co(0) = Eg(xo) g (x)) [- log (Xo x1)] (3)
1 (0) E g((x)) Egtte xx) (Dr.. (g( (x X1, (o)) x.))., EE ( T)
(4)
Lr Ex [DKL (g(xx) xx) (p(xx)))
(5)
where DKt(alle) (qllp) = F. flog - log denotes the Kullback-Leibler (KL) divergence from distri- butions p q. Generally speaking, diffusion probabilistic models assume the number of diffusion steps I' to be sufficiently large to satisfy two conditions: 1) the reverse distribution at each denoising step can be fitted with a Gaussian denoising generator pe (xx): 2) with a sufficiently small diffusion rate Br, the long forward diffusion process will successfully corrupt the data, making q(xx Xg) = (0, I). and hence approximately Lr becomes zero and depends on neither x, nor f.
What happens if T' is insufficiently large? Given a non- non-Gaussian data distribution g(xu), when the number of denoising steps is reduced, the true posterior q(x;-1 x,) not Gaussian and usually intractable (Feller) 1949), resulting in new challenges to current diffusion models. As noted in Xiao al.) (2022). when 3g not sufficiently small, the diffusion step becomes larger and the denoising distribution can be multi-modal and hence too complex to be well fitted by Gaussian. The authors propose to define pa (x xt) with an implicit generator and substitute the ELBO with
min Ee> Eq(0) [Dav (x(x-) X (po)lpp(xxt- xx))],
(6)
3

---

Published as a conference paper at ICLR 2023
where Dadv represents statistical distance that relies on an adversarial training setup. This modified objective can be minimized by leveraging the power of conditional GANs in fitting implicit multi- modal distributions (Arjovsky et al.] 2017; Goodfellow et al.] 2014 Nowozin et al., 2016). While the concept of diffusion has been used, the proposed models in Xiao et (2022) are shown to work the best only when the number of diffusion steps is limited to be as few as four, and start exhibit
deteriorated performance when further increasing that number.
3 TRUNCATED DIFFUSION AND ADVERSARIAL AUTO-ENCODING
We first introduce the idea of accelerating both the training and generation of diffusion models by truncating the diffusion chains and describe the technical challenges. We then develop the objective function and training algorithm for TDPM. We further reveal TDPM can also be formulated as an AAE (Makhzani et al. 2015)) empowered by diffusion models. While DDPM can be considered as a hierarchical version of variational auto- encoder (VAE) with fixed multi-stochastic-layee encoder, our derivation shows that TDPM can be considered as a hierarchical version of an AAE with fixed multi-stochastic-laye encoder but learnable implicit prior.
3.1 MOTIVATION AND TECHNICAL CHALLENGES
We propose novel method called TDPM to speed up the diffusion process and the generative model. The main idea is to shorten the forward diffusion chain that transforms the data into Gaussian noise, and use learned implicit distribution to sample the starting point of the reverse diffusion chain that reconstructs the data. To be more precise, we adopt the DDPM framework that defines a variance schedule (B1, .2,.., ..... Br), which controls the amount of noise added at each step of the forward diffusion process. The forward process has a simple analytical form as a Gaussian distribution:
g(xe xo) = NV (1 T))
02 - ai a; = Bi.
Here, X, is the noisy version of the data Xq step and āe is the cumulative product of the diffusion coefficients Oş The forward chain of length T is designed to be long enough to make the data distribution indistinguishable from Gaussian noise I). However, long forward chain also implies a high computational cost for the reverse process, which uses a learned neural network to predict the conditional distribution of the clean data given the noisy one at each step.
The proposed TDPM cuts off the last part of the forward chain and only keeps the first Trune steps {.1, ...... 3rmme } C (3,.82..... 3. We choose Truse to be much smaller than so that we can save lot of computation time in generation. The benefit of this truncation is illustrated in Figure [ where the bottom row shows the truncated diffusion chain. We can see that the data are only partially corrupted by noise and still retain some features of the original data. This means that we can recover the data more easily and accurately by applying a few Gaussian denoising steps from the corrupted data. Moreover, do not change the diffusion rates 3r for the first Thune steps, we do not compromise the quality of the forward and reverse processes between time O and Turnne-
However, truncating the forward chain also introduces new challenge for the reverse process. Unlike the original chain, where the starting point of the reverse process is Xy ~ (O, I), the truncated chain has unknown distribution of the corrupted data at step Tirnne- This makes it difficult sample from this distribution and initiate the reverse process. To overcome this challenge, we introduce an implicit generative model that approximates the distribution of the corrupted data by minimizing a divergence measure between the implicit and the true noisy distributions at step Timue- This way, we can use the implicit model to sample the starting point of the reverse process and then apply the leamed denoising network to generate the data.
3.2 HAND-CRAFTED TDPM OBJECTIVE FUNCTION
Mathematically, recall that the DDPM loss in 33 consists of three terms: Lo, I t= Lt-1, and Cr- 7 The training objective of conventional diffusion model focuses terms Σ 1 L--1 and Lo. It assumes Ey does not depend on any parameter and will be close to zero by carefully pre-defining the forward noising process such that (XT Xo) R p(XT) NO, 7).
4

---

Published as a conference paper at ICLR 2023
When the diffusion chains truncated at time Tuune < T, the forward diffusion ends at time T Tumee where the marginal distribution of the forward diffusion-corrupted data can be expressed as
((x. == g((x Xo xo)p(xx)dda)
which takes a semi-implicti form (Yin & Zhou 2018 whose density function is often intractable. To reverse this truncated forward diffusion chain, we can no longer start the reverse diffusion chain from a known distribution such as (0,, I). To this end, we propose TDPM that starts the reverse chain at time Timee from Pa (XTTm))))) ), an implicit distribution parameterized by th. We match pal (XT...)) to q( (mie) via loss term Lrum. between distributions and p. such as the Jensen-Shanne
DD (q(XT (XTrmn))) )) where Dig] p) is statistical distance divergence and Wasserstein distance. As
we keep all the diffusion steps before time Trane in TDPM the same as those in DDPM, we combine LTume with all the loss terms of DDPM before time Timune in (3) define the TDPM loss as
LIDPM == Eler (6) (V), Srmme (3b) == D (q(XT )l p (XTm..))
(8)
We note while in general Px ) in TDPM is intractable, we can employ a deep neural network- based generator Gy to generate random sample in single step via
XTimx = Gb (z), 2 ~ I).
(9)
We will discuss later that we may simply let vi 0 to avoid adding more parameters.
3.3 TDPM AS DIFFUSION-BASED ADVERSARIAL AUTO-ENCODER
Following the terminology of AAE, let us define the prior as Pyp (XTne))) ). the decoder (likelihood) as (x | X = J [Ie
Tous: pa(x;-1 }x,]] ]d IIx.x......ddi
(10)
which is empowered by reverse diffusion chain of length Timne. and the encoder (variational
posterior) as qxxrmmm. Xg). Thus we can view ((T defined in (7) as the aggregated posterior
(Hoffman & Johnson 2016 Tomezak & Welling 2018). In addition to imposing auto-encoding
data-reconsttuuttinn loss, the key idea of the AÄE (Makhzani et al. 2015) is to also match the
fixed prior. This idea differs AAE from VAE that regularizes the auto- encoder by matching the variational posterior to a fixed prior under the KL divergence. To this end, we introduce diffusion- based AAE (Diffusion-AARe AAE), whose loss function is defined as
aggregated posterior
(11) Diffusion-AAE AAE has two notable differences from a vanilla AAE: 1) its encoder is fixed and has no learnable parameters, while its prior is not fixed and is optimized to match the aggregated posterior, and 2) its decoder is reverse diffusion chain, with Tirane stochastic layers all parameterized by 6.
Lpmmusion-AAe = -E -E)()) Eg(xTame Xo) log Po (Xo Xmi..) ) + (X 1)P (Xr...))
Note in general as the likelihood in 10) is intractable, the first loss term in 111 is intractable. However, the loss of Diffusion-ANA is upper bounded by the loss of TDPM, as described below.
Theorem 1. The Diffusion-AAE loss in (77) is upper bounded by the TDPM loss
DDittusion-AAE I LTDPM
3.4 MATCHING THE PRIOR TO AGGREGATED POSTERIOR
Via the loss term L D (g(XT Ipp (XT )) in (8. we aim to match the prior Py (XT,m)) to the aggregated posterior g (XTmme) in TDPM. While we have an analytic density function for neither p nor q. we can easily draw random samples from both of them. Thus, we explore the use of two different types of statistical distances that can be estimated from samples of both 9 and p. We empirically show that TDPM can achieve good performance regardless of which distance used for optimization.
One possible statistical distance is based on the idea of GANs (Goodfellow et al. 2014 Arjovsky et al. [2177: Binkowski et al.) 20188), which are widely used to learn implicit distributions from empirical data. In this setting, we use a generator (.) Rdd + Rd to transform samples from an isotropic Gaussian p(z) into samples that approximate the corrupted data, and a discriminator D.(.) Rd + [0, 1) to distinguish between the samples from the corrupted data distribution q(XTimm xx)) and the implicit generative distribution p (XTm.... The generator and the discriminator trained by the following objective LGAN,
min max
Ex (log (x) + +E Ez~p(z) (G
(12)
5

---

Published as a conference paper at ICLR 2023
3.5 TRAINING ALGORITHM
As the objective in Equation 8 is a sum of different terms, following DDPM (Ho et al. 2020) to fix
the terms Co t) = I, , we can simplify Tiux Eer L, as an expectation defined as
Simple tune = Et,xo.e, [lle cx(x,
t)] t~ Unif([1,2,... Tuune), (0
(13)
where 6 is an injected noise at a uniformly sampled timestep index I, x, = voyXn vT aft is noisy image at time and eg is a denoising U-Net that predicts the noise in order to refine the noisy image X,. Therefore the final simplified version of (8) is constructed as
CAN Lsimple_rme trune + AC
(14)
While X, the weight of LTime. can be tuned, we fix it as one for simplicity. Here the TDPM objective consists of two parts: the denoising part Eø is focused on denoising the truncated chain, getting updated from Ssiple__rrme trume, while the implicit part G is focused on minimizing Eg (D (q(X Ilpp (X getting updated from Lrn
An interesting finding of this paper is that we do not necessarily need introduce separate set of parameters &o for the generator Gg., as we can simply reuse the same parameters of the reverse diffusion model (i.e., let 3o #) without clearly hurting the empirical performance. This suggests that the reverse diffusion process from T to Tinme could be effectively approximated by single step using the same network architecture and parameters as the reverse diffusion steps from Tinme to O.
Therefore, we provide two configurations to parameterize the implicit distributions. 1) To save parameters, we the implicit generator and denoising model share the same U-Net parameters but using different time step indices. Specifically, we first use XTm.. = Gy (xT) EØ (x(T,, t=Trmee + +1,, where XT ~ (0, I), to generate a noisy image at time Tuune- 2) We further explore employing a different model, e.g., StyleGAN2 (Karras al. (020a), for the implicit generator, which provides better performance but increases the model size to get XT'me. Then for t t= =uuuue. 1, we iteratively 1-0, refine it as Xt-1 = vas (x: /1-ar Eg (x,, t)) + Byzz, where 2, ~ N0, I) when > and 21 = 0. This process is depicted in Algorithms T and 2 in the Appendix. For the implementation details, please refer to Appendix D.6 and our code at https::///// /tt:://gitttt gightb...bbgg.ggggg com/ com Jeqgheng/ Eruncated-difie
3.6 RELATED WORK
In our previous discussions, we have related TDPM to several existing works such as DDPM and AAE. A detailed discussion other related works is provided in Appendix B
4 EXPERIMENTS
We aim to demonstrate that TDPM can generate good samples faster by using fewer steps of reverse diffusion. We use different image datasets to test our method and follow the same setting other diffusion models (Ho et al., 2020 Nichol & Dhariwal 2021 Dhariwal & Nichol 202T;; Rombach al., 2022) for our backbones. We also have two ways to set up the implicit generator that starts the reverse diffusion. One way is to reuse the denoising network, and the other way is to use separate network. We try both ways for generating images without any labels. For generating images from text, we use the first way with the LDM backbone. We provide comprehensive details, toy examples, and additional experimental results in Appendices [D4 D.8
We use FID (lower is better) and Recall (higher is better) to measure the fidelity and diversity, respectively, the generated images. We use CIFAR-10 (Krizhevsky et al., 2009), LSUN-bedroom, and L.SUN-Curchh (Yu et al. 2013) datasets in unconditional experiments, and CUB-200 (Welinder) cct al. 2010) and MS-COCO (Lin et al. 2014) for text-to-image experiments. The images consist of 32 32 pixels for CIFAR- 10 and 256 x 256 pixels for the other datasets.
4.1 EFFICIENCY IN BOTH TRAINING AND SAMPLING
We first look at the results on CIFAR-10. We use DDPM (Ho et al. 2020) or improved DDPM (Nichol Dhariwal 2021) as backbones. We use 4, 49, or 99 steps of reverse diffusion, which correspond

---

Published as a conference paper at ICLR 2023
Table 1:
Results of unconditional generation on CIFAR-10, 10, with the best FID and Recall in each group marked in bold. To compare TDPM (O) with GAN-hased methods, we use DDPM hackbone gen- erator and StyleGAN2 discriminator.
Table 2: Results on LSUN-Church and LSUN Bedroom (resolution 256 x 256). Similar to Table || TDPM (I (0mm===) uses DDPM backbone for the generator.
Church Bedroom
Method
NFE
FID
FID
Method
NFE
FID
Recall
DDPM backbone
DDPM backbone
DDPM
1000
7.89
4.90
DDPM
1000
3,21
0.57
TDPM
(T (Timn
100
4.33
3.95
TDPM (Irrae=99)
100
3.10
0.57
TDPM+ (Tmun==99)
100
3.98
3.67
TDPM+ 199))
=99)
100
2.88
0.58
DDIM
30
TO.38
6.62
DDIM
50
4.67
0.53
TDPM 449)
5.35
4.10
TDPM (Trraee
=49)
50
3,30
0.57
TDPM+
(T Tmmn==49
50
4.34
3.98
TDPM+ Trum=
499)
50
2,94
0,58
TDPM
Franc =4)
5
3.34
0.57
TDPM
(Trmm
5
4.98
4.16
TDPM+
(T Trunc=
5
4.89
4.09
TDPM+ (Tune
3.21
0.57
ADM backbone
Improved DDPM backbone
Improved DDPM
4000
2,90
0.58
ADM
1000
3.49
1.90
ADM+DDIM
250
6.45
2.31
IDPM (99)
100
2.97
0.57
TDPM (T (99))==99)
100
4.41
2.24
TDPM+ Trume =99)
100
2.83
0.58
Improved DDPM+DDIM
50
3.92
0.55
TDPM+ (T (TTmm==99)
100
3.61
1.88
TDPM (TTm
50
4.57
2.92
TDPM (TTrn==49)
50
3.11
0.57
TDPM+
50
3.67
1.89
TDPM+ (I 49))
2.96
0.58
TDPM (Tirae=4)
5
5,61
7.92
TDPM (Trrne=4)
3.51
0.55
TDPM+ (imn:=11
5
4.66
4.01
TDPM+ (Tirne =4
3.17
0,57
GAN-based
GAN-based
DDGAN
4
5.25
DDGAN
3.75
0.57
StyleGAN2
1
3.93
3.98
StyleGAN2
8.32
0.41
StyleGAN2-ADA
I
4.12
7.89
StyleGAN2-ADA
2.92
0.49
TDPM (O)
4.77
5.24
TDPM (Zrm:=0)
7.34
0.46
Table 3: Results of ImageNet-64x 64, evaluated with FID and Recall. TDPM+ is built with pre-trained ADM and an implicit model trained at Truse using StyIGAN-XL.
Method
NFE
FID
Recall
ADM
1000
2.07
0.63
TDPM+ (T 999)
100
1.62
0.63
TDPM+ (TTum:=49)
50
1.77
0.58
TDPM+ (Trm.:=4)
5
1.92
0.53
Figure 2:
Random generation results of TDPM+ (Trmm==4 on ImageNet-64x66à
StyleGAN (wo PG)
3.54
0.51
to 5, 50, 100 number of function evaluations (NFE). For the implicit generator, either reuse
the denoising U-Net or use a StyleGAN2 network (respectively, we call them TDPM and TDPM+).
For comparison, also include DDIM (Song al.) 2020) and DDGAN (Xiao et al. 2022). The comparison with more diverse set of baselines can be found in Table D in Appendix D.7
Table [ shows that TDPM get good FID with fewer NFE. TDPM+ can get better FID, and it is the best when NFE=100. Compared with TDPM with 0 steps of reverse diffusion (a GAN with DDPM's U-Net as generator and StyleGAN2 as discriminator) and StyleGAN2, TDPM with more than 0 steps of reverse diffusion has better recall and the FID is as good as StyleGAN2-ADA GAN with data augmentation for better training). This means TDPM can largely avoid the mode missing problem in GANs. We show some examples of generated images on CIFAR-10 in Figure I3
We also check how fast TDPM can train and sample. In training, we count how many images TDPM needs to well fit the truncated diffusion chain and the implicit prior. Figure 3 shows that when we use fewer steps of reverse diffusion, the diffusion part needs less time train. But the implicit prior needs more time to train because it has to model harder distribution, e.g., fitting the implicit prior with diffusion steps needs similar time to directly fit it on the data. When we use 99 steps of reverse diffusion, the diffusion chain and the implicit prior need similar time to train, and the whole model trains faster than both GAN and DDPM. In sampling, we compare TDPM with 0, 1, 4, 49, or 99 steps one NVIDIA V100
of reverse diffusion. We report both FID and the sampling time (s/image) GPU in Figure 4 When we use 4 steps reverse diffusion, the FID is much lower than 0 steps, and the sampling time is slightly longer. When
steps of reverse diffusion, the FID goes down
7

---

Published as conference paper at ICLR 2023
tog(kimgs
te
T (GAN)
Tmaa-4 4
Toae= 49
T9
DDPM
2212236
Figure 3: The required iterations (measured with iter- ated images) converge in the training. The iterations for < T Tume: (ce) and t= Timee (G marked in red and blue, respectively.
Eassuu speed-s2 Roess-up
speed-up
Figure 4: Evolution of FID and corresponding GPU time (s/mmgee) across different timesteps in the sam- pling stage.
Tirunc =
Tirnnc = 49
TOID
Trrunc = 99
Figure 5: Randomly generated images of TDPM using ADM (Dhariwal Nichol
2021
backhone L.SUN-
Church LSUN- LSUN-Bedroon Bedroom (256 256), with Tiree 4, 49, and 99. Note NFE group presents generated samples from the full model (x(x)) (left) and its implicit prior ~ pe (x(T (right); the full model sample is obtained by refining the implicit prior sample via the truncated reverse diffusion chain.
Timme TDPM. Each
slowly, but the sampling time goes up linearly. When we use 99 steps of reverse diffusion, the FID of TDPM is better than DDPM with 1000 steps. Because the FID does not change much when we use more steps of reverse diffusion, we suggest using a small number of steps, such more, to balance the quality and speed of generation.
4.2 RESULTS ON
AND MORE DIVERSE IMAGE DATASETS
To test the performance of the proposed truncation method on high-resolution images, train TDPM using two different diffusion models, DDPM (Ho et al., 2020) and ADM (Dhariwal & Nichol 2021 backbones
datasets 256 256 resolution, LSUN-Church and LSUN- Bedroom (Yu et al. 2015). We compare the FIDs of TDPM with those of the backbone models and some state-of-the-ar GANs in Tables 2 The results show that TDPM can generate images of similar quality with much smaller truncation steps Tumee, which means that it can produce images significantly faster than the backbone models. We also visualize the samples from the implicit distribution XT.... ~ po (x that TDPM generates and the corresponding that it finishes at the end of reverse chain in Figure
We further evaluate TDPM on ImageNet- 1K (with resolution 64x 64) that exhibits high diversity. Here we adopt the TDPM+ configuration, where we use a pre-trained ADM (Dhariwal & Nichol 2021) checkpoint for t < Trrnee and train a StyleGAN-XL (Sauer et al. 2022) based implicit model at t Thunne (for simplicity, See Appendix D.6 for more details). We compare both FID and Recall with our backbone models in Table (and show example generations in Figure 2 Similar to our observations in Table [ TDPM has good generation quality with small truncation steps Timnc- Moreover, properly training an implicit model Trume can further improve the performance of the backbone.
choose to not use the progressive growing pipeline of StyleGAN-XL:
8

---

Published as a conference paper at ICLR 2023
Table 4: Numerical results Figure 6 The GPU time
LDM ILDM
sampling (s/image) measured on one NVIDIA A100.
40
20
CUB-Bird
MS-COCO
NFB
GPU
LDM
TLDM
LDM
TLDM
0.15
100.81
10.59
48.41
16.7
20
50
1.57
30.85
7.32
18.25
7.47
100
4.10
11.07
6.79
8.2
7.22
00.155
50.57.55
100 1960
250 (11 21:0 1800 (43.096)
250
11.21
6.82
6.72
6.3
6.29
Figure 6: Quantitative texl-to-image results (FID
1000
41.09
6.68
6.29
and GPU time) across different NFE.
00= 99)
NFF=50 49)
(T
A bird with brown wings, black back, and red head.
TLOM
A green train is coming down tracks.
Figure 7: Example text-to-image generation results of I.DM and TLDM (i.e., TDPM with LDM hackbone) finetuned CUB-200 (top row) MS-COCO (bottom row), setting the number of times iterating through the reverse diffusion U-Net as 100 (left column), 50 (middle column), or (right column).
4.3 TEXT-TO-IMAGE GENERATION
Besides unconditional generation tasks, we develop for text-to-image generation the TLDM. conditional version of TDPM that leverages as the backbone the LDM of Rombach et al. (2022). which a state-of-ther publicly released model with 1,45B parameters pre-trained on LAION- 400M (Schuhmann et al. 2021). LDM consists of fixed auto-encoder for pixel generation and latent-diffusion module connect text and image embeddings. Here we fine-tune latent-diffusion part on CUB-200 and MS-COCO datasets with 25K and 100K steps as the baseline. Similar to the unconditional case, we fine-tune with the LDM loss for < T Trunc and GAN loss details about the setting can be found in Appendix D.6
Thune- More
The results of LDM with different DDIM sampling steps and TLDM with different truncated steps Similar applying diffusion directly the original saage-pixellee space, when the diffusion chain is applied in the latent space, we observe TLDM can achieve comparable or better performance than LDM
summarized in Figure 6 and Table 4
though it has shortened the diffusion chain of LDM to have much fewer reverse diffusion steps. For the case that NFE is as small as we although the FID of TLDM has become higher due to using fewer diffusion steps, the generated image using TLDM at NFE=5 is still visually appealing, as shown in Figure Compared with 50 and 250 steps using LDM, the sampling speed of TLDM using steps 10 and 50 times faster, respectively, while largely preserving generation quality. We provide additional text-to-image generation results TLDM in Appendix D.8
5 CONCLUSION
In this paper. we investigate how to reduce the trajectory length of the diffusion chain achieve efficient sampling without loss of generation quality. We propose truncated diffusion probabilistic modeling (TDPM) that truncates the length of a diffusion chain. In this way, TDPM can use a much shorter diffusion chain, while being required to start the reverse denoising process from an intractable distribution. We propose to learn such distribution with an implicit generative model powered by the same U-Net used for denoising diffusion, and validate with multiple ways to learn the implicit distribution to ensure the robustness of the proposed TDPM. We reveal that TDPM can be cast as an adversarial auto auto-encoder with learnable implicit prior. We conduct extensive experiments both synthetic and real image data to demonstrate the effectiveness of TDPM in terms of both sample quality and efficiency, where the diffusion chain can be shortened to have only few steps.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
H. Zheng and M. Zhou acknowledge the support of NSF-IIS 2212418 and IFML.
REFERENCES
Guillaume Alain, Yoshua Bengio, Li Yao, Jason Yosinski, Eric Thibodeau- Saizheng Zhang, and Pascal Vincent. Gsns: generative stochastic networks. Information and Inference: Journal of the IMA, 5(2):210-24, 2016.
Abdul Fatir Ansari, Ming Liang Ang, and Harold Soh. Refining deep generative models via discrimi- nator gradient flow. In International Conference on Learning Representations, 2021.
Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial
networks. In International Conference on Learning Representations, 2017. URL https://
openreview net/ htttt/forumiccc 2im-nki gw5xe
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings the 34th International Conference on Machine Learning- Volume 70, pp. 214-223,
2017.
Jacob Austin, Daniel Johnson, Jonathan Ho, Danny Tarlow, and Rianne van den Berg, Structured denoising diffusion models in discrete state-spaces spaces. arXiv preprint arXiv:2107.00 107003006, 2021.
Mikolaj Binikoski,, Dougal Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. URL https
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians.
Journal of the American statistical Association. 112(518):559-8777 2017.
Florian Bordes, Sina Honari, and Pascal Vincent. Learning to generate samples from noise through infusion training. arXiv preprint arXiv: 06975, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
https:://penrevvee net /orrrummmm-
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Waveg- rad: Estimating gradients for waveform generation, 2020.
Jacob Devlin, Ming- Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/8100 181.00400, 2018.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https::/ ew net wwt.moormmmmo/// AANuCvzaVE
W Feller. On the theory of stochastic processes, with particular reference to applications. In Proceedings of the [First] Berkeley Symposium on Mathematical Statistics and Probability, pp. 403-432. University of California Press, 1949.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik Kingma. Learning energy- smsed models by diffuusio recovery likelihood. In International Conference on Learning Representations, 2021.
Xinyu Gong, Shiyu Chang. Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
10

---

Published as a conference paper at ICLR 2023
Anirudh Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Variational walkback: Learning a transition operator as a stochastic recurrent net. arXiv preprint arXiv:1711.0222,, 2017.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arX1v:2207.12598. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Matthew D Hoffman and Matthew Johnson. ELBO surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, 2016.
Chin-Wei Huang, Jae Hyun Lim, and Aaron Courville. A variational perspective on diffusion-based generative models and score matching. arXiv preprint arXiv:2106.02808 2021.
Ferene Huszár. Variational inference using implicit distributions. arXiv preprint arXiv: arXiv:1702.0823: 2017.
Aapo Hyvärinen and Peter Dayan. Estimation of normalzal statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers can make one strong
gan. arXiv preprint arXiv:2 02.07774 2021.
Alexia Marttinnu,,,, Rémi Piché-Tailee Taillefer, Rémi Tachet des Combes, and loannis Mitliagkas. Adversarial score matching and improved sampling for image generation, 2020.
Alexia Jolicoeur-Martineau Ke Li, Rémi Fiché-Taillefe Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 14080, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1401-4410 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.0666: 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2020b.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Score matching model for unbounded data score. arXiv preprint arXiv:2106.05527, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //axii
org/ org/abs/ 1412. 6980
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. 1rXiv:1312.6114, 2013.
arXiv preprint
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707 2021.
11

---

Published as a conference paper at ICLR 2023
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132. 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation. arXiv preprint arXiv:/809.000 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015,
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02338 023388 2021.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05641 2015.
Chenlin Meng, Jiaming Song, Yang Song. Shengjia Zhao, and Stefano Ermon. Improved autoregres- sive modeling with distribution smoothing. arXiv preprint arXiv:2103.1508, 2021.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint 1610v:1610..330 03483, 2016.
Radford M Neal. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, pp. 113, 2011.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2 100.09672, 2021.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN:
Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, 2016.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevae: Effi cient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint arXiv:2201.003308, 2022.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 15/ 2015.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelit images with VQ-V- Advances in neural information processing systems, 32, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conereaatoaalconfeeencccc on machine learning. pp. 1278-1286. PMLR, 2014.
12

---

Published as a conference paper at ICLR 2023
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention. Springer, 2015.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL htttttt://oppeeeee///
net/ Eorum? mert/ooumm LerrummIddTIddxXIp22..
Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion models. arXiv preprint arXiv:2104.0260)) 2021.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11918-1 11930, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 15:12488-12443 2020.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b. httptttsss//pppeeeviiieee wet.eiiwwwww.eeeeeeeeeeeeeeeeeee net//orum?
id==xXTG12RRHH
Jakub Tomezak and Max Welling. VAE with VampPrior. In International Conference Artificial Intelligence and Statistics, pp. 1214-1223 PMLR, 2018
Dustin Tran, Rajesh Ranganath, and David Blei. Hierarchical implicit models and likelihood free variational inference. In Advances in Neural Information Processing Systems, pp. 5523-5533, 2017.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in neural information processing systems, 2021.
Aaron den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, and Alex Graves. Conditional image generation with PixelCNN decoders. Advances in neural information processing systems, 29, 2016.
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- fion, 2367):1661-1674, 201
13

---

Published as a conference paper at ICLR 2023
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Be- longie, and Pictro Perona. Caltech-ucsd birds 200. Technical Report CNS-TR- TR-201, Caltech, 09 2010. URL /ssp://ww-cootettttt//paaaa//d// http:////eeeee///eeeeeeeeeeeeeee http//eeeindetatll...1 pdf, http: /www. vi visiviiioo sin.. ca ttcc. www....eeeeee.iii.....eeeee pedi a a/CB/ -0w--00-----000 html
CUB-200
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688.
Citeseer, 2011.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs, International Conference on Learning Representations, 2022. URL net htttps://cperreye wwtp:/eeeooonnndneeeeeeaaaa For-mmi-----::00000000---
Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Conference on Machine Learning, pp. 5660-5669, 2018
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scall scale image dataset using deep learning with humans in the loop. arXiv preprint 000.xv::5066.03000 2015.
Shengyu Zhao, Zhijian Liu, Ji Lin, Mun Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. arXiv preprint htfi::2006.103300 2020.
Huangjie Zheng and Mingyuan Zhou, Exploiting chain rule and Bayes' theorem to compare proba- bility distributions. Advances in Neural Information Processing Systems, 34, 2021.
14

---

Published as a conference paper at ICLR 2023
A PROOF
Proof of Theorem 1
As the last terms in both losses are the same, we only need to show that the first is smaller than or equal to Lo + Eimn Lr-1 in (8). Using Jensen's 's inequality, we have
term in
(xx) E qx xx)) log pa (xo XTemc)
p Xo
((xg( exxxxm xx) log ((x Xxx
9 1::iinm--
Xo: X.
pl Xx:TT
XTimn
s ((x) Ex(exeme (x0) Eo(e 1 xo.e log
(xx1:Tmm -1
Xg. XX
p(Xo:Time- -1) q(XTinm (x0)
Ex(o ) Txx (x0) log
[11Tii (x0) p( 1/XTTmx
E Exx F Thmn (x0) log pl p(xo:Timn-11 g(xx)) Fxtxrom (x0) log (XTrmm xx))
q(X1:Imme Xu)
p(x)mm))
Tiae L + Diimm) Linne EEeer
Trme
(15)
where the second to last equality follows the same derivation of the ELBO in Ho et al. (2020)
B RELATED WORK
Diffusion probabilistic models (Sohl-Dickstein et al. 2013 Ho et al.
2020) employ forward Markov
chain to diffuse the data to noise and learn the reversal such diffusion process. With the idea
of exploiting the Markov operations (Goyal et al. 2017; Alain et al. 2016 Bordes et al.) 2017).
diffusion models achieve great success and inspire a variety of tasks including image generation and Milicoeu-MMartncau Martineau et al. 2020; Vahdat et al.)
audio generation (Kong et al. 2020 Chen al. 2020 Jolicoeur
202I). Recently, plenty of studies have been proposed to generalize diffusion model to continuous
time diffusion and improve the diffusion models in likelihood estimation (Vincent
2011 Song &
Ermon 2020 2019
Nichol & Dhariwal 2021 Song et al., 20216 a Kingma et al.) 2021).
Another mainstream is improve the sampling efficiency of diffusion models, which are known for their enormous number of sampling steps. Luhman & Luhman (2021) improve diffusion processes with knowledge distillation Man-Sannoomaa et al. (2021) propose learnable adaptive noise schedule. Song et al. (2020) and Kong & Ping (2021) exploit non non-Markovian diffusion processes and shorten the denoising segments. Jolicoeur- Jolicoeur-Martineauu
al. (202T) and Huang et al. (202T) use better SDE solvers for continuous-time models. Aside from these works, recently other types of generative models such as VAEs (Kingma & Welling 2013). GANs (Goodfellow et al. 2014), and autoregressive models (van den Oord et al.] (2016) have been incorporated to diffusion models. They are shown to benefit each other (Xiao et al. 2022 Pandey et al. 2022 Meng et al., 2021) and have a closer relation to our work. Xiao et al. (2022) consider the use of implicit models (Huszár 2017: Mohamed 2016 Tran et al. 2017 Yin & Zhou 2018 Li & Malik] 2018) to boost the
Lakshminaray efficiency of diffusion models, where they deploy implicit models in each denoising step. which has higher difficulty in the training as the number of diffusion steps increases. |Pandey et al. (2022) build diffusion models on top of the output of VAEs for refinement. Our work is also related if viewing TDPM as a diffusion model on top of an implicit model, where the implicit model can be parameterized with the U-Net or separate network.
C DISCUSSION
Potential societal impacts: This paper proposes truncated diffusion probabilistic model as novel type of diffusion- generative model. The truncated part can be trained as implicit generative models such GANs diffusion probabilistic models are competitive to existing diffusion diffusion-baased based ones and efficiency is largely improved. On the contrary of these positive effects, negative perspectives could also be
GANs jointly or independently with the diffusion part. The capacities of truncated
15

---

Published as a conference paper at ICLR 2023
seen, depending on how the models used. One major concern is the truncated diffusion technique proposed in this paper could potentially be a way to hack the existing diffusion models if the implicit models are maliciously used to fit the intermediate steps. For example, for some existing diffusion models, for safety concerns, the model's 's capacity to generate private data needs to be locked by hiding the diffusion ending point into unknown distribution. The technique of TDPM could be used to crack these existing online diffusion models by providing intermediate noisy images or fine-tuning the first few steps with TDPM to unlock the capacity. Besides, the capacity of generating good images can also be misused to generate ill-intentioned images at a much lower cost.
Discussions: In this work, we mainly focus on reducing the length of the diffusion chain of a finite-time diffusion model. Our model has shown its effectiveness in improving finite-time diffusion models and it non-trivial to further explore our model on continuous-time diffusion models (Song et al. (021b). Moreover, while in this paper DDPM is the primary baseline, TDPM can also be built on other recent diffusion models. While po(XTmux)) is parameterized as an implicit distribution, it can also be formulated as semi-implicit distribution (Yin & Zhou| 2018), which allows it to be approximated with a Gaussian generator. Xiao et al. (2022) also present a closely related work. While we share the same spirit to reduce the length of the diffusion chain, these two strategies not conflicting with each other. In future work we will look into the integration of these different strategies. There also exists plenty of options in approximating po (x When truncating the diffusion chain to be short, the implicit distribution still faces multi-modal and needs to fit with different methods depending upon the properties that we need. For example, in order to capture all modes, a VAE would be preferred, like done in Pandey et al. (2022). Below we provide an alternative fit the truncated distribution. Besides the training, it's
method proposed in Zheng& & Zhou (202T) also an open question whether TDPM can be incorporated into more advanced architectures to have further improvements and we leave this exploration for future work.
D ALGORITHM DETAILS AND COMPLEMENTARY RESULTS
Below we provide additional algorithm details and complementary experimental results.
D.1 ADDITIONAL ANALYSIS ON THE PARAMETERIZATUS OF THE IMPLICIT GENERATOR
As shown in Section 3 in general, the objective of TDPM consists of the training of the diffusion model 6g (a U-Net architecture (Ronneherger et al. 20155) with simple loss of DDPM Liimlle and the training of an implicit prior model G. with objective CCAN Without loss of generality, in our main uue
paper, we show two configurations to parameterize the implicit part for t = Tuunc: 1) the implicit generator shares the same U-Net architecture used for < Trunc: 2) the implicit generator is instantiated with a separate network. Below we explain this two configurations (denoted as TDPM+ in the main paper).
Configuration 1): At =tt==7 Timmee the Unet generates the noisy image at the truncated step: XTe.e Timune + 1), where XTem++ +1 ~ NN I) the pure noise image whose pixels are iid sampled from standard normal. For f = Truney T Timee 1,.... 1, the same Unet iteratively refines the noisy images by letting X Vike (x: vT-ai 4-11 + Bize; Z ~ NO I). 21 = O, where
+1, t <XTm...
Et-1 Eo (x,, t) is the predicted noise by the Unet.
Under this setting, the Unet-based generator plays two roles the same time and the training will be more challenging than using two different generators here. However, we can also see as Tiue gets larger, the distribution of p(x))...) will become more similar to noise distribution, and generating the noisy images will be more like generating noises. In this case, being able to generate both noisy images and predicting noise becomes easier for the generator.
Configuration 2) (TDPM+): Unlike previous configuration, where the implicit generator at step t = T shares the same U-Net architecture with < Tinnc- Another way is to parameterize G. with a separate generator. Although this configuration increases the total parameter of the generative model, it allows the model has better flexibility in the training stage. For example, these two networks can be trained in parallel or leverage a pre- trained model. In our paper, Stylegan2 generator architecture Karras et al. (2000b) for = T Timne- resulting in an increase of 19M and 28M for the generator parameters when handling 32 x 32 and 256 x 256 images.
conduct the experiments by using
16

---

Published as a conference paper at ICLR 2023
The process of training and sampling of these configurations summarized in Algorithm and
Algorithm Training
Algorithm 2 Sampling
1: repeat
XTiux++1 NN I) if Gi. shared with cæ then
Xo ~v ((xe)
2:
3: fo Uniform((1, Timee)
3:
XTame = E0(XTmme Tane + 1)
4: te (0,, (,0.)).0 (0,
4: else
5: Update with (14)
XTym Gy (XTonct)))
until converged
end if
7: for / = Time:...., .1 do
8: 2. N(0, I) if 1, else x =0
9: X;-1 var
X+
VI-aa
1-
co ex(x + Bize
10: end for
11: return Xo
D.2 ALTERNATIVES OF LEARNING THE IMPLICIT DISTRIBUTION
Another possible statistical distance is based conditional transport (Zengg & Zhou 2021) which is proposed to balance the model-seeking and mode-covering behaviors when fitting an empirical data distribution. this setting, we use the same generator G as before, but instead we use a conditional distribution Ty parameterized by 17 to find an optimized mapping between the samples of p and q. and critic & to measure the point-to pointtto-point cost Co the feature space. The generator, the conditional distribution, and the critic are trained by the following objective C
discriminator,
min max E (XeT (e)o())) XTinn ) Col (XTmure (z(z))]
V.
+ Exmp((x) [Extem (oyt way,, (Xine (x)) Co Co G (z))
(16)
Similar to (14). fit TDPM-CT with following loss
C Lsimple_trine + XCCT
(17)
max:
We empirically find out this objective has no significant difference than using GAN objective shown in Equation 14 in performance- as long as the generator is well trained.
D.3 CONDITIONAL TRUNCATED DIFFUSION PROBABILISTIC MODELS
For conditional generation, extend (4 and derive conditional version of TDPM:
LC C simple_trrue + ALe. Timec
(18)
where Simple trunc aims to train the conditional diffusion model with Simple_trmm trune
Er, [llce co (xe, t)]]3]] t 1 ?... Tinne), NN0, I), (19)
and the truncated distribution L. can be fitted with GAN or CT:
min max
Ec [Ex=g(xxem (c) log Do(x + Ez-p(x) log(1 - (G |c]]]
(20)
min max Ee [x (c) [G. |x X o) Cal (XTomu: (z(z, W.n
+ Expp(n) [Exxemnx, (XTam Go (z.)))) (X G (z,)
(21)
D.4 ANALYSIS ON TOY EXPERIMENTS
Although we present image experiments the main paper, our studies were firstly justified our method on synthetic toy data proof of concept. We adopt representative 2D synthetic datasets used in prior works (Gulrajani et al. 2017; Zheng & Zhou 2021). including Swiss Roll, Double Moons, 8-modal, and 25-modal Gaussian mixtures with equal component weights. We empirical
17

---

Published as a conference paper at ICLR 2023
data
GAN
balulededs
Figure 8: A comparison of DDPM HH 2020 TDPM-GAN, and TDPM-CT Swiss Roll toy data. We show the effects of a truncated diffusion chain with length T = 2 and T = 5 (Ttrmm=1 and T4. The first row displays the true distribution from (Xx) ((xx-1). Each row below the first one represents the corresponding denoising distribution xx (x.). DDPM assumes p(xr)= = -0 I) and we can observe a gap between the true data distribution g(xr-1) and its generative distribution yo (xr-1). TDPM learns pe (xT..)))) ). which can be observed to well approximate the true q(xr-1). which helps the model successfully recover the clean data distribution (x0). Below each model, we report empirical KI. divergence between data and generative distributions as the quantitative metric. More results on different toy data can he found in Appendix D
xt).
sample set X, consisting of | 2, 000 samples and illustrate the generated samples after 5000 training epochs. We take 20 grids in the range [-10, 10] for both the x and y axes to approximate the empirical distribution of ps and q, and report the corresponding forward KL DxL (rllp)) as the quantitative evaluation metric.
Figure 8 shows the results on the Swiss Roll data. We present short chain with T = 2 and longer chain with T = 5 to show the impacts of the number of diffusion steps. The first row shows that the data distribution is diffused with accumulated noise, and with more steps the diffused distribution will be closer to an isotropic Gaussian distribution. As one can see, truncating the diffusion chain short length will result in clear gap between (XTTm.)))) ) and ((, T). When DDPM (shown in the second row) samples from the isotropic Gaussian distribution, it becomes hard to recover the original data distribution from pure noise with only a few steps. Although we can see DDPM can get slightly improved with a few more steps (T' 5), as long as ((x)) is not close to Gaussian, DDPM can hardly recover the data distribution. By contrast, as shown the third and fourth rows, TDPM successfully approximates the non- Gon-Sausiian g(xr,..) with its implicit generator, and we can see the remaining part of the truncated chain is gradually recovered by the denoising steps. From both visualizations and DRr. (p)), we can see that TDPM is able to fit every step in such short chains.
TDPM-GAN and TDPM-CT -T both succeed fitting po (XTumn)))) but the latter one fits slightly better when the diffusion length is 2. When the length increases to 5, fitting the implicit distribution with GAN becomes easier. This observation demonstrate a benefit of combining the diffusion models and GANs. If the implicit generator is sufficiently powerful to model g(XT...). then the number of steps in need can be compressed to a small number. On the contrary, if the implicit generator cannot capture the distribution, we need more steps to facilitate the fitting of the data distribution.
Shown in Figure 9 Figure II we can see 8-modal Gaussian is more similar to an isotropic Gaussian after getting diffused, thus DDPM 25-Gaussians, we can observe GAN does not suffer from mode-collapse and provide a better approximation than CT, which results in better data distribution recovery in the final step.
recover distribution similar data with 5 steps. On
18

---

Published as a conference paper at ICLR 2023
(19g)
True data
DDPM
Kipticl Kipttc 22631-1234
Rigtoll eogle5233
Kuchol Moul
TDPM-GAN
TDPM-CT
R5 26
Figure 9: Analogous results to Figure 8 usiin 8-modal Gaussian data.
(TK))
(TK)
True data
(600)
DDPM
TDPM-GAN
Midtollseoll 3.30
Adaa)
Pylita)
TDPM-CT
Saliige Up
co
Rid poo
Figure 10: Analogous results to Figure 8 uusing 25-modal Gaussian data.
19

---

Published as a conference paper at ICLR 2023
832
True data
po00
N00
SERO
DDPM
Kustejl stylt-2227
TDPM-GAN
kiptel [ste
Klenal
51 64
C1 Bsax
Righa Higiiitteleoor
Figure 11: Analogous results to Figure 8 using Double Moons data.
20

---

Published as a conference paper at ICLR 2023
D.5 ADDITIONAL ABLATION STUDIES
Using Pre-trained diffusion backbones: Different from the default setting, here we put the implicit model of TDPM+ trained at Tiaee and pre-trained DDPM mode in the same pipeline of sampling. In this case we do not need to spend any time on pretraining the DDPM model, and only need train the implicit model for Tirme. As shown Table [ pre-trained DDPM for t < Tuune. the generation performance of TDPM trained under this two-step procedure is comparable to TDPM trained end-to-end.
when combined with
Table Results of adding separately trained implicit generator a pre-trained diffusion model on CIFAR-10.
Model (t I Trane)
Model (f < Trann))
FID
TDPM+ (DDPM backbone)
2.88
pre-trained DDPM
2.85
TDPM+ (9mm:=99)
TDPM+ (improved-DDee D-MM backbone)
2.83
pre-trained improved-DDDI -DPM
2.25
TDPM+
2.94
pre-trained DDPM
3.05
TDPM+ (mm
TDPM+ (improved- MDPM backbone)
2.96
pre-trained improved-DDPM.
2.60
TDPM+ (DDPM backbone)
3.21
pre-trained DDPM
3.25
TDPM+ (T (Trrun=-))
TDPM+ (improved-DDee DDMM backbone)
3.17
pre-trained improved-DDD DDPM
2.95
Sensitivity to noise schedule: Nichol & Dhariwal (021 show the noise schedule affects the training of DDPM. Here we examine if TDPM is sensitive to the choice of noise schedule. We compare the linear schedule with cosine schedule, which adds noise in milder manner. The results on CIFAR-10 are reported in Table O which suggest that TDPM not sensitive to the choice between these two schedules.
Table 6: Ablation study with different noise schedules CIFAR-I0. The number before and after "/" denotes
the FID using linear and cosine schedules, respectively.
Model
Steps FID, (linear cosine)
TDPM-GAN
T Timme=99
3.10 / 3.47
TDPM-GAN
T Timum=49
3.30 / 3.16
TDPM-CT
T 999
3.69 / 3.62
TDPM-CT
T Tirnn=49
3.97 3.24
On the choice of truncated step: As the diffused distribution could facilitate the learning of the implicit generator G (Arjovsky & Bottou 2017), where we could observe by increasing the number of diffusion steps, the FID of TDPM consistently gets better. A natural question is on which step should we truncate the diffusion chain. We study the signal-to-noise ratio (SNR) different diffusion step. Based on (x) N((axxo 1 ā,l), we calculate SNR as
SNR
0 a
O
II 3)) 1=1
We visualize the SNR evolution across time step in Figure where we can observe the SNR rapidly decays in the first 100 steps. According to previous studies in Arjovsky & Bottou (2017). injecting noise into the data distribution could smoothen the data distribution support and facilitate the GAN training. The SNR change in this interval indicates injecting noise in the level of E [1. 100] could bring more significant improvement for the GAN training. When the step is greater than 200, the SNR is change is no longer significant and close to zero, which indicates the implicit model
'The pre-trained checkpoints provided by:
https: wttptttbccc.eeeeeee/rroooo
diffusion
21

---

Published as a conference paper at ICLR 2023
might not be too informative, though it is easier to train. Our experimental observations in Figure also justify this conclusion: when training a GAN at Tirune = 4, the required number of iterations is similar to training it on clean data; by training the GAN model Trume = 99, the training of GAN is significantly facilitated. For Trunc > 100, we empirically examine to train a GAN and find it would converge faster than training the diffusion model for & Trunn-
Comparison of model efficiency: In complement of the results in Table we provide detailed
model size and generation time on v100 GPU. The results are summarized in Table 7 We can see TDPM has an increasing in the total number of parameter, as it involves discriminator to help train the implicit model, while its sampling efficiency also obvious.
Table Comparison of model size (the added parameters corresponds to the discriminator model in the training but not involved in the generation), and GPU time in generation.
Resolution
32x32
64x64
256x 256
Model
Parameter
Time (s/image)
Parameter
Time (s/image)
Parameter
Time (s/image)
DDPM
36M
31.03
79M
33.01
114M
62.93
TDPM, Trun=
36M+20M
3.13
79M+2IM
3.52
114M+24M
6.65
TDPM, Tumee 499
36M+20M
1.52
79M+2IM
1.55
114M+24M
TDPM, T Tuun==4
36M+20M
0.16
79M+21M
0.26
114M424M
0.65
TDPM. Tumn=
36M+20M
0.03
79M+2IM
0.05
114M+24M
0.14
00
17.5
80
15
12
60
SNR
10
40
7.5
20
2.5
0-
0.0
200
600
800
1000
100
120
Steps
Steps
Figure 12: Signal-to-non noise ratio evolution across different diffuse step T. The right sub-pane! shows a zoomed-in SNR evolution in the range of [30, 140] steps.
D.6 EXPERIMENTAL SETTINGS
D.6.1 MODEL. ARCHITECTURE
Generator: Our generator structure strictly follows the U-Net structure (Ronneberger et al. 2015) used in DDPM, improved DDPM, and ADM (Ho al. 2020 Nichol & Dhariwal] [022] Dhariwal & Nichol, (021). which consists of multiple ResNet blocks (He et al. 2016) with Attention blocks (Vaswani al. (017) injected in the bottleneck. Please refer these paper for more details on the architecture.
key difference between our model and previous diffusion models is that our model also train such U-Net as extra implicit generator Gg that takes a latent variable z ~ (O I) and fixed time index t Truss as input. However, this does not result in difference in the generator architecture. We parameterize Go with the same U-Net architecture for simplicity and the time embedding / Tiums is specified to be trained with the implicit loss shown (12) and (T6). We have also tested to use all zero time embedding for t Timue 1 and found no clear differences.
For our results of TDPM+, the generator G. specifically takes a StyleGAN2 architecture Karras et al. (2020b) and there is time-embedding in Gy.. An increase of generator parameter appears caused by separating the implicit model and denoising U-Net. Note that the generator is trained with GAN loss and without specially designed adaptive augmentation in Karras et al. (2020a). For the detailed model architecture please refer to the corresponding paper or their Github repository: https: / / thub. com NV1 abs http:/stylegann22 ada -yttrchh
22

---

Published as a conference paper at ICLR 2023
Discriminator: Similar to Xiao al. (2022), we adopt the discriminator architecture used in Karras et al. (2020b). but without the time step input. The discriminator discriminate Xmm. is from the diffused distribution (XT....) implicit generative distribution (x Please refer to Appendix C Xiao et al. (2022) for the detailed design.
Navigator:
Training with CST involves an extra module named navigator (Zheng & Zhou 2021 We strictly follow the architecture used in Zheng & Zhou (202T), ) where the navigator is an MLP taking the pairwise feature distance inputs. There is no time embedding used in the navigator as it is only used for the training at ( = Trunc- The feature is extracted from the layer before the final scalar output. Please refer to their Appendix D for detailed information.
Architecture for text-to-image experiments: We adopt the 1.45B LDM model (Rombach et al. 2022) that is pretrained on LAION-400M dataset (Schuhmann et al. 202I). The LDM model consists 8 (resolution 256 -> 32), a U-Net in the latent space, and a BERT (Devlin et al. (2018) text encoder transform raw text to a sequence of 1280-dimension embeddings. We only fine-tune the latent model in our experiments. In the training of the truncated part, the discriminator takes the first-half of the U-Net (downsampling backbone) with linear predicting head on top of it.
of a U-Net KL-rregularized autoencoder with downsampling-face
The generator uses an architecture stacked with 4 linear layers with 128 hidden units. Each intermediate layer is equipped with time-embedding layer and follows softplus activation. The discriminator and navigator have the same architecture, without time-embedding layers, and using leakyReLU as the activation function.
Architecture for toy experiments:
D.6.2 TRAINING CONFIGURATIONS
Datasets: We use CIFAR- 10 (Krizhevsky et al. 2009), LSUN- and LSUN-Church (Yu al. 2013) datasets for unconditional generation in the main experiments. Additionally, we apply CelebA(Liu et al., 2013) and CelebA-HQ (Lee et al. 2020) for complementary justification. For text-to-image experiments, we use CUB-200 (Welinder et al. (0100) and MS- MS-COCO (Lin et 2014). The images consist of 32 x 32 pixels for CIFAR-IO. For the other datasets, apply center-crop along the short edge and resize to the target resolution (64 x 64 for CelebA; 256 x 256 for the others).
Diffusion schedule: For all datasets, we strictly follow the diffusion process used in our backbone models, and instantiate the truncated diffusion schedule by obtaining the first Tinme diffusion rates (31,..., BTm... J For example, if our goal is fit a model with NFE=50, to truncate the diffusion process used in Ho et al. 2020) (3) = 10-1, Br = 0.02, T=1000), we first initialize B1, 82, 81000. and then taking the first 49 steps complete the truncation.
Optimization: We train our models using the Adam optimizer (Kingma & Ba 2015). where most of the hyperparameters match the setting in Xiao et al. (2022), and we slightly modify the generator learning rate to match the setting in Ho et (2.20), as shown in Table g
We train our models using V100 GPUs, with CUDA 10.1, PyTorch 1.7.1. The training takes approximately 2 days on CIFAR-10 with 4 GPUs, and week on CelebA and LSUN-Church with GPUs
Table 8: Optimization hyper-parameter
CIFARIO
CelebA
CelebA-IIQ
LSUN 10-4
Initial learning rate for discriminator
10-4
10-4
10-4
Initial learning rate for navigator (if applicable)
10-4
10
10-4
10
Initial learning rate for generator
1 x 10-5
1 x 10-5
2 x 10-5
x 10-5
Adam optimizer B1
0.5
0.5
0.5
0.5
Adam optimizer (52 EMA
0.9
0.9
0.9
0.9
0,9999
0.9999
0.9999
0.9999
Batch size
128
128
64
64
# of training iterations
800k
800k
0.5M
244(bedroom)/ hatpe/ddrwomm//22Mccchch 2cchurch)
# of GPUs
4
8
8
For TDPM+, where we use StyleGAN2 generator as Gg. we directly use their original training hyper-parameters and train the model in parallel with the diffusion model. For TLDM, we set the base learning rate as 10-5 and the mini-batch size is set to 64. For the ImageNet K-64 64 experiments, use StyleGAN-XL -XL generator as G and strictly follow all the default training hyper-parameterd To simplify the implementation and save computation, instead of applying the default progressive growing pipeline 16 x 16 + 32 32 64 64, directly train the implicit model on 64 x 64
23

---

Published as a conference paper at ICLR 2023
images corrupted Trrune- Without using the progressive growing pipeline, the result of StyleGAN- XL shown in Table 2 clearly worse than the progressive one reported in their paper (FID 1.51). However, when used the implicit model of TDPM, the final performance of TDPM becomes competitive with this result.
When evaluating the sampling time, batch of 128 samples. When evaluating the FID, and recall score, following the convention, we use 50k generated samples for CIFAR-I0, LSUN-bedroom and LSUN-church, 30k samples for CelebA- HQ (since the CelebA HQ dataset contains only 30k samples), 30k samples for the text-to-image datasets. The recall scores are calculated with the recipe Kynkäääniemi et al. (2019). In the sampling stage, we follow our backbone to apply the same guidance in the diffusion part (t < T Trrune) applicable. Specifically, for LDM backbone, we use classifier- free guidance (Ho & Salimans 2022) with scale 1.5 and there are no DDIM steps for TDLM.
Evaluation:
models trained CIFAR-10 and generate
D.7 ADDITIONAL RESULTS ON UNCONDITIONAL GENERATION
Table 9:
Full comparison of unconditional generation on CIFAR-I0. 10. Models are grouped by the orders of sampling steps, with the best FID, and Recall in cach group marked in bold. The TDPM+ results are produced using ADM and StyleGAN2 backbone. TDPM with NFE=1 is equivalent training GAN with the DDPM architecture the generator.
Model
NFE
FID,
Rocall 1
Improved DDPM
(Nichol & Dhariwal 2021
4000
2.90
UDM Kim et al
2021
2000
2.33
Likelihood SDE
Song el al.
202
2000
2.87
Score SDB (VE)
Song cl al.
20216
2000
2.20
0.59
Score SDE (VP) (Song ct al.
2021b
2000
0.59
NCSN (Song & Ermon 2019
1000
25.3
Adversirial (Jolicour-
2020
1000
VDM Kingma et
1000
D3PMS Austin el 4121
1000
7.34
Diffuse VAB (Pandey et 2022
T=1000
1000
8.72
DDPM (Ho at 2020
1000
3.21
0.57
Recovery EBM (Ga
2021
180
9.58
Gotta Go Fast (Jolccocr-- Martincau et
2021
180
2.44
LSGM /Vhddal et al. 2021
147
2.10
0.61
Probahility Flow (VP) Song et 1b
140
3.08
0.57
DiffuseVAF. (Pandey et al. 2022
1100
100
11.71
TDPM, (ours)
100
2.97
0.57
TDPM+, =9m (ours)
100
2.83
0.58
FastDDPM, T=50 Kong & Ping 2021
50
3.41
0.56
DDIM, T=50 (Song et 2020
50
4.67
0.53
SNGAN-DGilow (Ansari el al. 2021
0.48
TDPM, T 49 (ours)
50
3.11
0.57
TDPM+, Tem:=49 (ours)
50
2.96
0.58
Progressive distiallation Salimans Ho. 2022
2.57
Denoising Diffusion GAN
al. 2022
1=8
4.36
0.56
Progressive distiallation Salimans
4
3.00
Denoising Diffusion GAN Xiao
4
3.75
0.57
TDPM, T Tm (ours)
5
3.51
0.55
TDPM+ T =4 (ours)
5
3.17
0.57
Progressive distiallation Salimans & Ho 2022
2
4.31
Denoising Diffusion GAN Xiao et at. 2022
2
4.08
0.54
TDPM. T (ours)
4.47
0.53
TDPM+, Tem.=l (ours)
3,86
0.56
DDPM Disiilltion (Luhiman & Luhman 2021
9.36
6.51
SNGAN Miyato et al.
2018
21.7
0.44
AutoGAN Gong el al.
2019
12.4
0.46
TransGAN (Jang al.
2021
9.26
StylcGAN2 wo ADA
Karras ot al
2020a
8.32
0.41
StyleGAN2 w/ ADA
Karrus el al 2020a
2.92
0.46
StyleGAN2 w/ Diffaug (Zha ct
2020
5.79
0.42
Progressive distiallation
Salimans 1lo
Denoising Diffusion GAN
14.6
0.19
TDPM, =0 (ours)
7.34
0.46
24

---

Published as a conference paper at ICLR 2023
TOPM GAN
TDPN CT
TDPM GAN
TDPMI
Figure 13: Qualitative results on CIFAR-I0 CelebA (64 64).
shutt shutts
Thuae 99
Thrme 49
TTrune-
Figure 14:
Qualitative results of TDPM on LSUN- (256 256), with Timae = 99, 49, and 4. Note NFE = Timee 1 in TDPM. Each group presents generated samples from pa (xx) (left) and pe (XTm.: (right).
Threee 99
Trrunc 49
TTrunc
Figure 15: Analogous qualitative results to Figure 14 on LSUN Bedroom. Produced by TDPM.
Tiruce
Tirune
Tiranc
Figure 16: Analogous qualitative results to Figure 14 on CelebA-HO. Produced by TDPM.
25

---

Published as a conference paper at ICLR 2023
Thuece 99
Trrunc 49
TTrune
Figure 17: Analogous qualitative results to Figure 14 on LSUN-Church. Produced by TDPM-CT.
Thrace 99
TTrunc 49
TTrunc
Figure 18: Analogous qualitative results to Figure on -SUN-Bedroom. Produced by TDPM-CT.
Thrune 99
TTrune 49
Tirenc 4
Figure 19: Analogous qualitative results Figure 14 on CelebA-HQ. Produced by TDPM-CT.
26

---

Published as a conference paper at ICLR 2023
D.8 ADDITIONAL RESULTS TEXT-TO-IMAGE GENERATION
A white and gray bird with black wings.
An airplan flying over body of water.
TDPM SUntinc
TDPM -
TDPIN
TD PM
edroßoc uroalo Te0
A sign reads "TDPM".
Busy city street at dusk with sun setting.
Figure 20: Additional text-to-image generation results with different text prompt, produced by TLDM
with T Ta = 49.
27

---

Published as a conference paper at ICLR 2023
A cluster of flower on the wooden table.
The bathroom has a big mirror.
The bagel is put in squre plate.
A clock tower near the river.
Figure 21: Additional text-to-image generation results with different text prompt, produced by TLDM
with Tanne 4.
28


Reviews:
[{"review_id": "fIG4C56JTKW", "reviewer": "Reviewer_6UUu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For faster synthesis in diffusion models, this work proposes a sampling procedure that simulates a truncated diffusion process. For example, if a standard diffusion model simulates a diffusion process for $t \\in [0, T]$, its truncated variant (which the authors call the TDPM) simulates $t \\in [0, T_{Trunc}]$, where $T_{Trunc} < T$. To then sample from this process, $x$ is drawn from $p_{T_{Trunc}}$ (rather than $p_T$), and then the diffusion is reversed to $t=0$ as in a standard diffusion model. Of course, $p_{T_{Trunc}}$ is usually not a simple distribution anymore and cannot be trivially sampled from; the authors thus choose to model it implicitly in a GAN-like fashion.", "summary_of_the_review": "The authors propose a hybrid GAN/diffusion model that trades off between the strengths and weaknesses of GANs and diffusion models. While the basic motivating principle has already been explored, I am on the fence about whether its simplicity and execution may warrant greater attention and discussion.", "strength_and_weaknesses": "Strengths:\n- TDPMs boast shorter diffusion times. \n- There appears to be little sacrifice in sample quality (in terms of FID) and mode coverage (in terms of recall).\n\nWeaknesses:\n- The ability to perform likelihood evaluations via the deterministic ODE framework is lost, as the $p_{T_{Trunc}}$ model is now implicit.\n- It is unclear how to choose $T_{Trunc}$, and how to control the trade-off between speed versus sample quality and mode coverage.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "f4wbyw-DALi", "reviewer": "Reviewer_vJWX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to improve sampling speed of diffusion-based generative models by minimizing the number of reverse steps. Instead of using a distillation technique, this paper truncates the diffusion process, stopping adding noise to samples before the samples become pure white noise. Then, an implicit generative model is employed to model this hidden-noisy distribution for sampling. Experiments on unconditional and text-conditional generation tasks show that the proposed method performs reasonably well in case of a limited number of reverse steps. ", "summary_of_the_review": "Though the idea combined with likelihood-based and implicit generative models is always appealing to me, the insufficient empirical justification makes me hesitate to accept this work. if the manuscript is updated in the revision phase, I reconsider my initial evaluation. \n\n", "strength_and_weaknesses": "**Strengths**:\n* The problem tackled in this paper is important both in academic and practical scenarios. And, the proposed method, combining diffusion process and implicit generative models, is also reasonable, since the hidden-noisy distribution after truncated diffusion process, seems to be a quiet unimodal distribution, which will be well estimated by GANs. \n* Experiments on unconditional and text-conditional generation tasks show that TDPM reasonably performs well even in the case of a very limited number of reverse steps. \n\n**Weaknesses**: \n* In my opinion, the main weakness of this work is insufficient comparison to previous work. To improve the sampling speed in the DDPM framework, there are several methods published in ICLR last year. This manuscript briefly discusses the limitation of DD-GAN and progressive distillation, but none of them are empirically compared to TDPM. So, this makes it difficult to evaluate the real value of TDPM in practical scenarios. \n\n**Detailed comments**\n\nAs shown in Eq.(14), the authors try to train a network by jointly optimizing the denoising objective and adversarial loss. This may hurt the stability of optimization for large-scale diffusion models. Instead of joint optimization, what about using this two-phase approach? First, the standard denoising objective is used to pre-train the diffusion model. Then, we truncate the diffusion process and train GANs to match prior to aggregated posterior. \n\nI failed to find insights from recasting TDPM as an adversarial auto-encoder, since this reinterpretation seems to be fairly straightforward. Honestly, even if the derivation is straightforward, it would be great to give many insights through the reinterpretation. \n\nIt would be much better to compare the proposed method with progressive distillation or DD-GAN. In my understanding, DD-GAN was not proven to be working well on ImageNet-scale datasets, but progressive distillation works well on many large-scale datasets. \n\nThere are some minor comments to the manuscript:\n* I couldn’t find \\mathcal{L}_{\\textrm{T}_{\\textrm{trunc}}^{\\textrm{CT}} in the main section. It first appears in the appendix. \n* Figure 5 shows some samples having some artifacts caused by watermarks in the training set. It would be better to include clean samples. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y_0jpBcIs1D", "reviewer": "Reviewer_MGCM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces truncated diffusion models, which truncates the forward process at certain timesteps so that when sampling, only a small portion of the reverse process is needed, which saves sampling time. To map noise to the noisy data at the truncated timestep, a conditional GAN is trained. The paper also builds up the relation between truncated diffusion models and AAE. Experimental results show that the model can generate similar or better samples than DDPM with a small number of NFEs. ", "summary_of_the_review": "I think this paper propose a novel and effective idea of speed-up sampling from diffusion models. The empirical results are also promising. Therefore, I think this is a good submission to be accepted.", "strength_and_weaknesses": "Strength:\n\n1. The idea is quite interesting. Previously people have observe when sampling from DDPM and do denoising for 1000 steps, the steps close to clean sample $x_0$ are more important to steps close to white noise $x_1000$. Some noise scheduling techniques are designed based on this observation, but this method takes an interesting approach to utilize this observation. It simply by-pass the early steps with a single step model such as GAN, and start sampling from diffusion models from a truncated time step.\n\n2. Although I feel like it is a combination of denoising diffusion GAN by Xiao et al. and traditional DDPM, but the benefit of such a combination is clear. Basically, the model can be interpreted as applying one step of denoising diffusion GAN, and using DDPM for remaining steps. However, it obtains better results, and it maintains the nice property of DDPM (while DDGAN is more like a GAN). \n\n3. Strong experimental results. It is nice to see we can even surpass the sample quality of DDPM with fewer sampling steps, while many previous speed-up methods have a degrade in sample quality.\n\nWeakness:\n1. Just like DDGAN, it loses some property of DDPM such as likelihood estimation, due to the GAN. \n\n2. Not necessarily a weakness, but I have one question. I am interested in the results of truncated at 4 steps. If you truncated only at 4 steps out of 1000 steps, is it almost equivalent to training a GAN (or DDGAN with T=1)? Because at t=4, the noisy sample $x_4$ is almost clean, and you need to train a GAN to map white noise to $x_4$. However, from DDGAN paper, it seems like DDGAN at $T=1$ does not work very well. I would be interested in knowing more details on this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HIjJQlKUvC", "reviewer": "Reviewer_YWr1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose TDPM, a truncated diffusion probabilistic model that essentially skips the diffusion steps by truncating the start/end of the process, stopping at an implicit non-gaussian distribution which can be sampled from another generative model. The goal is to reduce the number of steps without compromising image quality. The authors also related the proposed TDPM to an adversarial autoencoder AAE in the same way as how DDPM is sort of a VAE. Given T_trunc as the end point instead of T where T_trunc is a lot less than T, the discriminator is thus trying to discriminate between sample from this implicit distribution at T_trunc and from the generator. Interestingly, this generator can even be the same as the denoising network (by specifying appropriate t). The results seem convincing,", "summary_of_the_review": "Overall I really like the paper. The problem of speeding up diffusion is an important one and the paper does a good job of convincing the readers the validity of the proposed approach. The results seem to support the claim that the proposed method can speed up the process without sacrificing image quality. With that said, based on the fact that t_trunc=4 already provide quite good results already, I feel like it is mostly shifting toward being a GAN than a diffusion. Still, table 1 and 2 suggest that having diffusion even as small as 4 steps does provide benefit over pure GAN.\n\nOthers:\n\nL_CT in section 3.5 is not defined in the main paper, even though it does in the supplementary material. \n\nIs there any particular reasons for the choice of 4, 49 and 99 T_trunc?", "strength_and_weaknesses": "Strength\n\n-well motivated approach\n\n-well written paper \n\n-extensive analysis and discussion\n\n\nWeakness\n\n-the step to truncate T_trunc need to be decided at training time, and it's not clear what is the best T_trunc to set.\n\n-experiments could have been better, with more diverse and higher quality datasets. The main paper only shows cifar-10, LSUN-church, and LSUN-Bedroom. Something like ImageNet for example, could be more convincing.\n\n-introducing adversarial aspect also means potential training difficulty that comes with GAN\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster