ST-DDPM: Explore Class Clustering for Conditional Diffusion Probabilistic Models
Authors: Zhijie Lin
Zijian Zhang
Zhou Zhao
Venue: ICLR 2022 Submitted (2022)
Keywords: conditional generation
diffusion models
decoupling
interpretability

Abstract: Score-based generative models involve sequentially corrupting the data distribution with noise and then learns to recover the data distribution based on score matching. In this paper, for the diffusion probabilistic models, we first delve into the changes of data distribution during the forward process of the Markov chain and explore the class clustering phenomenon. Inspired by the class clustering phenomenon, we devise a novel conditional diffusion probabilistic model by explicitly modeling the class center in the forward and reverse process, and make an elegant modification to the original formulation, which enables controllable generation and gets interpretability. We also provide another direction for faster sampling and more analysis of our method. To verify the effectiveness of the formulated framework, we conduct extensive experiments on multiple tasks, and achieve competitive results compared with the state-of-the-art methods(conditional image generation on CIFAR-10 with an inception score of 9.58 and FID score of 3.05).

Under review as a conference paper at ICLR 2022
ST-DDPM: EXPLORE CLASS CLUSTERING FOR CON- DITIONAL DIFFUSION PROBABILISTIC MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Score-based generative models involve sequentially corrupting the data distribution with noise and then learns to recover the data distribution based on score matching. In this paper, for the diffusion probabilistic models, we first delve into the changes of data distribution during the forward process of the Markov chain and explore the class clustering phenomenon. Inspired by the class clustering phenomenon, we devise novel conditional diffusion probabilistic model by explicitly modeling the class center in the forward and reverse process, and make an elegant modification the original formulation, which enables controllable generation and gets inter- pretability. We also provide another direction for faster sampling and more analysis of our method. To verify the effectiveness of the formulated framework, we conduct extensive experiments on multiple tasks, and achieve competitive results compared with the state-of-the-e methods (conditional image generation on CIFAR- 10 with inception score of 9.58 and FID score of 3.05).
INTRODUCTION
Deep generative models such generative adversarial networks (GANs) Goodfellow al. (2014):
Zhu et al. (2017) y Brock et al. (2018), flows Rezende & Mohamed (2015): Kingma & Dhariwal (2018): Ho al. (201)), variational autoencoders (VAEs) ( Kingma Welling (2013): Maaløe et al. (019). score-based generative models SSog & Ermon (2019); Ho et al. (2020g: Song et al. (2020b) and autoregressive models Van Oord et al. () Oord et al. (2016); Salimans al. (2017) have been proposed to generate high quality samples in wide variety of data modalities. The based generative models involve sequentially corruptig the data distribution with noise and then learns to recover the data distribution based on score matching (e.g. the gradient of the log probility density Song & Ermon (2019); Song et al. (2020b), the noise corruption Ho et al. (2020)).
Given the Markov chain that gradually corrupts the data with noise, in this paper, first delve into the changes of data distribution during the forward process and visualize these behaviours. Specifically, measure the class clustering by considering the intra-class to inter-class variance ratio at each time step. Low values of the variance ratio demonstrate better class separation since the samples are concentrated around their corresponding class mean. We notice that in the early stage of forward process, the data distribution maintains stable intra-class variance and inter-class variance, leading a balanced variance ratio. As the noise gradually increases, the class separation decreases sharply and the data distribution is ultimately converted into the noise distribution, as shown in Figure (Left). The class clustering phenomenon reveals the process of diffusion where the corruption noise first corrupts the datapoints without increasing the variance ratio, and then greatly decreases the class separation and mixes the datapoints from different classes, which generates the noise distribution.
Inspired by the class clustering phenomenon, we formulate a novel conditional diffuuion probabilistic model by explicitly modeling the class center in the forward and reverse process, which enables controllable generation. Concretely, as shown in Figure T (Right), the early stage of forward process, samples gradually clustered to their corresponding class center while all the samples finally gathered into the noise distribution as the diffusion progresses. The reverse process is then modulated by the class center during generation:

---

Under review as a conference paper at ICLR 2022
=
T= 500
T= 800
1000
O
A1
forward
forward
A2
3
3
Noise
3
B1
3
3
B
B2
3
Figure I: Left: An example of the class clustering phenomenon. Right: The overview of the proposed conditional diffusion probabilistic model where A,, B, are samples from different classes, and A, B are the class centers.
By explicitly introducing the class center, we make an elegant modification the original formulation Ho et al. (2020) and further decouple the module for conditional information encoding and the module for noise prediction. The formulation that exploits the class center is much simpler since don't need knowledge to estimate the gradient of log likelihood Song et al. (2020b).
need to train separate model or apply heuristies and domain
2. The learned class centers, which get interpretability, also reflect the common characteristics of datapoints from the same class (e.g. template face, number).
3. With the explicitly-modede class centers, we can naturally speed up the reverse process with earlier starting, which different from strided sampling [Song et al. (2020a).
Due to the guided "shift" from sample to its corresponding class center during both the forward and reverse process, we name the conditional DDPM as ST-DDPM. To verify the effectiveness of the formulated framework ST-DDPM, we conduct extensive experiments on the task of conditional image generation, free-form image inpainting Yu et al. (0019) : Liu et al. (2018), attribute-to cetribute-to-imann to-image synthesis] Yan et al. (0166) and text-to-image synthesis Reed et al. 6)016); Xu et al. (2018): Zhang al. 22088), and achieve competitive results compared with the original formulation. We also provide another direction for faster sampling and further explore the relationship between the guided class centers and other score-based methods.
2 RELATED WORK
Score-based Generative Models. Score-based generative models Song & Ermon (2019) Ho et al. (2020); (Song et al. (2020b); Tae et al. (2021); Kim et al. (2021): Tashiro et al. (2021) involve sequentially corruptig the data distribution with noise using Markov chains and then learns to convert the noise distribution to the data distribution based on score matching. Score matching with Langevin dynamics (SMLD) Song & Ermon (2019) learns estimate the gradient of the log probility density with respect to data at each time step and samples from a sequence of decreasing noise scales for generation. Denoising diffusion probabilistic model (DDPM) Ho et al. (2020) trains the model with denoising score matching to predict the noise corruption at each step and then reverse the noise distribution during generation. Song et al. Song et al. (020b) further propose a unified framework to generalize prior works by introducing the stochastic differential equations (SDEs). There are other methods to learn the reverse process of Markov chains including infusion training Bordes et al. (2017), variational walkback Goyal et al. (2017). generative stochastic networks Alain et al. (2016) and others (Sllimans et al. (2015): : Song et al. (2017): Levy et al. (2017); Nijkamp et al. (001))
Lawson al. (011))
2

---

Under review as a conference paper at ICLR 2022
3 DENOISING DIFFUSION PROBABILISTIC MODELS
The DDPM Ho et al. 2020) employs a forward diffusion Markov chain to step-by-step convert a real data distribution into standard Gaussian distribution, and then build parameterized Markov chain to model the reverse process. Specifically, the forward process starts from Xo which comes from the real data distribution g(xu), and performs ' steps of diffusion generate X1 X2.".. XT sequentially. The forward trajectory is given by
I
((xo.)) = g(x0) II
g(x:(x(-1) = N(x: VI 1, 9.I)
(1)
where 31, 32, ... BT correspond to aforementioned diffusion rate of each forward step and are fixed variance schedule. The reverse process is conducted by parameterized operators px(x
px(Xo:T) p(x)) II po (xx)
(x (x,) (xxx- (); Ex( ())
where p(x))
N (xT;; 0, I). The DDPM is trained to minimize variational bound of negative log [- log po (x
likelihood Eg
denoted by:
po Xx- (x,) = L. log xx xx-1)
B (- log)o S Eag logg(xx)
Rewrite L (3) by Bayes rule establishes a connection between pa (Xt (x) and g(xx-1|xx, x0) for backward operators:
p(x)) Σ log
po (xx-1 log po (x (x1)
L Eg
log
(3)
(XT xo)
g [xx 1 Xx, xo)
T
= Eg
KL(g(xT xo) ((x)))
po (0(0((x-1\xx,x)))
log po x1)
where the forward process posterior conditioned upon xo is related to the forward process distribution via Bayes rule, denoted by:
(x xo) X1))
((x:
Fortunately, all right items have closed form expressions, which exactly derive Gaussian distribution, denoted by:
q(x:-1 xo) (x:-1; va--18t Xo
Varll āI-1) X1:
1 at-1
B,1),
a.
where az 1 3r and o = i 04 Consequently, the KI. divergences between Gaussians in T. 4 can be computed analytically. Further simplications and reparameterizatioos// Ho et al. (2020)
come to the following variant the variational bound:
Lgimple (0) Ex [ll colva /1 āzc, t) [1²]
4 MEASURING CLASS CLUSTERING RGB AND FEATURE SPACES
We first delve into the changes of data distribution during the forward process and visualize these behaviours. Concretely, we measure the class clustering in RGB and feature spaces by considering the intra-class inter-class variance ratio at each time step, given by
2
C Eig | /x//2
intra- intra-class class
(S)
N
E, || - 14l12
inter-de - class
3

---

Under review as a conference paper at ICLR 2022
=
=
intra
inter
250
500
750
1000
T 800
T 1000
inter - class
intra
0
250
500
750
1000
Forward process steps T
Figure 2: The changes of data distribution and class clustering in RGB space on MNIST dataset.
T = 500
T 1000
Oftra class of class
2-
250 500
750
1000
process steps
Figure 3: The changes of data distribution and class clustering in feature space on CIFAR-10 dataset.
where C is the number of classes, N is the number of data points, Zr. is the j-th sample in class i, μı is the mean of samples of class and μ is the mean across all the samples. Low values of the variance ratio demonstrate better class separation. We compute the variance ratio and visualize the data distribution on MNIST and CIFAR-10 dataset. Note that for better visualization, the feature vectors for CIFAR- 10 dataset are computed based on pre-trained ResNet18 backbone.
As shown in Figure 2 and Figure 3 given by t-SNE visualization, in the early stage of forward process, the data distribution maintain stable intra-class variance and inter-class variance, leading to balanced class clustering (T approximately from 0 to 800). As the noise gradually increases, the class separation decreases sharply and the data distribution is ultimately converted into the noise distribution (T 1000). The class clustering phenomenon inspires to explicitly introduce the class center class template during forward and reverse process controllable generation.
THE FORMULATION OF ST-DDPM
Inspired by the class clustering phenomenon, we formulate the conditional diffusion probabilistic model ST-DDPM by explicitly modeling the condition the class center in the forward and reverse process. Concretely, we introduce the condition encoding network to obtain the class center or class template as (c). where is the given condition (c.g. corrupted image, class label or attributes), and then rebuild the diffusion operators and modify the original recurrence relation Eq. given by:
g(,, X2-1. u) = -x(x:: Va,x,- + +a (1-a² Ju, B,I)
X; = Vaxx (1-a jut B.e
4

---

Under review as a conference paper at ICLR 2022
where we add condition-related item for Gaussian mean to make the forward process has "shift" toward the class center. With the recurrence relation, we can derive the general x sampling formula:
X₁ = v x, + a a u + vI ā,e, = at a + (1 - a; yul + vI ā,E
(7)
Algorithm 1: Training
repeat
X ((x))
(C)
1 Uni (1, 21,,....,
N(..I
Optimize || c- colvarx + -1- a,e, - "v-aa ||
until converged;
Algorithm 2: Sampling
Xy (0,1)) for T to do
E ~ (0, if 2 2,
else E 0 (x(x,, t)
X
X
8.
VT-ae
nr
m
+
return X0
See Appendix A.I for detailed derivation. An intuitive explanation of the proposed diffusion operators and the weight-schedulingg u can given as follows: Since ā, is closer to the early stage, the class-guides forward process first corrupts the sample while maintains the specificity of samples. As the diffusion procedures, the samples from the same class gather at the class center. As ây gets smaller and closer to 0, the data distribution is ultimately converted into noise distribution. In total, inspired by the class clustering, the well-designed weights firstly allow the specificity of datapoints, then gather datapoints from the same class for universality, and finally turn the distribution into pure noise by eliminating the impact of class means.
For convenience, we denote ai (1 - at Ju m, and aj (1 (1- a. )u = n. Then the forward process posterior distribution g(x, [x,, Xo, u) can be derived from probability density:
(x, [x,, Xo, u) (x (xc, xo. u), o21)
where
N4 [x(x,, Xo, u) Var(1 a
Bevar-1
āt-
of Br(1 Ū+ 1) (1 a:
x
/o:
See Appendix A.2 for detailed derivation. Then directly model parameterized p u, t) to predict jur (Xe, Xo, u). Similar to Ho et al. 2020), we further reparameterize Eq. (7) for e, N No I). then the forward process posterior mean becomes:
X
vI aico u,t)
B.
Me (x,,
u
X;
cx u, m
(10)
Vā
Vas
VT
ai
We then optimize the simplified loss to predict the corruption noise E:
Loimple 8) = Ex,x,.e [|] e ca (VVáx + n, + vI aie, u, t) [13]
(11)
Since we explicitly model the class center during the forward process, from Eq. 7 we have
Vo
(12)
VT-
where we can extract u out of parameters and use (x t) to predict the condition encoding module uş and the denoising module eg, making the final simplified loss Lsimpte (0, 0) becomes:
X Va vi-ai and decouple
Tisimpte (0, 0) = Ex,xo.6 (|6- {
n 60l colVaixo + n. + /1 o,E,1) /I ā
1
(13)
5

---

Under review as a conference paper at ICLR 2022
6
0
0
0
1
1
1
1
2
2
2
2
2
2
3
2 3 4 5 6
2 3 4 5 6 7 8
2 3 4 5
3
3
3
3
3
3
3
4
4
4
4
4
4
4
555
5
5
5
5
5
6
6
6
b
6
6
6
6
7
7
7
7
7
7
7
7 8
7
Trusk Hore frog
8
8
8
8
2
8
8
9
9
9
9
9
9
9
99
Class center
Figure 4: Left: Class-conditional samples on 28 MNIST and the trained conditional encoding
network ug (last row). Right: Class-conditional samples on 32x32 CIFAR-10.
Algorithm| 1 and Algorithm separrtely display the complete training procedure with this simplified objective and sampling procedure with the proposed ST-DDPM.
EXPERIMENTS
To verify the effectiveness of the formulated framework ST-DDPM, we conduct extensive experiments on the task of conditional image generation, free-form image inpainting Yu et al.) (2019); Liu et al. (2018) and text-to- image synthesis Reed et al. (2016); Xu et a (2018); Zhang et al. (2018). Following set T 1000 and set the diffusion rate increasing linearly from 3 = 10- to Br 0.02 for all experiments. For the denoising network cg. we use the same architecture as [o al. (020),, which is U-Net based
prior work Ho et al. (2020).
Wide ResNet since focus the
impact of our formulation. More experimental details can be found in the Appendix B
6.1 CLASS-CONDITTINAL IMAGE GENERATION
For class-conditional image generation, the con- dition refers to an one-hot vector corresponding to specific class label. To consturet the condi- tion encoding network ug, we map the one-hot vector initialized embedding and employ stacked convolution and upsample layers to pre- dict the class center.
Table 1: CIFAR-10 sample quality.
Model
HDJ
1St
Unconditional
NCSN Song & Ermon
2019
25.32
8.87
:: 0.12
NCSNVZISSo
10.87
8.40
0.07
DDPM Ho al 2020
3.17
9,46 0.11
Figure 4 (Left) presents the class-conditional synthesis results and the learned class centers on the 28 MNIST dataset. As shown in the
NCCNN+ cont. (deep. VE Song al.
2020b
2.20
9.89
Conditional
Projection Discriminated Miyato & Koyama 2018
17.5
8.62
FQ-GAN et al 2020
5,34
8.50
last row, the learned class centers also exhibit the shape of corresponding numbers, further in- dicating the class clustering phenomenon men- tioned in the previous section. Figure 4 (Right) presents the conditional synthesis results on the 32x32 CIFAR- 10 Krizhevsky et al. (2009) dataset, also verifying the capability and effectiveness of our formulated method for controllable generation. The results of negative log likelihoods are presented in Appendix C and more examples be found in the Appendix
BigGAN Brock grid. DIPM
14.73
9.22
4.25
9.18
cond. DDPM ST-DDPM
3.82
9.40
3,05
9.58 :: 0.09
To explore the sample quality, we compute the Inception scores and FID scores on CIFAR-10 dataset, shown in Table I For comparison, following (Song et al. (2020b), introduce time-dependent classifier and compute the gradient for class-condittonal sampling (grad. DDPM). Also, we build a simple conditional DDPM (cond. DDPM) by directly injecting class embedding along with the

---

Under review as a conference paper at ICLR 2022
PIN
Figure 5: Left: The progressive diffusion modulated by the class center us on CIFAR-10 10 dataset. Right: The progressive decoding with the same intialized noise modulated by different class centers us on CIFAR-10 dataset.
Figure 6: Left: The progressive decoding modulated by the class center up on 256x256 CelebA-HQ dataset. Right: More examples and corresponding class center Ug.
timestep embedding. Details are given in the Appendix B With the same denoising architecture as the unconditional DDPM Ho et al. (2020), the proposed ST-DDPM even obtains better IS score of 9.58 and FID score of 3.05. For class-conditional generation, our model achieves competitive results with the state-of-the methods, and better sample quality than most methods in the literature, indicating the effectiivnees of the proposed method.
As shown in Figure 5 we further visualize the progressive diffusion and decoding of ST-DDPM modulated by different class centers. The ST-DDPM explicitly models the class center during the forward and reverse process, making the class clustering more significant. Figure (Left) presents the progressive diffusion where the class clustering phenomenon is marked by the red bounding box. Figure 5 (Right) shows the progressive decoding toward different classes with the same initialized (Right)
noise, qualitatively verifying the capability of ST-DDPM for controllable generation.
6.2 IMAGE INPAINTING
Image inpainting Yu et al. (2019); Liu et al. (201)) aims synthesize contents for the missing regions of a given image such that the result is visually realistic and semantically correct. Different from class-conditional generation where the conditional encoding network is confined to limited and discrete space (i.e. the number of class labels), the condition space image inpainting is continuous and non-enumerable. To further prove the effectiveness of ST-DDPM that explicitly models the class center, we consider different images for imputation as different class and build a continuous conditional encoding network to generate the class center. Concretely, as the input condition and apply a U-Net based architecture to predict its corresponding class center, making the class centers enumeeallle The reverse process also starts from noise and is modulated by the predicted center.
denote the corrupted image
Figure separately presents the inpainting results on the 256 x 256 Place2 Zhou et al. (2017) J, 256 256 CelebA-HQ Liu al.) (2015) and 256x25. LSUN church Yu et al. (2015) dataset, which indicates the effectiveness and scalability of the ST-DDPM method. More examples can be found in the Appendix E
We also visualize the progressive diffusion decoding and the predicted class center in Figure 6 Notice that the decoupled condition encoding network has learned to predict the template face (i.e. Us)) based uncorrupted region, and the decoding process is modulated by the class center for inpainting, further demonstrating the effectiveness and interpretability of our proposed ST-DDPM. The further evaluation results can be found in Appendix C
7

---

Under review as a conference paper at ICLR 2022
Figure 7: Inpainting results with different initialized noise on 256x256 Place2 dataset (first row),
256x256 CelebA-HQ dataset (second row) and 256x256 I.SUN church dataset (third row).
SE
Homanrch
Manche
Figure 8: Attribueett--mmae synthesis results on the 64x64 64 LFW dataset. (Left: Image progression conditioned attribute (Interpolations on the attribute vectors). Right: Visualization of class centers us for test subset.)
6.3 ATTRIBUTE-TO- TO-MAGEE SYNTHESIS
Attribute-to-image synthesis Yan et al. 2016) re- quires to generate object image from level visual attributes (e.g. age, gender, lighting), which are also continuous values. Similarly, map the attribute vectors to class center (i.e. template face) by condition encoding network and leverage the proposed ST-DDPM formula- tion. By interpolating between the minimum and maximum attribute value to modify the value progressively, we can further explore the capa- bilities of ST-DDPM for image synthesis continuous condition space.
Saail 2-A1
Figure 9: Interpolations on the condition space for the 64x64 LFW dataset.
Figure presents the attribute-to-ine to-imae synthesis results on the 64x LFW Huang et al. (2008) dataset. As shown in Figure g (Left), samples generated by progressive condition are visually consistent with attribute description, indicating that our method can effectively explore the continuous condition space. Figure & (Right) visualizes the class centers predicted by the condition encoding network and the u also shows the shape of template face, demonstrating the interpretability of our method. Since decouple the condition encoding network and denoising network, explore the condition space by interpretation of two different conditions, given by À up + (1 shows the samples generated by interpolations the condition space. The interpretation smoothly introduces attributes from two different conditions, which can be extended to multi-condition generation in the future.
further
X) - (C22)) Figure 9
6.4 TEXT-TO-IMAGE SYNTHESIS
Text-to-image synthesis Reed al. (2066); Xu et al. (2018); Zhang et al. (011)) aims to generate realistic and text-consistent images according to the given natural language descriptions. We extend the ST-DDPM to the more challenging cross-modal generation by mapping the encoded word
8

---

Under review as a conference paper at ICLR 2022
500000
Ptp.p..... 5
Figure 10: Left: Speedup on the 32x32 CIFAR-10 10 dataset for different time steps. Right: Faster diffusion denoising on the 32x32 CIFAR-10 10 dataset with different time steps.
sequences to class centers (details in the appendix Also, we obtain Inception score of 4.52 and FID score of 18.18, demonstrating the effectiveness of our ST-DDPP method. More generated examples and visualization results of class centers can be found in the Appendix
6.5 FASTER DIFFUSION DENOISING
Different from strided sampling Song et al. 2020a), since we decouple the denoising network and condition encoding network and the diffusion process is modulated by the class center and datapoints from the same class gather earlier, as shown in Figure [ (Left), we can devise a way for faster diffusion denoising that starts from earlier starting points XT ny c, where e ~ NO I) (details in the appendix B By reducing the number of reverse steps, we enable faster sampling without harming quality as much possible.
Figure 10 presents the speedup and generated examples on the 32x32 CIFAR-10 dataset with different time steps. With 700, our method can achieve speedup of 1.43x and the sample quality is slightly decreased. With T = 900, our method can achieve speedup of 1.12x without harming the sample quality. Also, our technique earlier starting (ES) can be combined with strided sampling (SS) and achieve a speedup of 11.1% and detailed derivation and more results are given in the Appendix D
6.6 SCORE-BASED METHODS CONTROLLABLE GENERATION
With bayes rule, Song et al. (2020b) imple- ment controllable generation by introducing gradient guidance, which is given by extra time-dependent classifier or domain knowledge. Computationally, an advantage of the proposed approach over the guidance strategy in Song al. (2020b) is that backpropagation through classifier or other estimation is needed. The class centers, the guidance of ST-DDPM, are predicted in feed-forward manner and could be cached. To further explore the relationship. we separately visualize the normalized guidance during the reverse process. As shown in Fig- ure II. the learned class centers with weight-scheduling, a similar trend with the gradient of classifier during the reverse process. Notice that the learned class centers also have better class separation compared with the unstable gradient guidance.
CHA
nan-
0.0-
am
0.00
Jeno
Figure 11: The normalized guidance for different classes: Left: our learned class center and weight- scheduling. Right: the gradient of classifier.
guidance without extra estimation, show
7 CONCLUSION
In this paper, for the diffusion probabilistic models, we delve into the changes of data distribution during the forward process of the Markov chain and explore the class clustering phenomenon. To enable controllable generation, inspired by the class clustering phenomenon, we devise novel conditional diffusion probabilistic model ST-DDPM by explicitly modeling the class center in the forward and reverse process, and make an elegant modification to the original formulation. The learned class centers also reflect the common characteristics datapoints from the same class (e.g. template face). Extensive experiments verify the effectiveness and scalability of our method.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Guillaume Alain, Yoshua Bengio, Li Yao, Jason Yosinski, Eric Thibodeau-Lautee Saizheng Zhang. and Pascal Vincent. Gsns: generative stochastic networks. Information and Inference: Journal of the IMA, 5(2):210-249, 2016.
Navaneeth Bodla, Gang Hua, and Rama Chellappa. Semi-supervised fusedgan for conditional image generation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 669-683, 2018.
Florian Bordes, Sina Honari, and Pascal Vincent. Learning to generate samples from noise through infusion training. arXiv preprint arXiv: 069755533. 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint wxXiv:809.11096 2018.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233, 2021.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint wrXiv:1406.2661 2014.
Anirudh Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Variational walkback: Learning transition operator as stochastic recurrent net. arXiv preprint arXiv:1111.02282,, 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722-2730. PMLR, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Searned-Mille Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in Real-Life Images: detection, alignment, and recognition, 2008
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Score matching model for unbounded data score. arXiv preprint arXiv:2/06.05527, 2021.
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. arXiv preprint arXiv:1807.0380 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Dieterich Lawson, George Tucker, Dai, and Rajesh Ranganath. Energy-inspiree models: Learning with sampler-induced distributions. arXiv preprint 1910:11990.1265.. 2019.
Daniel Levy, Matthew D Hoffman, and Jascha Sohl-Dickstein. Generalizing hamiltonian monte carlo with neural networks, arXiv preprint arXiv:1711.09268, 2017.
Guilin Liu, Fitsum Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 85-100. 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Lars Maalóe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: very deep hicrarchy of latent variables for generative modeling, arXiv preprint arXiv: 090.. 022.0022222.22222 2019.
10

---

Under review as a conference paper at ICLR 2022
Takeru Miyato and Masanori Koyama. egans with projection discriminator. arXiv preprint arXivv1802.056637, 2018.
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Structure guided image inpainting using edge prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-based model. arXiv preprint arXiv: 19044.99000000 2019.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko- ray Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint arXiv:1606.05328, 2016.
Tingting Qiao. Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1505-1514, 2019.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lec. Generative adversarial text to image synthesis. In International Conference on Machine Learning, pp. 1060-1069. PMLR, 2016.
Yurui Ren. Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. Structureflow: Image inpainting via structure- appearance flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 181-190, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. International Conference Machine Learning, pp. 1538. PMLR, 2015.
Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv: 1602.0766, 2016.
Tim Salimans, Diederik Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. In International Conference on Machine Learning, pp. PMLR, 2015,
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprins arXiv:1701.055// 2017.
Jiaming Song, Shengjia Zhao, and Stefano Ermon. A-nice-me: Adversarial training for meme. arXiv preprint arXiv:/706.07566 2017.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint wrXiv:2010.02502 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprini arXiv: 9090005600 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.09011, 2020.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13450 2020b.
Jaesung Tae, Hyeongju Kim, and Taesu Kim. Editts: Score-based editing for controllable text-to- speech. arXiv preprint arXiv:2110.0258: 2021.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. arXiv preprint arXiv:2107.03502, 2021.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference Machine Learning, pp. 1747-1756. PMLR, 2016.

---

Under review as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv:1706.037760 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Min Wang, Congyan Lang, Liqian Liang, Gengyu Lyu, Songhe Feng, and Tao Wang. Attentive generative adversarial network to bridge multi-domain gap for image synthesis. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2020.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint arXiv:2106..00 038000 2021.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1316-1324, 2018.
Xinchen Yan, Jimei Yang. Kihyuk Sohn, and Honglak Lee. Attibute2image: Conditional image generation from visual attributes. In European Conference on Computer Vision, pp. 776-791. Springer, 2016.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:/506.000 03366, 2015.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018.
Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4471 4480, 2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang. Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions pattern analysis and machine intelligence, 41(8):1947-1962, 2018.
Yang Zhao, Chunyuan Li. Ping Yu. Jianfeng Gao, and Changyou Chen. Feature quantization improves gan training. arXiv preprint arXiv:2004.0200 020888, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): (066):4452-1 4444-41266666 1464, 2017.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2022
A EXTENDED DERIVATIONS
A.1 FORWARD PROCESS GENERAL FORMULA DERIVATION
x Vapxi-1 + at(1 at)u VBC.
(14)
= Vai Val + a
+ v 1 + as (1 (1- at)uu BLe
(15)
Va14-184-22
2 + Vana x (1 0 u you -1-
a² + BB4
(16)
Vi-4-1x--2 + ai (a 1- (1 o 1 at (1 0z ) u v + BrE
(17)
Vara--ix-2 + a (at -a a + a uu va a +1 065
(18)
1 az (1 aj a uu v 0,0%-16
1
(19)
(20)
Vāxxo a (I VI- Ú,6.
(21)
A.2 FORWARD PROCESS POSTERIOR DISTRIBUTION DERIVATION
a(x:-1 Xx, Xo. u) gxx, 1. (x- u) ((x, No: u)
(22)
(xx xo, u)
= -xxss varx- +
(23)
q(x--1 u) Nx + a (1 a - J, (1 5
(24)
q(x Xo. u) = Nxix V /,20 + a (1 - (1-a;)u ), (1- (1-áa:))
(25)
For the reason that all these multivariate Gaussian distributions have diagonal covariance matrix, we can treat and u as univariable, i.e., each element follows following derivation. By denoting
at (1 a u m, and at (1 a, u ny, we can derive q X1, Xo, u) as follow:
(x: -mxx---mm))
(x:-1 /ac-ja n1-1)
cxp
28
2 exp
2(1 (1-m) 1)
q(x1-1 11,, Xo. u)
xx n2 k1)
exp
(2m(1-a,)
(26)
E
exp
(27)
2
8:11-a--1)
2 2- a--1 (1)
āt-l
E
x + 1
+ m + 2 2 - 2m;x:]
(28)
0z
3r
+
x م
2
-
1nn-1X(-1
1 āj
(29)
B1 ā1-1)
[x] + xxx3 + nn + 2v
2 - 2n.x,]
(30)
(1 )
The coefficient of x²__ as 1- 04
T-a:
+
T-oc
B - so we try to transform E to the
1-a4
form of [x,-1 (ax; + bxo + +o)²
13

---

Under review as a conference paper at ICLR 2022
We extract the coefficient of xi as a²:
a
Qt-1
Be ④t-1)
(31)
1 - αz
(1 ō,) )
(1 (t-1)(
-(+) Be(1 x2-1)
(32)
- (1-a,)2
(1 -ي- (1 - 1 + o4)
(33)
(1 1 01)²
(1 0g )ox(1 (i2-1) 1)
(34)
a1)
0r (1 xx-1) ?
(35)
(1 0x)
Varl | Cq 1)
(36)
αt
We extract the coefficient of x as b2:
3,0r-1
Br(1 - 0 1
b²-
(37)
1- Ctt
(1 04))
3,00=-111
- (1+)) - 3e σt )ōt
(38)
(1 ā,)²
B, [11 - [1) (1 a
(39)
(1 (1-a)) àz,²
3r0t:-1
- 0t - a2 + ārl
(40)
(1 - a,)²
fāa-1
(41)
(1 a,)²
b + Biya 01-1
(42)
δi
The coefficient of x, 1x, is -2 -
2a, which means that a (00((1-80-1)
1-6e
1-
The coefficient of x, 1Xo is 2
B,yar-1
= 2b, which means that = B.vā.-1
l-āt
The coefficient of x;X0 is 2 2 01-1
1) = 2 yaiy T8,(1- 1) which is equal to 2ab.
(1-a,)²
(1-0,)²
Note that m, n, ai
Vonn;-1
of - a/)u vara 1
(43)
at (1 - - 0 o
a u
(44)
a (1 - j a (a) a m
(45)
at (1 of - of + af)u
(46)
1
a, (1 a )
(47)
m
(48)
14

---

Under review as a conference paper at ICLR 2022
The coefficient of x should be equal to 2c:
1 - 01-1
m, - 2 3e n:-1
2/a:
(49)
1 0q
1 - 0t
1 ac-1
Br
-2
0:
(n:
varlh-1) - 22
11=11
(50)
1 0lt
-2/at
1 0t-1
n2
201
0t-1
n -
3e
n
(51)
1 Ōi
1 - δ2
1 0t-1 n1
a [1 0t ) + +1
-2
0t
11-1
(52)
at
1 d(-1 I1
aq 02 1 a1
=2 01
11-1
(53)
1 āt
1 ūt-1
-2 0t
n2 2n
(54)
ā1-1
=2
a 1- at
n n-11
(55)
=2c
(56)
- av-1 I1 n
c=var
(57)
1 0t
(58)
The coefficient of x, should be 2ac:
ā1-1
m +2 3r(1 - -ār-1)
2
(59)
1 āt
(1 -1-60 )2
Øt-1
(nz (atil-11) 1) + 2 Bi(1 - 0
2
(60)
1 āt
(1 - )
B1 - a1-1)
1- à
01-1 I1
2
It + +2 at
(61)
(1 0,)²
1 az
1- at
81 - ōt-1 1) - (1 - (1) a(-1)
04-1
-2
(1 - 0z) 1
I1. + +22/a
1
n1
(62)
(1 dz (3c - + an)
1 - 01-1 n1-1
-2
D2 + 2 2/a1
(63)
(1 (1- 1-aa)2
1- āi
(1 σ )) 1-00++ + k1)
1- 01-1
-2
n2 + 2-/aa
0:
I1:
(64)
(1 02
1- at
(1 04 )))(( 01-1)
1 - i4-1
-2
n2 + +220i
n1-1
(65)
(1- -2))) 2
1 āt
(1 12
1 - ā6-1 11
-2
n1 + 2/a
(66)
(1 )z)² 2
1 a
- āx-1
Vas(1 [1 - az-1)
2
V ar
n1 D
(67)
1 σt
1 att
-20a
(68)
The coefficient of xg should be 2bc:
Bc(1 5(-1)
Be
a/
n1 + +-
as n1-1
(69)
(1 - ag
2-2 Biyāl-1
āx-1 va 1 āe
nr 1 1
(70)
0t
-20c
(71)
15

---

Under review as a conference paper at ICLR 2022
The remaining items should be equal to c²:
0t-1
m
+
31
n
8[(1 a1-1)
2
(72)
1 - ōe
1 04
(1 0z)²
04-1
8: ni
Br(1 (4-11)
=
1 - āt
(n.
+
1 - āt
(1 - xx)²
n
(73)
0t-1
B1 0
0t-1
3
ūt-1
ni +
at
+
n.
2/a
n1n--11
(1 - ā1)
1 - āt
1 - āz
1- ōq
(74)
(1 a1) ax-1)
3r(1 ar-1)
a4 at +
ac-1
n +
n 2/a 02
n114-1
(1-00 (1-a,)²
1 aq
(75)
(1 02- (1 - 02
Be)
a4 a + az
1 1 a4-1
n
n - 2/
n1
(76)
(1 - k²)² )2
- αt
(1 - az-1) (1
1 a()
1 01-1 1111-11
n1. + ni-1 - 2a
2
(77)
(1 āt)²
(1 02 )(rp( --p- +1
n. + n
2
2/00 0+
11,14-1
(78)
(1 -,,)
1 a+
(1 a(-1)
1 - āz-1
=0;
n1; mi-1
2
2
2/ar a1 αt
11111111
(79)
(1 at) )2
ā:-l
m.
(80)
(81)
Finally we have a var(1-ar-1)
b Deyār-1
c=
1-8 1-a; 1-an
n. -n111 and
1-as
1-ū;
E [x--1 + (ax; + bxx + c))²
(82)
X votel Cz.
BI Bivor-1
1 ōt-1
Xx
xo
+ Vol
ng D(-1)
(83)
Ct.
a2
Now we have the forward process posterior distribution as follows:
1
(x1-1 - 11)
(x(-1 /x,, X0, Xo, u)
v fm
B
1-ac
or
exp
2 8=11-a
(1- -c)
V2TO
exp
2g²
(84)
which is right probability density function of some Gaussian distribution. Then we have:
Bc(1 ā(-1) o²
(85)
(1 a.)
16

---

Under review as a conference paper at ICLR 2022
voi(1 01-1)
1
1 a(-1
14
X1 +
Xg
Val
Dt + D1-1
(86)
1- 0i
1 1- at
Var(1 Ōr-1)
31vā x 11 v1 Ō
1 - 0t-1
+
Var
nr + n
(87)
1 04
Vox
1 Ōt
var(1 - ā
X4 +
3
X1
nt
VI 046
Vai
1 - 0/-1
+ 11-1
(88)
1-ar δe
1 1-4
Vau
1- āt
[ai(1 - (((1-((1)) x
3r
BIV1 ā1
31
ar(1 1
+
X1
11; + n
(1 - 1aa))/oo
(1
04))
(0v(11 - (4t)
(1 o4
(1 - 24) Voa
(89)
01 āe + 31
Be
31 + 01 at
n1 + n-1
(90)
(1 - ār) /ot
Vav 1
(1 k1) /0
1
B.
1
X4
n1 -1
(91)
Vat
val
1 a1
Vat
1
11
1
X1
c
(n
1a;n
(92)
Val
varv
1
04
Val
1
3r
Xt
Vat m
(93)
Voi
varv
1
at
1
B1 C ms)
(xt-
(94)
Vou
v1 αt
B IMPLEMENTATION DETAILS
Follows the backbone of PixeICNN+ Salimans et al. (017)) and DDPM Ho et al. (020)). our employ a U-Net architecture based on Wide ResNet for noise prediction Gæ and replace the weight normalization Salimans & Kingma (2016) with group normalization Wu & He (2018) to make the implementation simpler. We use four feature map resolutions for 32x32 models, and six resolutions for 64 x 64 and 256x256 models. All models have two convolutional residual blocks per resolution level and self-attention blocks at 16x 16 resolution between convolutional blocks. The time embedding is then specified by adding the Transformer Vaswani et al. (2017) sinusoidal position embedding into each residual block.
To predict the class centers [ig. we stack convolutional residual blocks and upsample layers to map feature vectors to three-channels map for CIFAR-10 and LFW dataset. For image inpainting on CelebA-HQ. LSUN-church, Place2 datasets, also employ a U-Net architecture for pixel-to-pin tixel prediction. For text-to-image synthesis on CUB bird dataset, we stack convolutional residual blocks and upsample layers with attention mechanism from the pre-trained word embeddings to predict the class center.
For image inpainting, we use the Irregular Mask Dataset collected Liu et al. (2018), which contains 55,116 irregular raw masks for training and 24,866 for testing. During training, for each image in the batch, we first randomly sample mask from 55,116 training masks, then perform some random augmentations on the mask, finally we use it to mask the image and get our class center for training. So the training masks are different all the time. The mask is irregular and may be 100During testing, we use 12,000 test masks sampled and augmented from 24,866 raw testing masks. These 12,000 masks are categorized by hole size according to hole-to-image area ratios (0-20%, 20-40%, 40-60%).
Follow the training setting of DDPM Ho et al. (0220). we set T 1000 and set the diffusion rate increasing linearly from 81 = 10-4 to 37 = 0.02 for all experiments. The dropout rate is set to 0. We use the Adam optimizer, and set the learning rate to 2 x 10 for x2x32 32 images and 2 x 10-5 for higher resolution without any sweeping. The batch size is set to 128 for CIFAR-10, 64 for LFW, and 24 for high-resolution Places2, LSUN-church CUB- and Celea-HQ datasets. We use EMA on model parameters with a decay factor of 0. 0.9999. We train the 32x32 models with four Nvidia RTX 2080Ti GPUs and higher resolution models with four A6000 GPUs.
17

---

Under review as a conference paper at ICLR 2022
Table 2: CIFAR-I0 results. NLL measured in bits/dim.
Model
FID
NLL Test (Train)
Unconditional
Gated PixeICNN 65.93
3,03 (2.90)
DDPM
3.17
$3.75 (3.72)
Conditional
cond. DDPM
3.82 3.05
$3.75 (3.72)
ST-DDPM
$3.74 (3.69)
Table 3: Evaluation results on the Place2 dataset.
FID.
Mask
0-20%
20-40%
40-60%
CA Yu et al. EdgeConnect Nazen et al. StructureFlow Ren al. cond. DDPM ST-DDPM
2018
4.8586
18.4190
37.9432
2019
3.0097
7.2635
19,003
2019
2.9420
7.0354
22.3803
2.0401
6.5726
17.2834
1.7498
6.1887
14.5111
For comparison, following Song et al. (202bb): Dhariwal Nichol (2021), 1, we introduce a time- dependent classifier and compute the gradient for Lass-conditiona. sampling (grad. DDPM), given by:
E Ey (zt. vI āi va, log (o(y(
(95)
where & is the parameters of the pretrained classifier, y is the class label. Also, we build the simple conditional DDPM (cond. DDPM) by directly injecting a class embedding along with the timestep embedding into the denoising network, given by:
= c0 c,
(96)
where c the given condition (e.g. class label).
C MORE EVALUATION RESULTS
We measure the negative log likelihoods (lossless codelengths) CIFAR-10 dataset, given in Table 2
Further, evaluate the model on image inpainting, follow prior works Yu al. (2019); Liu et al.) (2018) by reporting the FID) score on Place2 dataset. Table 3 shows the comparison results. The ST-DDPM achieves competitive results comparable to prior GAN-based methods for the FID score, showing the great potential of our method.
Table 4: Evaluation results of eox-tto-mmage generation.
Model
FID
ISt
StackGAN Zhang et
2017
51.89
3.70
StackGAN+ (Zhang ct 2018)
15.30
3,82
FusedGAN Bodla el al. (2018)
3.92
AttnGAN Xu al. 20
4.36
MirrorGAN (Qiao et al. (2019)
4.56
AGAN-CL Wang al. (2220)
4.97
ST-DDPM
18.18
4.52
18

---

Under review as a conference paper at ICLR 2022
D FASTER SAMPLING WITH STRIDED SAMPLING
Given the Eq. we can get a transfer relation as follows:
v X0 + n, + B.Et
(97)
As t increases, vāa get closer to zero, leading to an approximate transfer relation as follows:
BBe
988)
Therefore, we provide another thought for faster sampling that is to select a carlier starting timestep.
Also, inspired by DDIM Song et al. (2020a) with strided sampling, into similar non-Markovian inference processes, which enables new and fast generative processes. Specifically, we introduce family of inference distribution indexed by R
adapt formulations
T
4a (x) = do xo)
II
go (xt X0)
(99)
where
2o (x) -N(x;; varxo nr, (1 II)
(100)
and for all t >>
lo (x,-111x,, xo) = N(x-11 Voi -0 V āc-1 07. X
VoxX0 ar ,o,1). (101)
They designed to ensure that (- (x, = NXxei /12o nt, T.) for all t.
Then we employ a trainable generative process po (XD:r) where each po (x models the reverse
conditional distribution 9. (xx-111 X,, Xu). Specifically, for some zo q(xo) and e, NN I), can obtain T3 using Eq. 7 and train a model ea to predict e, from o without zn. We can then predict the denoised observation So (xx, 1) with éy:
é co(xi,t)
(102)
vI ae
Xt 11.
át
fo(x,,
(103)
Vo
Then we can define the generative process with fixed prior po px(x)) {
N(xx;; 0, I) and
N(xo: (x o)) tf Go (x--1 fo (xx, t)) otherwise,
Po po(x
(104)
from which one can generate sample x from via:
X = Va--1 n. or b + + V --- - of è + oec. (105)
0
We do these modifications to ensure the optimal solution for ST-DDIM is also the same as that for ST-DDPM, which enables to adopt other forward processes with smaller steps for accelerated sampling without having to retrain the model. We take the same settings with DDIM for and o, &r. is
where r is sub-sequences of [1.... .T) and σr (1) // a (1((11))))) an) ār
hyperparameter that we can directly control.
The evaluation results of FID on the CIFAR-10 dataset with strided sampling (SS) and earlier starting points (ES) with different 7 are presented in Table With earlier starting and strided sampling our method can achieve a speedup of 11.1x without significant decreasing in sample quality. Compared with existing works Watson et al. (2021) (Song et al. (2020a), method can also achieve competitive evaluation results with the same number of steps. Also, more generated examples with different
and sampling steps T are presented in Figure
19

---

Under review as a conference paper at ICLR 2022
Table 5: Evaluation results of FID on the CIFAR-10 dataset for ST-DDPM with strided sampling (SS)
and earlier starting (ES), and the complete sampling step T = 1000.
SS
ES + SS
Step
10
20
50
100
300
90
270
0.0
13.69
6.42
4.64
4.29
4.09
4.45
4.10
0.2
14.84
7.54
4.89
4.68
4.18
4.62
4.33
7
0.5
15.40
8.88
5.81
4.88
4.25
4.88
4.53
1.0
27.67
16.23
8.88
6.27
4.54
6.25
4.61
10
T-100
--0.0
0-0.
p005
Figure 12: More generated samples with different sampling steps.
E ADDITIONAL SAMPLES
Figure 13 15
16
17
18 present more generated examples separately on the CIFAR-10 Krizhevsky
al. 2009p. CelebA HQ
Liu et (2013), LSUN-church Yu et al. (2013), Place2 Zhou (2017)
and LFW Huang et al. (2088) dataset. We also visualize the class center in the second column in
Figure 15
16
Figure 14 shows the generated examples and class centers on the widely-used x56x256 CUB
bird Wah et al. (20T)) dataset. Notice that the class centers are also bird-like templates, demonstrating the effectiveness of our method.
20

---

Under review as a conference paper at ICLR 2022
(a) Plane
(b) Car
(c) Bird
(d) Cat
(c) Decr
(f) Dog
(g) Frog
(h) Horse
ii Ship
(i) Truck
Figure 13: More condiconniiional samples on 32x32 CIFAR-IO. -00.
21

---

Under review as a conference paper at ICLR 2022
This bird has brown crown, a yellow belly, and pointed bill.
This is blue with black and has a long, pointy beak.
is completely black with short blunt bill.
with a large beak, brown wings and white neck and belly.
Ground truth
ST-DDPM
A small sized bird that is The small bird has white very large bird with yellow curved beak, and gray wings.
small bird with grey breast, short black bill and brownish red crown.
mostly yellow that has black chest marking
belly, breast and through along with short beak.
Ground truth
ST-DDPM
A black crown and lighter The bird has a bright yellow back and feathers this bird color on it's breast, belly, has long narrow beak. abdomen.
This bird has yellow belly with brown wings and brown crown.
This colorful bird has bright blue wings and tail along with a long, pointed beak.
Ground truth
Center H&
Figure 14: Text-to-image synthesis results on the x56 CUB bird dataset.
22

---

Under review as a conference paper at ICLR 2022
I
I
I
TP
Figure 15: More generated samples on 256 CelebA-HQ dataset.
23

---

Under review as a conference paper at ICLR 2022
G EiLL
ShL
SPL
Figure 16: More generated samples on 256 LSUN-church dataset.
24

---

Under review as a conference paper at ICLR 2022
Figure 17: More generated samples on 256x256 Place2 dataset.
25

---

Under review as a conference paper at ICLR 2022
Pg(G)
1 0.1
== 0.2
0.3
== 0.4
= 05
=
= 0.7
== 0.8
A 0.9
(g(C))
U
Figure 18: More interpolation samples on 64x64 LFW dataset.
26


Reviews:
[{"review_id": "6ObJaHK93K8", "reviewer": "Reviewer_HJtB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe conditional framework proposed in this paper is general and is applicable to different tasks. The learned condition embedding contains some semantic information.\n\nQuestions:\n1. The earlier starting (ES) is only slightly mentioned in Sec.6.5. Although the author claims that details can be found in Appendix B, I didn't find how it is derived and what is the principle. Besides, it seems that only using strided sampling (SS) can work even better than combing ES and SS, e.g., Table 4 in Appendix shows that SS with 100 steps can achieve a FID of 4.29, which is better than 4.45 of ES+SS.\n\n2. Is the decoupling of $\\epsilon_\\theta$ reasonable? According to Eq.(11) and Eq.(12), the optimal $\\epsilon_\\theta^*(x_t, u, t)=E[\\epsilon|x_t]=E[\\frac{x_t - \\sqrt{\\overline{\\alpha}_t} x_0 }{\\sqrt{1-\\overline{\\alpha}_t}}|x_t] - \\frac{n_t}{\\sqrt{1-\\overline{\\alpha}_t}}$. The first term is still related to $u$, since the joint distribution of $x_0, x_t$ is $q(x_0|c) q(x_t|x_0,c)$, which is related to $u$. However, the author models the first term with $\\epsilon_\\theta(x_t, t)$, which is independent of $u$.\n\n3. Algorithm 1 and Algorithm 2 don't show how to get the label $c$. Besides, in Algorithm1, shouldn't $u=u_\\phi(x_0)$ be $u=u_\\phi(c)$?\n\n4. At the beginning of Section 5, the author denotes $u_\\phi(c)$ as the class center. However, $u_\\phi(c)$ is actually learned, which is not rigorously the class center. It is likely to cause misunderstanding.\n\n5. As mentioned earlier, the optimal $\\epsilon_\\theta^*(x_t, u, t)=E[\\epsilon|x_t]$. What will the optimal condition embedding $u_\\phi(c)$ be?\n\n6. I didn't find a strong relationship between the clustering phenomenon and the formulation of ST-DDPM. How the phenomenon motivates ST-DDPM should be discussed in detail and formally.\n\n7. To distinguish this paper from prior conditional methods, the strengths than the traditional conditional DDPM (i.e., the cond. DDPM in Table1) should be discussed more.\n\n8. This paper is about conditional diffusion models, thereby it would be better if experimental comparisons with prior conditional diffusion models (e.g., [1*, 2* ,3*]) are added. The author should at least discuss these works in the related work section.\n\n9. Some missing experimental details. For example, what is the batch size?\n\n[1*] Improved Denoising Diffusion Probabilistic Models\n\n[2*] Diffusion Models Beat GANs on Image Synthesis\n\n[3*] ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models\n", "summary_of_the_paper": "The paper proposes a new forward conditional process. The transition kernel is controlled by a condition embedding. It also derives a reverse process. The $\\epsilon_\\theta$ module is decomposed into two parts. The paper conducts experiments on different conditional generation tasks.", "summary_of_the_review": "This work has some questions, which should be addressed to reach the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2SCAT-wxvaa", "reviewer": "Reviewer_FjT6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength - \n1) The idea of doing controllable generation from diffusion models across different tasks by efficiently conditioning the model is interesting. \n2) The paper shows strong conditional image generation results. Qualitative results for the image inpainting tasks also look good.\n3) Extensive experiments across different tasks and datasets.\n\nWeakness - \n1) I am not convinced by the motivation of the paper. As the noise strength added in the initial stages of forward process is small and it grows over time, it is natural to expect the intra-class to inter-class ratio to remain stable in the initial stages of the forward process.\n2) Adding on to the above point, the motivation around modeling class centers is not clear. The experiments do not reveal the intra-class variance to decrease during the forward process, but rather they remain stable. Why should then the forward process pull the data points towards the center?\n3) The results of faster diffusion denoising do not convince me. By starting from earlier starting points around the center, I expected the FID for instance when using 700 steps to not drop from 3 to 7. As a baseline, what's the performance difference when we naively use 700 steps for the DDPM model instead of using 1000 steps?\n4) I found the experiments around image inpainting hard to understand. Can the authors make it clearer as to how is the training done for it? Do you randomly mask out regions of the image during training at each iteration? How mc-uch portion of the image is masked out at max?\n5) I couldn't find a comparison of the text to image generation against other works. Can the authors include the FID numbers for other related works?", "summary_of_the_paper": "The paper observes the ratio of intra-class variance to the inter-class variance during different time steps of the forward process of diffusion models and notes that the ratio initially remains stable for the early steps of the process and then greatly increases when the data distribution finally gets converted to noise distribution. Taking inspiration from this class clustering process, the paper proposes to explicitly model the class center in the forward and the reverse process, whereby during the forward process the samples are pulled towards the center, before ultimately becoming noise. A separate conditioning network is used to model the centre. By using class as conditions for the conditioning network, the paper is able to perform class condition generation from a diffusion model and shows good image generation performance in terms of FID and Inceptions scores. The paper also shows good performance for other controllable image generation tasks such as image inpainting, attribute conditioned generation, and text to image generation by efficiently conditioning the diffusion model.", "summary_of_the_review": "The paper achieves good empirical performance but needs more polishing around the motivation and some experimental details. I will be happy to increase my rating if my queries are clarified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0qqeRTGmYG", "reviewer": "Reviewer_Xxqm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "< Strength >\n1. This paper provides a new variant of a score-based generative model, and it can be attractive for many machine learning researchers.\n\n2. This paper suggests a new interesting finding, the changes of the ratio of intra-class and inter-class.\n\n3. From a new finding, this paper proposes a new class center-based algorithm for conditional image generation. The proposed model, ST-DDPM, shows a significant performance improvement on the FID score and Inception Score.\n\n< Weakness and Questions >\n1. The quantitative results are reported for the CIFAR-10 dataset only. There are qualitative results for other datasets such as FFHQ and LSUN, but there are no quantitative results for that kind of dataset.\n\n2. This paper only reports the FID and inception score. I am curious about the results of negative log-likelihood.\n\n3. There are many new score-based model research [1,2,3], and it will be interesting if further discussion will be added to the paper.\n\n[1] Tae, Jaesung, Hyeongju Kim, and Taesu Kim. \"EdiTTS: Score-based Editing for Controllable Text-to-Speech.\" arXiv preprint arXiv:2110.02584 (2021).\n\n[2] Kim, Dongjun, et al. \"Score Matching Model for Unbounded Data Score.\" arXiv preprint arXiv:2106.05527 (2021).\n\n[3]  Tashiro, Yusuke, et al. \"CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation.\" arXiv preprint arXiv:2107.03502 (2021).", "summary_of_the_paper": "Score-based generative models show great performance on image generation tasks.\nScore-based generative models are composed of the two-phase, forward process, and reverse process.\nOn the forward process, the score-based model gradually corrupts the data with noise, and the reverse process restores the image by denoising.\nThis paper measures the ratio of intra-class and inter-class, and they find that the class separation decreases sharply and the data distribution is totally covered by the noise.\nFrom a new finding, this paper proposes a class center on both forward and reverse processes for a controllable generation.", "summary_of_the_review": "Please see the weakness and questions section.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0_R_eVBah-j", "reviewer": "Reviewer_49RM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n* The method is well-motivated through empirical observations that are clearly set forth in the paper.\n* The paper is written in a way that is accessible to a wide audience, yet contains detailed derivations in the appendix.\n* I expect the paper to be interesting to a wide audience due to the wide range of tasks it considers, and separately due to the proposed method being more (easily) interpretable than existing score-based generative model.\n\n**Weaknesses**\n* The baseline conditional generation method considered in the paper (cond. DDPM) appears to be worse than it's unconditional counterpart (DDPM) in terms of the FID / IS metrics (Table 1). One would expect the conditional generative model would outperform the unconditional one by a large margin. It seems that the baseline may not have been properly tuned / optimised, and absent a well-tuned baseline it's difficult to judge the performance of ST-DDPM.\n* Faster sampling experiments are not particularly strong, and miss important baselines (e.g., \"Learning to Efficiently Sample from Diffusion Probabilistic Models\" by Watson et al.). The proposed technique still requires 70%-95% of the denoising steps of the full generation chain - far more than competing methods or hand-crafted beta schedules.\n\n**Minor comments**\n* Was not clear what \"ST\" in ST-DDPM stands for. Is that \"shift\"?\n* How were the 2D projects in Figures 2 and 3 made? If t-SNE or a similar dimensionality reduction method, please include hyper-parameters in the Appendix.\n* The word \"Appendix\" was not consistently capitalized (noticed this for \"appendix B\", but there may be more).\n* What is a \"fire embedding\" (Section 6.1)?\n* Left plot of Figure 10 is somewhat trivial: skipping x% of the training steps gives you a 1 / (1 - x / 100) speed up. E.g. 1.43 = 1 / 0.7, 1.12 ~= 1 / 0.9. This comes from the fact that all denoising steps have the same runtime complexity.", "summary_of_the_paper": "The authors propose a modification of DDPM for conditional generation, in which they explicitly model the conditional mean (\"cluster center\") in the forward and reverse diffusion processes. This modification is motivated by the empirical observation that for the large portion of the diffusion samples form several distinct cluster. The resulting model, ST-DDPM, shows competitive results on conditional image generation and inpainting; and enjoys interpretability by means of cluster center visualisation.", "summary_of_the_review": "The paper provides an interesting and well-motivated modification of DDPMs for conditional generation, but needs stronger baselines to help assess the performance of the proposed modification.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting approach for defining conditional diffusion models. The core idea of this work is based on a new analysis of how class centers evolve in the forward diffusion process. On this positive side, this work builds on top of this analysis and introduces conditional diffusion processes that are guided towards class centers. This paper shows marginal improvements in small image datasets (MNIST and CIFAR-10) and auxiliary applications such as image inpainting and attribute-based image synthesis (demonstrated through only qualitative experiments). On the negative side, the proposed approach has the fundamentally limiting assumption that a class can be represented by a cluster center in the RGB space. Unfortunately, this assumption does not hold for practical datasets such as ImageNet where samples in each class have high diversity, and the class centers in the pixel space are not very distinct for different categories. The reviewers have rated this paper slightly above the borderline. They have acknowledged the novelty of the proposed guided diffusion. But they have criticized the submission for the lack of experiments on more common and challenging benchmarks. They also have criticized this work for not providing quantitative results on the auxiliary tasks. I agree with the reviewers that these experiments would shed light on whether the class center idea would hold for more challenging scenarios.

In the rebuttal, the authors provided additional quantitative results for the text-to-image generation task. However, these results show that the proposed method is outperformed by prior works. Most other auxiliary tasks including image inpainting and attribute-to-face generation are still demonstrated through qualitative experiments without detailed quantitative results. 

In summary, given the limitation of the proposed approach, this submission currently lacks an in-depth analysis of the proposed work on challenging benchmarks and a detailed quantitative comparison to relevant baselines for the auxiliary tasks. Because of these concerns, we believe that the paper in its current form is not ready for publication at ICLR at this point.
Decision: Reject