Diffusion-GAN: Training GANs with Diffusion
Authors: Zhendong Wang
Huangjie Zheng
Pengcheng He
Weizhu Chen
Mingyuan Zhou
Venue: ICLR 2023 poster (2023)
Keywords: deep generative models
diffusion models
data-efficient stable GAN training
adaptive data augmentation

Abstract: Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective in practice.  In this paper, we propose Diffusion-GAN, a novel GAN framework that leverages a forward diffusion chain to generate Gaussian-mixture distributed instance noise. Diffusion-GAN consists of three components, including an adaptive diffusion process, a diffusion timestep-dependent discriminator, and a generator. Both the observed and generated data are diffused by the adaptive diffusion process via different noise-to-data ratios at each timestep. The timestep-dependent discriminator learns to distinguish the diffused real data from the diffused generated data at each diffusion timestep. The generator learns from the discriminator's feedback by backpropagating through the forward diffusion chain, whose length is adaptively adjusted to balance the noise and data levels. We theoretically show that the discriminator's timestep-dependent strategy gives consistent and helpful guidance to the generator, enabling it to match the true data distribution. We demonstrate the advantages of Diffusion-GAN over strong GAN baselines on various datasets, showing that it can produce more realistic images with higher stability and data efficiency than state-of-the-art GANs.

Published as a conference paper at ICLR 2023
DIFFUSION-GAN: TRAINING GANS WITH DIFFUSION
Zhendong Wang".? Huangjie Zheng Pengcheng He?, Weizhu Chen', Mingyuan Zhou'
The University of Texas at Austin, *Microsoft Azure AI {zhendong, wang. huangjie. sheng) @ utexassedu,, (penhe,wzchen) @ microsoftt.cm mingyuan. zhou mccombs, utexas. edu
ABSTRACT
Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective in practice. In this paper, we propose Diffusion-GAN, novel GAN framework that leverages a forward diffusion chain to generate Gaussian- mixture distributed instance noise. Diffusion- GAN consists of three components, including an adaptive diffusion process, a diffusion timestep-dependen discrimi- nator, and generator. Both the observed and generated data diffused by the same adaptive diffusion process. At each diffusion timestep, there is a different noise-to-dan do-ata ratio and the timestep-dependent discriminator learns distinguish the diffused real data from the diffused generated data. The generator learns from the discriminator's feedback by backpropagating through the forward diffusion chain, whose length is adaptively adjusted to balance the noise and data levels. We theoretically show that the discriminator's 's timestep- deestep-dependent strategy gives consistent and helpful guidance to the generator, enabling it to match the true data distribution. We demonstrate the advantages of Diffusion-GAN over strong GAN baselines various datasets, showing that it can produce more realistic images with higher stability and data efficiency than ofate-of-the-arr GANs.
INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow al. 2014) and their variants (Brock al. 2018 Karras et al.) 2019 22200a) Zhao et al. 2020) have achieved great success in synthesizing photo-realistic high-resolution images. GANs in practice, however, are known to suffer from variety of issues ranging from non-convereence and training instability to mode collapse (Arjovsky) and Bottou| [0177] Mescheder et al., 2018). As result, a wide array of analyses and modifications has been proposed for GANs, including improving the network architectures (Karras et al. 2019
Radford et al., 2016: Sauer et al. 2021 Zhang et al.) 2019), gaining theoretical understanding of
GAN training (Arjovsky and Bottou| [017]] Heusel et al. 2017; Mescheder et al. 2017 2018
changing the objective functions (Arjovsky et al. Bellemare et al. [017]] Deshpande et al.
2018 Li al., 201 Nowozin et al. |l. [ and / |021 Yang al. 2021), regularizing the weights and/or gradients (Arjovsky et al. [017]] [ et al. 2018 Mescheder et al., 2018
Miyato et al. 2018a| Roth et |l.7 Salimans et al. (2066). utilizing side information (Wang et 2018 Zhang et al. [017]] 2020b), adding a mapping from the data to latent representation (Donahue al. 2016; Dumoulin
2016 Li et al. 20176), and applying differentiable data
augmentation (Karras et al. 2020a| Zhang et al. 2020a
Zhao et al. 2020).
A simple technique stabilize GAN training is to inject instance noise, i.e., to add noise to the discriminator input, which can widen the support of both the generator and discriminator distribu- tions and prevent the discriminator from overfitting (Arjovsky and Bottou| 2017, Sønderby et al.) 2017). However, this technique is hard to implement in practice, as finding suitable noise distri- bution is challenging (Arjovsky and Bottou, 2017). Roth et al. (2017) show that adding instance noise to the high-dimensional discriminator input does not work well, and propose to approximate it by adding a zero-centered gradient penalty on the discriminator. This approach is theoretically and empirically shown converge in Mescheder al. (2018). who also demonstrate that adding zero-centered gradient penalties to non-saturating GANs can result in stable training and better or comparable generation quality compared to WGAN-GP (Arjovsky et al. 2017). However, Brock

---

Published as a conference paper at ICLR 2023
Diffusiin Process: y- 101 0),
yir = 100
ylr 300
yle= 600
yit=T= 1809
Timesteg Oependent Disofminator: POy00)
It 100
Yy 10-000
Yo 1t 600
%/ - 1000
Difusion Procees: yy 9ly.lt..)...
T eolept cotuteed
Figure 1: Flowchart for Diffusion GAN. The top-row images represent the forward diffusion process of real image, while the bottom-row images represent the forward diffusion process of generated fake image. The discriminator learns distinguish diffused real image from diffused fake image all diffusion steps.
al. (218) caution that zero-centered gradient penalties and other similar regularization methods may stabilize training at the cost of generation performance. To the best of our knowledge, there has been no existing work that is able to empirically demonstrate the success of using instance noise in GAN training on high-dimensione. image data.
To inject proper instance noise that can facilitate GAN training, we introduce Diffusion- GAN, which uses diffusion process to generate couusian-mixtun distributed instance noise. We show graph- ical representation of Diffusion-GAN in Figure T In Diffusion-GAN, the input to the diffusion process is either real or generated image, and the diffusion process consists of series of steps that gradually add noise to the image. The number of diffusion steps is not fixed, but depends on the data and the generator. We also design the diffusion process to be differentiable, which means that we can compute the derivative of the output with respect to the input. This allows us to propagate the gradient from the discriminator to the generator through the diffusion process, and update the generator accordingly. Unlike vanilla GANs, which compare the real and generated images directly, Diffusion-GAN compares the noisy versions of them, which are obtained by sampling from the Gaussian mixture distribution over the diffusion steps, with the help of our timestep-dependen dis- criminator. This distribution has the property that its components have different noise-to-data ratios, which means that some components add more noise than others. By sampling from this distribution, we can achieve two benefits: first, we can stabilize the training by easing the problem of vanishing gradient, which occurs when the data and generator distributions are too different; second, we can augment the data by creating different noisy versions of the same image, which can improve data efficiency and the diversity of the generator. We provide theoretical analysis to support our method, and show that the min- max objective function of SrffusiionGANN which measures the dif- ference between the data and generator distributions, is continuous and differentiable everywhere. This means that the generator in theory can always receive a useful gradient from the discriminator, and improve its performance.
Our main contributions include: 1) We show both theoretically and empirically how the diffusion process be utilized provide model- and domain-agnostic differentiable augmentation, en- abling data-efficient and leaking-free stable GAN training. 2) Extensive experiments show that
Diffusion-Gi GAN boosts the stability and generation performance of strong baselines, including Style- GAN2 (Karras et al. (2020b), Projected GAN (Sauer et al.) (022I, and InsGen (Yang et al. 2021] achieving state-of-the-nt results in synthesizing photo-realistic images, as measured by both the Fréchet Inception Distance (FID) (Heusel et al. 2017) and Recall score (Kynkäänniemi et al. 2019
2 PRELIMINARIES: GANS AND DIFFUSION-BASED GENERATIVE MODELS
GANs (Goodfellow al. 2014 are class of generative models that aim to learn the data dis- tribution p(x) of a target dataset by setting up min-max game between two neural networks: a generator and discriminator. The generator G' takes input random noise vector sampled from a simple prior distribution p(z), such as a standard normal or uniform distribution, and tries to produce realistic-looking samples G(z) that resemble the data. The discriminator D receives either
2

---

Published as a conference paper at ICLR 2023
real data sample drawn from p(æ) or fake sample (z)) generated by G, and tries to correctly classify them as real or fake. The goal of is to fool into making mistakes, while the goal of D is to accurately distinguish G()) from æ. The min-max objective function of GANs is given by
min max ( D) Bm-p(=))logl [log( D (=)) ] Be-n(=)llog( - D(G()))))
In practice, this vanilla objective function is often modified to improve the stability and performance of GoNdGGoodflllow et al., [2144] Miyato et al. 2018a| Fedus et al. 2018), but the general idea of adversarial learning between G and D remains the same.
Diffusion-based generative models (Ho et al. 2020b Sohl-Dickstein et al. 2015 Song and Ermon 2019) assume (x6)))
(a ( dd where 21...... ................................ 2T are latent variables of the same di- mensionality as the data 20 ~ p(Zo). There is a forward diffusion chain that gradually adds noise to the data Zo ((xo) in T steps with pre-defined variance schedule Bg and variance a²:
q(F1:T (o) = II g( x1-1), g( ( x1-1) = Maaz; /1 - -, 3,0² I). notable property that æt an arbitrary time-step can be sampled in closed form as
g( Φo) = . (a,;; a, (1 az)o² where a; := - 3t, a, == II. a. as,
(1)
A variational lower bound Blei al. 2017 then used to optimize the reverse diffusion chain
po == (, II po(x::1 xz).
DIFFUSION-GAN: METHOD AND THEORETICAL ANALYSIS
To construct Diffusion-GAN, describe how to inject instance noise via diffusion, how to train the generator by backpropagating through the forward diffusion process, and how to adaptively adjust the diffusion intensity. We further provide theoretical analysis illustrated with toy example.
3.1 INSTANCE NOISE INJECTION VIA DIFFUSION
We aim to generate realistic samples yg from a generator network G that maps latent variable z sampled from simple prior distribution p(z) high-dimensional data space, such images. The distribution of generator samples v G(zz,, z p(=) is denoted by Pa (x)
p(æ; =)p(=)dzz To make the generator more robust and diverse, we inject instance noise into the generated samples
Ty by applying diffusion process that adds Gaussian noise at each step. The diffusion process can be seen as a Markov chain that starts from the original sample æ and gradually erases its information until reaching noise level o² after T steps.
We define mixture distribution g ax) that models the noisy samples y obtained at any step of the diffusion process, with mixture weight for each step t. The mixture components g æ., t) are Gaussian distributions with mean proportional to æ and variance depending on the noise level at step t. We use the same diffusion process and mixture distribution for both the real samples x ~ p(x) and the generated samples xy ~ Pz(x). More specifically, the diffusion iiffusion-inuuee mixture distributions expressed as
x ~ p(x), y 4 x)) ( ax) = Σt Teg(y x. t),
T Pg (m)) yo ( (.))) ((,,, 1T,) Σt T.4(U, x.,
where 7(y) a)) is a T-component mixture distribution, the mixture weights тe are non-negative and sum to one, and the mixture components q(y t) are obtained via diffusion in Equation ( expressed as
< x, t) Nu: Va,a, (1 aa)ooa1..
Samples from this mixture can be drawn as ~ Px == (isccret(mm)....... RT); y ~ 7 xx, t).
By sampling y from this mixture distribution, we can obtain noisy versions of both real and gen- crated samples with varying degrees of noise. The more steps we take the diffusion process, the more noise we add to and the less information diffusion- -nduuced mixture distribution to train a timestep-dependent discriminator guishes between real and generated noisy samples, and generator G that matches the distribution of generated noisy samples to the distribution of real noisy samples. Next we introduce Diffusion biffusionGGAN that trains its discriminator and generator with the help of the inffussion-indueed mixture distribution.
preserve from We can then use this
that distin-
3

---

Published as a conference paper at ICLR 2023
3.2 ADVERSARIAL TRAINING
The Diffusion-GAN trains its generator and discriminator by solving a min-max game objective as
V(G,D) Pe ur-ot 2.1) Da, t])] (=>-p(()))-77:3B>>-) Ga(z).)) [log(1 - (v t)]] (3) Here, p(æ) is the true data distribution, Pr is discrete distribution that assigns different weights Te each diffusion step E {I,.. T) and q xx,) is the conditional distribution of the per- turbed sample given the original data and the diffusion step t. By Equation (2), with Gaussian reparameterization, the perturbation function could be written as y V + vT 0,06, where
0; = - III as is the cumulative noise level at step t, a is a scale factor, and ~ ~N(0, is a Gaussian noise.
The objective function in Equation (3) encourages the discriminator to assign high probabilities to the perturbed real data and low probabilities to the perturbed generated data, for any diffusion step The generator, on the other hand, tries to produce samples that can deceive the discriminator at any diffusion step f. Note that the perturbed generated sample Y/ gly ()()),t)
be rewritten as y Ga(G)( + /(1 - ā.) e N(0, I). This means that the objective function in Equation (3) is differentiable with respect to the generator parameters, and we can use gradient descent to optimize it with back-propagation.
The objective function Equation 33 is similar to the one used by the original GAN (Goodfellow) (ct al. 2014), except that it involves the diffusion steps and the perturbation functions. We can show that this objective function also minimizes approximation of the Jensen-Shannon (JS) divergence between the true and the generated distributions, but with respect to the perturbed samples and the diffusion steps, follows:
Dis(p t) (p,. E.-p. [Ds (p(y [t)llp, [t)]]
(4)
The IS divergence measures the dissimilarity between two probability distributions, and it reaches its minimum value of zero when the two distributions are identical. The proof of the equality in Equation (4) given in Appendix C A natural question that arises from this result whether mini- mizing the JS divergence between the perturbed distributions implies minimizing the JS divergence between the original distributions, i.e., whether the optimal generator for Equation (3) is also the optimal generator for Dis (p(x)) (x)). We will answer this question affirmatively and provide a theoretical justification in Section 3.4
3.3
ADAPTIVE DIFFUSION
With the help of the perturbation function and timestep dependency, we have new strategy to optimize the discriminator. We want the discriminator D) to have a challenging task, neither too easy allow overfitting the data (Karras al., 2020a Zhao al. 2020) too hard impede learning. Therefore, we adjust the intensity of the diffusion process, which adds noise to both y and yg, depending how much D can distinguish them. When the diffusion step is larger, the noise-to-data ratios are higher and the task is harder. We use 1 - ãr to measure the intensity of the diffusion, which increases grows. To control the diffusion intensity, we adaptively modify the maximum number of steps T.
Our strategy is make the discriminator learn from the easiest samples first, which are the original data samples, and then gradually increase the difficulty by feeding it samples from larger t. To do this, we use a self-paced schedule for T, which depends on a metric ra that estimates how much the discriminator overfits to the data:
Ey.e-p(y.)) (gg(D,D))))))) () 0.5)], = T + sign(ra - dearges) C,
(5)
where is the same as in Karras et al. (2020a) and C' is constant. We calculate and update T every four minibatches. We have two options for the distribution pr that we use to sample for the diffusion process:
uniform: Discrete ( ........
PT
priority:
Discrete
Σ
The 'priority' option gives more weight to larger t, which means the discriminator will see more new samples from the new steps when 7 increases. This is because want the discriminator focus on
4

---

Published as a conference paper at ICLR 2023
the new and harder samples that it has not seen before, this indicates that it is confident about the easier ones. Note that even with the 'priority' option, the discriminator can still see samples from smaller t, because qyy æ) is mixture of Gaussians that covers all steps of the diffusion chain.
To avoid sudden changes in T during training, we use an exploration list Eppt that contains t values sampled from Pr. We keep tept fixed until we update T, and we sample t. from tepl to generate noisy samples for the discriminator. This way, the model can explore each sufficiently before moving to a higher T. We give the details of training Diffusion-GAN in Algorithm I in Appendix F
3.4 THEORETICAL ANALYSIS WITH EXAMPLES
To better understand the theoretical properties of our proposed method, we present two theorems that address two important questions about the use of diffusion- based instance noise injection for training GANs. The proofs of these theorems are deferred to Appendix [ The first question, denoted as (a), whether adding noise to the real and generated samples in a diffusion process can facilitate the learning. The second question, denoted as (b), is whether minimizing the JS divergence between the joint distributions of the noisy samples and the noise levels, p(y. t) and (y, t), can lead to the same optimal generator minimizing the JS divergence between the original distributions of the real and generated samples, p(x) and (2(2))
To answer (a), prove that for any choice of noise level t and any choice of convex function f. the f-divergence (Nowozin et al. 2016) between the marginal distributions of the noisy real and generated samples, (y () and g [t) is smooth function that can be computed and optimized by the discriminator. This implies that the diffusion-based noise injection does not introduce any singularity or discontinuity in the objective function of the GAN. The JS divergence is a special case of f-divergence, where f(u)= = - log(2u) - log(2- - 2u).
Theorem (Valid gradients anywhere for GANs training). Let p(æ) be a fixed distribution over X and z be a random noise over another space Z. Denote Gg Z + X as func- tion with parameter 0 and input and Py(at) as the distribution of Go(z). Let q(y xx,)
N(y Vorar, (1 Tz)o²T), where ât E (0,1) and a >> O. Let ((ylt) = I p(a))q() | f)da and 9a t) = f (x))((()) x. t)dx.. Then, Vt, if function Go is continuous and differentiable, the f-divergence Dy(1(1) (y) t)) is continuous and differentiable with respect to B.
Theorem I shows that with the help of diffusion noise injection by gly xx l). VI. y and yo are defined on the same support space, the whole X, and D D (4((1)) ()lla(y [y] ()) is continuous and dif- ferentiable everywhere. Then, one natural question is what if Dy(g(y) |t)ll, t)) keeps a near constant value and hence provides little useful gradient. Hence, we empirically show that by in- jecting noise through mixture defined over all steps of the diffusion chain, there is always good chance that a sufficiently large t is sampled to provide a useful gradient, via the toy example below.
Toy example. We use the same simple example from from/Arjovsky al. (2017) to illustrate our method. Let x = (0, 2) be the real data and Øg = (0, =) be the data generated by a one-parameter generator, where = is a uniform random variable in (0, 1J. The JS divergence between the real and the generated distributions, Dis (p(x)) (). is discontinuous: it is log 2 when 0 = 0 and zero otherwise, so it does not provide useful gradient to guide 0 towards zero.
We introduce diffusion- noise to both the real and the generated data, as shown in the first row of Figure 2 The noisy data, y and yy have supports that cover the whole space I² and their densities overlap more or less depending on the diffusion step t. In the second row, left, of Figure we plot how the IS divergence between the noisy distributions, (i (t) [t), varies with 0 for different / values. The black line with L = 0 is the original JS divergence, which has discontinuity at = 0. As increases, the JS divergence curves become smoother and have non- zero gradients for a larger range of 0. However, some values of t, such as t 200 in this example, still have flat regions where the JS divergence is nearly constant. To avoid this, we use a mixture of all steps to ensure that there is always high chance of getting informative gradients.
For the discriminator optimization, as shown in the second row, right, of Figure 2 the optimal discriminator under the original JS divergence is discontinuous and unattainable. With diffusion- based noise, the optimal discriminator changes with t: a smaller t makes it more confident and a larger makes it more cautious. Thus the diffusion acts like scale to balance the power of the
5

---

Published as a conference paper at ICLR 2023
1x28 1-400
1,00
6-
0.73
0.30
oytima
-150
Figure 2: The toy example inherited from Arjovsky et al. (2017). The first row plots the distributions of data with diffusion noise injected The second row shows the divergence and optimal discriminator value with and without our noise injection.
discriminator. This suggests the use of differentiable forward diffusion chain that can provide various levels of gradient smoothness to help the generator training.
Theorem 2 (Non-leaking noise injection). Let æ p(ær), q(y x)) and Bg Pa(ar), Yg ((Ug ggg), where q(y x)) is the transition density. Given certain g(y x) if could be reparame- terized into y f(x) ) + h(e), E < p(e), where p(E) is known distribution, and both f and are one-to-one mapping functions, then we could have p(y) Py( +> p(x) = (P
To answer question (b), we present Theorem 2 which shows a sufficient condition for the equality of the original and the augmented data distributions. By Theorem 2 the function maps each æ to a unique y. the function / maps each to unique noise term, and the distribution of is known and independent of æ. Under these assumptions, the theorem proves that the distribution of is the same as the distribution of yg, if and only if the distribution of x is the same as the distribution of Tgg. If we take y & as the y introduced the theorem, then for Vt. Equation (2) fits the assumption made. This means that, by minimizing the divergence between q(y [t) and % (y t). which is the same as minimizing the divergence between p(x) t and Pg (x) (t, we are also minimizing the divergence between p(æ) and (æ). This implies that the noise injection does not affect the quality of the generated samples, and we can safely use our noise injection to improve the training of the generative model.
3.5 RELATED WORK
The proposed Diffusion-GAN GAN can be related to previous works on stabilizing the GAN training, building diffusion- bratud generative models, and constructing differential augmentation for data- efficient GAN training. detailed discussion on these related works deferred to Appendix
4 EXPERIMENTS
We conduct extensive experiments to answer the following questions: (a) Will Diffusion-Gal out- perform state-of-the-art GAN baselines on benchmark datasets? (b) Will the diffusion- noise
injection help the learning of GANs in domain-agnostic tasks? (c) Will our method improve the performance of data-cfficient GANs trained with a very limited amount data?
Datasets. We conduct experiments image datasets ranging from low-resolution (e.g., 32 X 32) to high-resolution (e.g., 1024 x 1024) and from low-diversity to high-diversity: CIFAR-I0 Krizhevsky 2009), STL-10 (Coates al. 2011).
YYu et al. 2015). LSUN- details on these benchmark datasets provided in Appendix
Yu et al. 2015). AFHQ(Cat/Dog//W
(Choi et al. 2020),, and FFHQ (Karras et al. 2019). More
Evaluation protocol. We measure image quality using FID (Heusel et al. 2017). Following Karras et al. (2099 2020b), we measure FID using 50k generated samples, with the full training set used

---

In the Published as a conference paper at ICLR 2023
Table 1: Image generation results on benchmark datasets: CIFAR-IO, CelebA, STL-I0, ASUN-Bedroom LSUN- -SUN-Chureh, and FFHQ. We highlight the best and second best results each column with bold and underline, respectively. Lower FIDs indicate better fidelity, while higher Recalls indicate better diversity.
CIFAR-10
CelebA
STL-10
LSUN-Bedroom
LSUN-Church
FPHQ
Methods
(32
32)
64)
(256 x 256)
(256
256)
(1024
1024)
FID
Recall
FID
Rocall
FID
Recall
FID
Recall
FID
Recall
FID
Recall
StyleGAN2 (Karms et al. 20020s
8.32"
0.4"
2.32
0.55
11.20
0.44
3.98
0.32
3.92
0.39
4.41
0.42
SlyleGAN2 + DillAng Zhao et all
2020
5.79
0.12
2.75
0.52
12.97
0.39
4.25
0.19
4.66
0.33
4.46
0.41
StyleGAN2 + ADA (Karras et
2020a
2,92
0.49
2.49
0.53
13,72
0.36
7.89
0.05
4.12
0.18
4.47
0.41
Diffusion StyleGiAN2
3.19
0.58
1.69
0.67
11,43
0.45
3,65
0.32
3.17
0.42
283
0.49
(a) CIFAR-10
(b) CelebA
(c) STL-10
(d) LSUN- Bedroom
(c) LSUN Church
(f) FFHQ
Figure 3: Randomly generated images from Diffusion StyleGAN2 trained on CIFAR- 10, CelebA, STL-10, LSUN-Becroom,
and FFHQ datasets.
as reference. We use the number of real images shown to the discriminator to evaluate convergence (Karras et al.) 2020al Sauer et al.) (002T). Unless specified otherwise, all models are trained with 25 million images to ensure convergence (these trained with more or fewer images are specified in table captions). We further report the improved Recall score introduced by Kynkäänniemi al. (2019) to measure the sample diversity of generative models.
Implementations and resources. We build Diffusion-GANN: based on the code of StyleGAN2 Karras et al. 2020b), ProjectedGAN (Sauer et al., 2021), and InsGen (Yang et al. 2021 to an- swer questions (a), (b), and (c), respectively. Diffusion GANS inherit from their corresponding base GANs all their network architectures and training hyperparamters, whose details provided in Appendix G Specifically for StyleGAN2 and InsGen, we construct the discriminator as D (y. t), where f is injected via their mapping network. For ProjectedGAN, criminator could be ignored to simplify the implementation and minimize the modifications to Pro- jectedGAN. More implementation details are provided in Appendix H. By applying our diffusion- based noise injection, we denote our models as Diffusion following experiments, we train related models with their official code the results are unavailable, while others are all reported from references and marked with We run all our experiments with ei- ther 4 or 8 NVIDIA V 100 GPUs depending on the demands of the inherited training configurations.
empirically find & in the dis-
4.1 COMPARISON STATE-OF-THE-ARE GANS
We compare Diffusion-GAr GAN with its state-of-the-art GAN backbone, StyleGAN2 (Karras et al. 2020a), and to evaluate its effectiveness from the data augmentation perspective, we compare it with both StyleGAN2 DiffAug (Zhao et al. 2020 and StyleGAN2 ADA (Karras et al. 2020a), in terms of both sample fidelity (FID) and sample diversity (Recall) over extensive benchmark datasets.
We present the quantitative and qualitative results in Table/Tand Figure Qualitatively, these gener- ated images from Diffusion StyleGAN2 are all photo-realiste and have good diversity, ranging from low-resolution (32 x 32) to high-resolution (1024 x 1024). Additional randomly generated images
7

---

Published as a conference paper at ICLR 2023
T Scheddlle on CIFAR-10 and ST
Discriminstor outpptts on CIFAR-10
14.0
pal von Ingenreeiss tiae
400
Tra ring Propress million mai imageo
Traning Pragress mil or real Imaged)
Figure 4: Plot of adaptively adjusted maximum diffusion steps I and discriminator outputs of Diffusion-GANN
can be found in Appendix L Quantitatively. Diffusion StyleGAN2 outperforms all the GAN base- lines in generation diversity, measured by Recall, on all 6 benchmark datasets and outperforms them in FII by a clear margin on 5 out of the 6 benchmark datasets.
From the data augmentation perspective, we observe that Diffusion StyleGAN2 always clearly out- performs the backbone model StyleGAN2 across various datasets, which empirically validates our Theorem 2 By contrast, both the ADA (Karras et al.] 2020b) and Diffaug (Zhao et al.. 2020) techniques could sometimes impair the generation performance sufficiently large datasets, e.g.. LSUN-Bedroom and LSUN- ChUhhhhhh,, which is also observed by Yang et al. (202)) on FFHQ This is possibly because their risk of leaking augmentation overshadows the benefits of data augmentation.
To investigate how the adaptive diffussin process works during training, we illustrate in Figure $ the convergence of the maximum timestep I in our adaptive diffusion and discriminator outputs. We that T is adaptively adjusted: The for Diffusion StyleGAN2 increases as the training goes while the T for Diffusion ProjectedGAN first goes up and then goes down. Note that the T is adjusted according to the overfitting status of the discriminator: The second panel shows that trained with the diffusion-based mixture distribution, the discriminator is always well behaved and provides useful learning signals for the generator, which validates our analysis in Section 3.4 and Theorem !
Memory and time costs. Generally speaking, the memory and time costs of a Diffusion-GAN are comparable to those of the corresponding GAN baseline. More specifically, switching from ADA (Karras et al. 2020a) to our diffusion-based augmentation, the added memory cost is negative, the added training time cost is negative, and the added inference time cost is zero. For example, for CIFAR-10 10. with four NVIDIA V100 GPUs, the training time for each 4k images is around 8.0s for StyleGAN2, 9.8s for StyleGAN2-ADA, and 9.5s for Diffusion-StyleGAN2
4.2 EFFECTIVENESS OF DIFFUSION-GAN FOR DOMAIN-AGNOSTIC ALGMENTATION
To verify whether our method is domain-agnostic, we apply Diffusion-GAN onto the input feature vectors of GANs. We conduct experiments on both low-dimensional and high-dimensional feature vectors, for which commonly used image augmentation methods no longer applicable.
25- A5-Gaussians Example. We conduct experiments on the popular 25-Gaussians generation task. The 25-Gaussians dataset is 2-D toy data, generated by mixture of 25 two-dimensional Gaussian distributions. Each data point is 2-dimensional feature vector. We train a small GAN model, whose generator and discriminator are both parameterized by multilayer perceptrons (MLPs), with two 128-unit hidden layers and LeakyReLu nonlinearities.
The training results are shown in Figure [S] We observe that the vanilla GAN exhibits severe mode collapsing, capturing only a few modes. Its discriminator outputs of real and fake samples depart from each other very quickly. This implies strong overfitting that the discriminator stops providing useful learning signals for the generator. However, Diffusion- GAN successfully captures all the 25 Gaussian modes and the discriminator is under control to continuously provide useful learning signals. We interpret the improvement from perspectives: First, non-leaking augmentation helps provide more information about the data space; Second, the discriminator is well behaved given the adaptively adjusted diffusion based noise injection.
the discriminator happened so
ProjectedGAN. To verify that our adaptive diffusion- basssd noise injection could benefit the learn- ing of GANs on high-dimensional feature vectors, we directly apply it to the discriminator feature
8

---

Published as a conference paper at ICLR 2023
USIE TVP
Figure 5: The 25-Gaussians example. We show the true data samples, the generated samples from vamilla GANs, the discriminator outputs of the vanilla GANs, the generated samples from our Diffusion-GAN, and the discriminator outputs of Diffusion-GAN.
ProjectedGANNs generally leverage pre-trained neu- ral networks to extract meaningful features for the adversarial learning of the discriminator and generator. Following Sauer al. 202TT) we adaptively diffuse the feature vectors extracted by EfficientNet-vO and keep all the other training parts unchanged. We report the performance of Dif- fusion ProjectedGAN on several benchmark datasets in Table which verifies that our augmentation method is domain-agnootic. Under the ProjectedGAN framework, we see that with noise properly injected into the high-dimensional feature space, Diffusion ProjectedGAN shows clear improvement in terms of both FID and Recall. We reach state-of-the -o-t------h--ttttt art FID results with Diffusion ProjectedGAN on STL-10 and LSUN-Bcdroom/ Bedroom/Church datasets.
space of ProjectedGAN (Sauer al., 2021
Table 2: Domain agnosica experiments on ProjectedGAN.
CIFAR- (32 x 32)
STI.110
64)
LSUN.Bedrooo (258 250)
LSUN-Church (256 256)
Domain agmaatn Tasks
FID
Recall
FID
Recall
FID
Recall
FID
Recall
ProjectedGAN Sauer al. 2021]
3.10
0.45
7.76
0.35
2.25
0.55
3.42
0.56
Diffusion ProjectedGAN
2.54
0.45
0.35
1.43
0.58
1.85
0.63
4.3 EFFECTIVENESS OF DIHFUSION-GAY FOR LIMITED DATA
We evaluate whether Diffusion-GAN can provide data-efficient GAN training. We first generate five FFHQ (1024 x 1024) dataset splits, consisting of 200, 500, 1k, 2k, and 5k images, respectively, where 200 and 500 images are considered to be extremely limited data cases. We also consider AFHQ-Cat, -Dog, and -Wild (512 512), each with as few around 5k images. Motivated by the success of InsGen (Yang et al., 2021) on small datasets, we build our Diffusion-GAN upon it. We note limited data, InsGin convincingly outperforms both StyleGAN2+ADA and +DiffAug, and currently holds the wate-of-the-aa performance for data-efficient GAN training. The results in Table 3 show that our Diffusion-GAN method can help further boost the performance of InsGen in limited data settings.
Table 3: FFHQ (1024 1024) FID results with 200, 500, 1k, 2k, and 5k training samples; AFHQ (512 512) FID results. To ensure convergence, all models are trained across 10M images for FFHQ and 25M images for AFHQ. We bold the best number in each column.
Models
FFHQ (200)
FFHQ (500)
FFHQ (1k)
FFHQ (2k)
FFHQ (5k)
Cat
Dog
Wild
InsGen Yann et ul. 2021
102.58
54.762
34,90
18.21
9.89
2.60*
5,44
1.77'
Diffusion InsGen
63.34
50.39
30.91
16.43
8.48
2.40
4.83
1.51
5 CONCLUSION
We present Diffusion-GAN, novel GAN framework that uses a variable-length forward diffu- sion chain with a Gaussian mixture distribution to generate instance noise for GAN training. This approach enables model- and domain-agnostic differentiable augmentation that leverages the ad- vantages of diffusion without requiring costly reverse diffusion chain. We prove theoretically and demonstrate empirically that Diffusion-GAA can prevent discriminator overfitting and provide non-leaking augmentation. We also demonstrate that Diffusion-G..n GAN can produce high-resolution photo-realistic images with high fidelity and diversity, outperforming its corresponding ofate-of-the- art GAN baselines standard benchmark datasets according both FID and Recall.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
Z. Wang, H. Zheng, and M. Zhou acknowledge the support of NSF-IIS 2212418 and IFML.
REFERENCES
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URLhetps. //openzeview. wett//orummmdeeeeeeeeeeet.tmmmee _awpaa-hHa__heeeeeee_eee
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pages 214 223, 2017.
Marc G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The Cramer distance as solution to biased Wasserstein gra- dients. arXiv preprint arXiv: 70543 2017.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe.
Variational inference: A review for statisti- cians. Journal of the American statistical Association, 112(51 18::889-877, 2017.
Ashish Bora, Eric Price, and Alexandros G. Dimakis. AmbientGAN: Generative models from lossy measurements. In International Conference on Learning Representations, 2018. URL https:
//openzeview.oo [/oeeneeriew.net//ormmma..
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv: 11096, 2018.
Yunjey Choi, Youngjung Uh. Jaejun Yoo, and Jung- Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8185-8194, 2020.
Adam Coates, Andrew Ng. and Honglak Lee, An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on arrificial intelli- gence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Ieee, 2009.
Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing. Generative modeling using the sliced Wasserstein distance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3483-3491, 2018
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. editors, Advances in Neu- ral Information Processing Systems, 2021. htt.tttts////ppeeee
net/ / forum?id
AAMuCvzaVE.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 19X70 2016.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606 160.004006.0004 2016.
William Fedus, Mihaela Rosca, Balaji Iakshminarayanan, Andrew M Dai, Shakir Mohamed, and Ian Goodfellow. Many paths to equilibrium: GANs do not need to decrease a divergence at every step. In International Conference on Learning Representations, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farkes Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pages 8672-2680, 2014.
10

---

Published as a conference paper at ICLR 2023
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pages 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
Jonathan Ho. Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006. 1239, 2020a.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020b.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila, Training generative adversarial networks with limited data. Advances Neural Information Processing Systems, 33: 1100-12111 14, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11-0-1199 2020b.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. CoRR, abs/1312.6114, 2014.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Tuomas Kynkääänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás P6ezos. MMD GAN: Towards deeper understanding of moment matching network. Advances neural information processing systems, 30, 2017a
Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin. Alice: Towards understanding adversarial learning for joint distribution matching. Ad- vances neural information processing systems, 30, 2017b.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In International Conference Learning Representations, 2020.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101 023388, 2021.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of GANs. Advances in neural information processing systems, 30, 2017.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In International conference machine learning, pages 3481-3490. PMLR, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018a. URL htt.:/tttss//ppeeeviieeeeoo httpa//aeaeaaaeeeeeeeooooooeoaaa

---

Published as a conference paper at ICLR 2023
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018b.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with 2021.
text-guided diffusion models. arXiv preprint arXiv:21
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, 2016.
Kushagra Pandey. Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevae: Effi- cient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint arXiv:2201.00308, 2022.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1..064434 2015.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/ 11111111166660 .06434, 2016.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint arXiv:2004.06125, 2022.
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. Advances in neural information process- ing systems, 30, 2017.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, pages 2234 2242, 2016
Robin San-Roman Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion mod- cls. arXiv preprint arXiv:2104. 02600, 2021.
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected GANs converge faster. Advances in Neural Information Processing Systems, 34, 2021.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. ArXiv, abs/1503.03558 2015.
Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amortised MAP inference for image super-resolution. In 5th International Conference on Learning Repre- sentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net, 2017. BtRLhttps: /ttptttt/////eeeee////
net. meet/forummiiei
6GLle
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint wxXiv:2010002502. 2020.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffuusio implicit models. In Interna- tional Conference on Learning Representations, 2021a. URL /tthttts///ppeeeeieeew net/
Forum?id=Stlqiaane
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pages 11918-11930, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b. URL
net.
2RRHS
Proe-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung- Kien Nguyen, and Ngai-Man Che- ung. On data augmentation for gan training. IEEE Transactions on Image Processing, 30: 1882- 1897, 2021.
12

---

Published as a conference paper at ICLR 2023
Ting- Ting-Chun Wang, Ming- Yu Liu, Jun Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional GANs. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2018.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:21 2021.
Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. Advances in Neural Information Processing Systems, 0199378-9390 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint xrxiv: 00.5500603366 2015.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Mctaxas, StackGAN: Text to photo-realistic image synthesis with stacked generative ad- versarial networks. In Proceedings of the IEEE international conference on computer vision, pages 5907-5915, 2017.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative ad- versarial networks. In International Conference on Machine Learning, pages 7354-7363, PMLR, 2019.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for gen- erative adversarial networks. In International Conference on Learning Representations, 2020a. URL / wwt.:/toummdid-ee $11xxkkKK
Hao Zhang, Bo Chen, Long Tian, Zhengjue Wang. and Mingyuan Zhou. Variational hetero-encoder randomized GANs for joint image-text modeling. In International Conference on Learning Rep- resentations, 2020b.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. Advances in Neural Information Processing Systems, 33:7559-7570, 2020.
Huangjie Zheng and Mingyuan Zhou. Exploiting chain rule and Bayes' theorem to compare prob- ability distributions. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL net/forum dff.ormmmdd-fffggIoTTT55
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion proba- bilistic models. arXiv preprint arX1v,2002.09671, 2022.
13

---

Published as a conference paper at ICLR 2023
Appendix
A RELATED WORK
Stabilizing GAN training. A root cause of training difficulties in GANs often attributed to the JS divergence that GANs intend to minimize. This is because when the data and generator distributions have non-overlapping supports, which are often the case for high-diment nigh-dimensional data supported by low-dimensional manifolds, the gradient of the JS divergence may provide useful guidance to optimize the generator (Arjovsky and Bottou| [017]] Arjovsky et al., 2017] Mescheder et al.) 2018 Roth et al., 2017). For this reason, (Arjovsky al. (2017) propose to instead use the Wasserstein- distance, which in theory can provide useful gradient for the generator even if the two distributions have disjoint supports. However, Wasserstein GANs often require the use of critic function under the 1-Lipschitz constraint, which is difficult to satisfy in practice and hence realized with heuristics such as weight clipping (Arjovsky al. 2017 gradient penalty (Gulrajani et al. 2017). and spectral normalization (Miyato et al. 2018a).
While the divergence minimization perspective has played an important role in motivating the con- struction of Wasserstein GANs and gradient penalty-based regularizations, cautions should be made on purely relying on to understand GAN training, due to not only the discrepancy between the divergence in theory and the actual min-max objective function used in practice, but also the poten- tial confounding between different divergences and different training and regularization strategies (Fedus et al. 2018 Mescheder et al. 2018). E.g., Mescheder et al. (2018) have provided simple example where in theory the Wasserstein GAN is predicted to succeed while the vanilla GAN is predicted to fail, but in practice the Wasserstein GAN with finite number of discriminator updates per generator update fails to converge while the vanilla GAN with the non-saturating loss can slowly converge. Fedus al. (2018) provide rich set of empirical evidence to discourage viewing GANs purely from the perspective of minimizing specific divergence at each training step and emphasize the important role played by gradient penalties on stabilizing GAN training.
Diffusion models. Due to the use of a forward diffusion chain, the proposed Pirtusion-GAA can be related to diffusion- based (or score- score-based) deep generative models (Ho al. 2020b; Sohl- Dickstein et al. 2015 [Song and Ermon (019) that employ both a forward (inference) and a reverse (generative) difrusion chain. These diffusion-basse
generative models are stable to train and can generate high-fidelity photo-realistic images (Dhariwal and Nichol, 2021 Ho et al. 2020b Nichol jet al. 202T Ramesh al., 2022, [Song and Ermon 2019 Song et al. 202 However, they are notoriously slow in generation due to the need to traverse the reverse diffusion chain, which involves going through the same U-Net-based generator network hundreds or even thousands of times (Song et al. 202 For this reason, a variety of methods have been proposed to reduce the generation cost of iiffuusionbbsed generative models (Kong and Pingl 2021 Luhman and Luhman 2021 Pandey
al. 2022 San-Roman al. 2021 Song et al. 2021a Xiao al. 2021 Zheng et al. 2022).
A key distinction is that Diffusion-GAN needs a reverse diffusion chain during neither training nor generation. More specifically, its generator maps the noise generated sample in a single step. Diffusion-GAN can train and generate as quickly as vanilla GAN does with the same generator size. For example, it takes around 20 hours to sample 50k images of size 32 32 from DDPM (Ho et al. 2020b) on an Nvidia 2080 Ti GPU, but would take less than a minute to do so from Diffusion GAN.
Differentiable augmentation. As Diffusion- SrAfussion-GNNN transforms both the data and generated samples before sending them the discriminator, we can also relate it to differentiable augmentation (Karras et al. 2020a] Zhao et al. 2020) proposed for data-efficient GAN training. Karras et al. (2020a) intro- duce a stochastic augmentation pipeline with 18 transformations and develop an adaptive mechanism for controlling the augmentation probability. Zhao al. (2020) propose to use Color Translation + Cutout as differentiable augmentations for both generated and real images.
While providing good empirical results some datasets, these augmentation methods are developed with camain--peccffee knowledge and have the risk of leaking augmentation into generation (arras et al. 2020a). As observed in experiments, they sometime worsen the results when applied to a new dataset, likely because the risk of augmentation leakage overpowers the benefits of enlarging the training set, which could happen especially if the training set size is already sufficiently large.
14

---

Published as a conference paper at ICLR 2023
By contrast, Diffusion-GAN uses differentiable forward diffusion process stochastically trans- form the data and can be considered as both a domain -ammin-agnostic and model-agnostic augmentation method. In other words, Diffusion-GAN. can be applied non-image data even latent features, for which appropriate data augmentation is difficult to be defined, and easily plugged into an existing GAN improve its generation performance. Moreover, prove in theory and show in experiments that augmentation leakage is not a concern for Diffusion-GAN Tran et al. (2021) provide a theoreti- cal analysis for deterministic non-leaking transformation with differentiable and invertible mapping functions. |oora et al. (2018) show similar theorems to us for specific stochastic transformations, such as Gaussian Projection. Nonsolve+Noose, and stochastic Block-Pixels, while our Theorem includes more satisfying possibilities as discussed in Appendix B
B PROOF
Proof of Theorem 7 For simplicity, let æ ~ Pr. 2g ~ Fg y Pr Uo ~ Po Vā, and
be = (1 0z)o²² Then.
pr / ax dr
Po / Pp (yr
~~ p(>) Fg = g0 yg a,3g + bie,e p(e)
Dy(pr (1)rl, = / Por,((U)
Prr,(()
dy
= Ey-psr((M)
s
Pr (rr,:(3)
(Pg
[
P (argo + b,E)
= -zzx(())exp(e))
Por (u:900) + b,e)
Since N (yr; a, b,1) is assumed to be an isotropic Gaussian distribution, for simplicity, in what follows we show the proof in uni-variate Gaussian, which could be easily extended to multi-variate Gaussian by the production rule. We first show that under mild conditions, the prr,(()) and (y) are continuous functions over y.
lim P cel Av) lim (x) Ay; ai be)dz AV-10
Ay
x
(x)
lim N(y Ay
Ay; axx, be
/ pr(z) lim Ag C₁
((y Ay) axz)²
exp
dx
C2
pr(x))(() axx, bz)dz
(r),)
where C1 and C2 are constants. Hence, ( is a continuous function defined on y. The proof of continuity for p is exactly the same proof. Then, given go also continuous function, is clear to see that
(B)llly is a continuous function over 6.
Next, we show that Dy(prr.(() (()):..()))lp..((()) is differentiable. By the chain rule, showing (r)Pr:(u)lllpgs to be differentiable is equivalent to show pr),(y), ( and f are dif- ferentiable. Usually, f is defined with differentiability (Nowozin al. 2016).
Vepgr,((,go(2) + boc) V. / ((()s(())))())) bie; axx, bi)dz
/ (x)
1 exp (urgo(>) (z) + bie 4x16 dr,
C2
C2
15

---

Published as a conference paper at ICLR 2023
Vepgss(aego(3)) b(e) Vo /
/ (((sg)((agg0z)) + bie; act, be)dx
(VBLz'ppzz') (argo((z) + bre; aego (3)) bb)l
((ago(z) + bie aego(2')) (2)) )
(ggpp())
V exp
C2
where C1 and C2 are constants. Hence, pr (y) and pr ((y) are differentiable, which concludes the proof.
Proof of Theorem 2 We have p(y) = p(x)q(y) x) and py(U) = wx)
= If p(xx) Py (z). then p(y) Pz(
+ Let ~ p(y) and y ~ Pg (y). Given the assumption on < æ),, we have
y f(x) + g(c). ~ p(x), p(c)
Y3 f(mg) g(Gg) Fg ~ Pa (x)) Eg ~ p(c).
Since f and g are one-to-one mapping functions, f(x) and g(c) are identifiable, which indicates f(x) & f(xg) 2 zg. By the property of moment-genen seenent-eeneratng functions (MGF), given f(x) is independent with g(c), we have for vs
M2 = (s(w(()) (x)
My, (8) M)( M (8).
where My(s) Ey--ov) (e" ") denotes the MGF of random variable y and the others follow the same form. By the moment-generating function uniqueness theorem, given 2 Y, and g(e) 2 g(c)). we have My = My, (s) and (s) = Mg(c,)(s) for Vor Then, we could obtain ((x) ((,)) for vs. Thus, Mf(a) = Mf(ar)) = f(x) 2 f(zg) = p(x) = p(xg), which concludes the proof.
Discussion. Next, we discuss which 7(y x) fits the assumption we made on it. We follow the discussion of reparameterization of distributions used in Kingma and Welling (214)). Three basic approaches are:
Tractable inverse CDF. this case, let e U(0,)) I). and (é(e, y, x) be the inverse CDF of q(v x). From (e,, x), if y f(x) + g(e), for example, y ~ Cauchy(a;, 7) and y Logistic(z, s), then Theorem 2 holds.
2. Analogous to the Gaussian example, y ~ -- -, No I). For any "location-scale" family of distributions we can choose the standard distribution (with location 0, scale = 1) as the auxiliary variable e, and let g(.) location + scale e. Examples: Laplace, Elliptical, Student's Logistic, Uniform, Triangular, and Gaussian distributions.
-
3. Implicit distributions. q(v) æ:) could be modeled by neural networks, which implies y f(x) + g(c), ~ p(c), where f and are one-to-one nonlinear transformations.
16

---

Published as a conference paper at ICLR 2023
C
DERIVATIONS
Derivation of equality in JSD
JSD(p(U, (3,,
1 DKL. p(U.C) ()
p(y,t) t) + Py (3.
p(y,t) t) + Py (3.
+ DKL
Pg (2, 1)
2
2
2
1
2 p(y,t)
2 (3).
log ply, + Ps (g(y, ()
+ E (..
log ply, 1) (y
2
1
2 p(y ())
log p [t)p (t) + Po( (t)
2
2 Pg (y) (()
2 Pa
log
p(y (t) Py( [t)px(t)
2.p(y
2 (3 ()
(1)
Eyrplerl log p(y [1) + (y t)
E Eympolu10)
log p(y 1) + Pg (y ())
2
Beap, (1) (IIJDD(ly (y [t)]]
D DETAILS OF TOY EXAMPLE
Here, we provide the detailed analysis of the JS divergence toy example.
Notation. Let X' be compact metric set (such as the space of images [0, 1]d) and Prob(X) denote the space of probability measures defined on X. Let P. be the target data distribution and Pg be the generator distribution. The JSD between the two distributions P,, E Prob(A') is defined as:
I
Pis(Pr1PP)) = DKL
DKL (P, + _ DKL (TP, I
where Pm is the mixture (P, + Pg)/2 and DxI. denotes the Kullback-Leibler divergence, i.e. (2) da.. More generally, the f-divergence (Nowozin et al. 2016
DKL (Pr = .xx (x)) log( (x)
between Pr and PP, is defined as:
(r(()) (x(x))
Dj ()(P))P))) (
dx,
(8)
where the generator function f: R.. IR is convex and lower-semicontinuous function satisfying f(1) = 0. We refer Nowozin et 2016) for more details.
We recall the typical example introduced in Arjovsky and Bottou (2017) and follow the notations.
Example. Let Z ~ U[0.1] 1] be the uniform distribution on the unit interval. Let X ~ Pr be the distribution (0, Z) E R², which contains 0 on x-axis and a random variable Z on the y-axis. Let Xg ~ P., be the distribution of (0, Z) E I2, where 0 is a single real parameter. In this case, the (is(Pr ((P)) is not continuous,
O
if 0 0,
Das (P.
log
if +
For notation simplicity, g and G both denote the generator network in GANs in this paper.
17

---

Published as a conference paper at ICLR 2023
1nd 1228 Ted0
Oydmma Docchini oaa Valla 141
2.00
-120
Figure 6: We show the data distribution and Dss(Pr|IPp)
which can not provide usable gradient for training. The derivation is as follows:
2 (x)
2 ('P
(Ts(Prrl Pp)
(x)
log (x(x) + P2(a)
E py)pp(
log
2
Pr (r(y) + Py (v)
2 1[21 0] (x2)
Ex =00
log [1]
2
())((x2) + [2] = 0 (x22)
1
2-1 0) U
+
By =0 0
log
2
1 = -U)2())) + 1ly1 = 0 (yz)
1
2 1 [1]
0]
1 log
2. 1/11
log
91
0
2
1 = 0)
1[2) 0
1 = 0] + 1yy1
0
[ i = 0,
1lgg2 if + 0.
Although this simple example features distributions with disjoint supports, the same conclusion holds when the supports have a non empty intersection contained in set of measure zero (Arjovsky and Bottou 2017). This happens be the case when two low dimensional manifolds intersect in general position (Arjovsky and Bottou| (017). To avoid the potential issue caused by having non-overlapping distribution supports. common remedy to use Wasserstein- 1 distance which in theory can still provide usable gradient (Arjovsky and Bottou 2017 Arjovsky et al. (2017). In this case, the Wasserstein-1 distance [Ø].
Diffusion-based noise injection
In general, with our diffusion noise injected, we could have,
Pr / pr(x) Vara (1 -2) I)dx
Po = / Po(x:))(( Varar oz 2 Ndd
1 B.p. log 2pr" pritt + Por
log 2p'
Das Dus(pr (pr. (pr.llpgg.)
Prit + Py
For the previous example, we have Yy' and Yg, such that,
Y = (2) ~ Pr = N(n (0,4) Yg,, (Vg,1 No.2) ~ Pg = NN aa,0, (V).))
where f(-) = - N- a a, a, and b, abbreviations for vor and (1 do² The supports of Y/ and y2 both the whole metric space 8² and they overlap with each other de- pending on shown in Figure 2 As f increases, the high density region of Y and Y get closer
18

---

Published as a conference paper at ICLR 2023
since the weight a, is decreasing towards 0. Then, derive the JS divergence,
D ups(p,,lllp
2- D-N(y1 y1 (
2 -Nii vo
log N(w (0,
+ WWi a,0, (322)
2 2-N(yg a,0, b,) (Wg.2)
(3 log NVg,,1 2 E Vn,
1,, (1).2)) + N (Wy,, a,0, be) (yg,))
2.N(yi 10,
2 nn ~N(0bb)))
log N(y b,) + N( ae0, 6t)
N (Mg.1 a,0, be)
2 R V..1~Naa00.b.
log NV9.,1 b,) + (B9..1 (g0,bb)) bb)
which is clearly continuous and differentiable.
We show this D)s(Pr'sellps) (Plg..) with respect to increasing t values and a 0 grid in the second row of Figure 2 As shown in the left panel, the black line with t = shows the origianl JSD, which is not even continuous, while as the diffusion level increments, the lines become smoother and flatter. It is clear to see that these smooth curves provide good learning signals for 0. Recall that the Wasserstein-1 distance is |0] in this case. Meanwhile, we could observe with an intense diffu- sion, e.g., 800, the curve becomes flatter, which indicates smaller gradients and much slower learning process. This motivates us that an adaptive diffusion could provide different level of gradi- ent smoothness and is possibly better for training. The right panel shows the optimal discriminator outputs over the space X. With diffusion, the optimal discriminator is well defined over the space and the gradient smooth, while without diffusion the optimal discriminator is only valid on two find that smaller drives the optimal discriminator to become more
star points. Interestingly, assertive while larger makes discriminator become more neutral. The diffusion here works like a scale to balance the power of the discriminator.
E DATASET DESCRIPTIONS
The CIFAR-10 dataset consists of 50k 32 x 32 training images in 10 categories. The STL-10 dataset originated from ImageNet (Deng al., 2009) consists of 100k unlabeled images in 10 categories, and resize them to 64 x 64 resolution. For LSUN datasets, sample 200k images from LSUN- Bedroom, use the whole 125k images from
and resize them to 256 x 256 resolution for training. The AFHQ datasets includes around 5k 512 512 images per category for dogs, cats, and wild life; we train a separate network for each of them. The FFHQ contains 70k images crawled from Flickr at 1024 1024 resolution and all of them for training.
F ALGORITHM
We provide the Diffusion-GAN algorithm in Algorithm
G HYPERPARAMETERS
Diffusion- GAN is built on GAN backbones, so we keep the learning hyperparameters of the original GAN backbones untouched. Diffusion-Gaal introduces four new hyperparameters: noise standard deviation Tmaz. T increasing threshold dtarget, and / sampling distribution Pr.
The is fixed as 0.05 for images (pixel values rescaled to [-1 .1]) in all our experiments and shows good performance. I T'mx could be fixed as 500 or 1000, which depends the diversity of the dataset. We recommend a large Tmax for diverse datasets. dearget is usually fixed as 0.6, which does not influence much about the performance. p* has two choices, 'uniform' and 'priority". Generally, (a = 0.05, Tiaax = 500, dearget = 0.6, Pr "uniform" ') is a good starting point for new dataset.
In our experiment, we find StyleGAN2 models are not sensitive to the values dhargets SO set diarget 0.6 for them across all dataset, only except that we diarget 0.8 for FFHQ
19

---

Published as a conference paper at ICLR 2023
Algorithm Diffusion-GAN
while < number of training iterations do
Step 1: Update discriminator
Sample minibatch of noise samples {21, 22. 2m ~ Pz 2)
Obtain generated samples (f, g.2: 2g g. by :.g = G(z).
Sample minibatch of m data examples T2, 3m p(x).
Sample (21,t2...... {m from tept list uniformly with replacement.
For E (1. m), sample yj e(yy|æ,, [2) and Yo.j (( Fg.5.
Update discriminator by maximizing Equation G).
Step 11: Update generator
Sample minibatch of noise samples {21, ø2..... ................................ 2m Ps (z) Obtain generated samples (g.2.. g.2...... 2 g. by I = G(z).
Sample (t1, ................................ fm from tppt list with replacement.
For j E (1.2..... m), sample Vg.j ~ q(Ug.. Tggs (2)
Update generator by minimizing Equation (3).
Step III: Update diffusion
if mod 4 == then
Update T by Equation 55)
Sample lept [0,.. 10...... 4,, 1....... 132), where in px for K (1,...,32) 32). Px is in Equa- tion (6). {lept has 64 dimensions.)
end if end while
Datasets
Fd
CIFAR-I0 (32 32, 50k images) STI-110 (64 x 64, 100k images)
0.45
LSUN-Church (256 x 256, 120k images)
0.2
LSUN-Bedrood (256 x 256, 200k images)
0.2
Table 4: drarget for Diffusion ProjectedGAN
(drarget = 0.8 for FFHQ is slightly better than 0.6 in FID). We report dharget of Diffusion Project- edGAN for our experiments in Tabie We also evaluated two sampling distribution Pr. ['priority', 'uniform'] J. defined in Equation (6). In most cases, 'priority' works slightly better, while in some cases, such as FFHQ, 'uniform' better. Overall, we didn't modify anything in the model archi- tectures and training hyperparameters, such as learning rate and batch size. The forward diffusion configuration and model training configurations are as follows.
Diffusion config. For our diffusion- based noise injection, we set up a linearly increasing schedule for (e, where (2000)) and set 3g = 0.0001 and B7 = 0.02. We adaptively modify T ranging from Tmin = 5 to Thax 1000. The image pixels are usually rescaled to [-1, so we set the Guassian noise standard deviation o = 0.05. For feature level injection in Diffusion ProjectedGAN, we set fn 0.0001,
(1, 2..... T). For pixel level injection in StyleGAN2,
follow Ho et al.
3r 0.01, Tiin 5, Tiax 500, and
0.5. We list all these values Table
Model config. For StyleGAN2-bbsed models, we borrow the config settings provided by Karras et al. (2020a), which include [ ('auto", 'stylegan2', 'cifar', 'paper256', 'paper512', 'stylegan2"]. We create the 'stl' config based on 'cifar' with a small modification that we change the gamma term to be 0.01. For ProjectedGAN models,
the recommended default config (Sauer et al.) 2021) which is based on FastGAN (Liu et al. 2020). We report the config settings used for our experiments in Table
20

---

Published as a conference paper at ICLR 2023
Diffusion config pixel, priority Diffusion config for pixel, uniform
Bg 0.0001, BT 0.02, min
5, T Tmx 1000, a 0.05
3o = 0.0001. 57 = 0.02, Tmin
5. T = 500, a = 0.05
Diffusion config for feature
Bo 0.0001, 37 = 0.01, Tmin 5, mmx 500, 0.5
Table Diffusion config.
Dataset
Models
Config
Specification
StyleGAN2
cifar cifar default
Diffusion StyleGAN2
diffusion-pixel. dearget 0.6, 'priority"
CIFAR-10 (32 32)
ProjectedGAN
diffusion-fcantt
Diffusion ProjectedGAN
default
diffusion-feature
StyleGAN2
Diffusion StyleGAN2
diffusion-pixee dharget 0.6, 'priority'
STL-10 (64 64)
ProjectedGAN
default
dilfusion- diffusion- feature
Diffusion ProjectedGIAN
default
StyleGAN2
paper256
Diffusion StyleGAN2
paper256
Liffusion-pixce duerge 0.6, "priority"
LSUN-BBdroom (256 x 256)
ProjectedGAN
default
diffusion-feature Liffusion-feature
Diffusion ProjectedGAN
default
StyleGAN2
paper256
Diffusion StyleGAN2
paper256
diffusion pixell,--peeleeee diarges 0.6, 'priority"
LSUN-Church (256 256)
ProjectedGAN
default
diffusion-fcature diffusion- feature
Diflusion ProjectedGAN
default
StyleGAN2
paper512
Diffusion StyleGAN2
paper512
diffusion-pixel, diarget 0.6, "priority"
AFHO-Caf/eeg/Wid (512 512)
InsGen
default
Diffusion InsGen
paper512
diffusion-pixel, diarges 0.6, 'uniform'
StyleGAN2
stylegan2
Diffusion StyleGAN2
stylegan2
diffusion-pixtt diarges 0.8, 'uniforn'
FFHQ (1024 x 1024)
InsGen
default
Diffusion InsGen
stylegan2
diffusion-pixel, dearget 0.6, 'uniform'
Table 6: The config setting of StyleGAN2-based models and ProjectedGAN based models. For StyleGAN2- based models, borrow the config settings provided Karras al.] (2020a), which includes ['auto", 'style- gan2', 'cifar', 'paper256', 'paper512', 'paper1024'). We create the 'stl" config based on 'cifar' with small modifications that we change the gamma term to be 0.01. For ProjectedGAN models, we use the recommended default config (Sauer al. 2021). which is based on FastGAN.
H IMPLEMENTATION DETAILS
We implement an additional diffusion sampling pipeline, where the diffusion configurations are set in Appendix G The T' in the forward diffusion process is adaptively adjusted and clipped to [Tmin: T. As illustrated Algorithm| [ at each update step, we sample from hept for each data point æ, and then use the analytic Gaussian distribution at diffusion step to sample Next, we use and instead of æ for optimization.
Diffusion StyleGAN2.
We inherit all the network architectures from StyleGAN2 implemented by Karras et al. (2020a). We modify the original mapping network, which is there for label con- ditioning and unused for unconditional image generation tasks, inside the discriminator to inject Specifically, we change the original input of mapping network, the class label to our discrete value timestep f. Then, we train the generator and discriminator with diffused samples y and f.
Diffuson ProjectedGAN. To simplify the implementation and minimize the modifications to ProjectedGAN, we construct the discriminator as (3y)). where is ignored. Our method is plugged as data augmentation method. The only change the optimization stage that the discriminator is fed with diffused images y instead of original images æ.
Diffuson InsGen. To simplify the implementation and minimize the modifications to InsGen, keep their contrastive learning part untouched. We modify the original discriminator network
21

---

Published as a conference paper at ICLR 2023
Ablation on the adaptive schedule
untormadat
100
30
Training Progress million real
Figure 7: Ablation study on the T adaptiveness.
to inject similarly to Diffusion StyleGAN2. Then, we train the generator and discriminator with diffused samples y and t.
1 ABLATION ON THE MIXING PROCEDURE AND T ADAPTIVENESS
Note the mixing procedure described in Equation (6) referred to as "priority mixing" in what fol- lows, is designed based our intuition. Here we conduct ablation study on the mixing procedure by comparing the priority mixing with uniform mixing on three representative datasets. We report in Table 7 the FID results, which suggest that uniform mixing could work better than priority mixing in some dataset, and hence Diffusion-GANN may be further improved by optimizing its mixing pro- cedure according to the training data. While optimizing the mixing procedure is beyond the focus of this paper, it is worth further investigation in future studies.
Table 7: Ablation study on the mixing procedure. "Priority Mixing" refers to the mixing procedure in Equa- tion (6) and "Uniform Mixing" refers to sample t uniformly at random.
CIFAR-10
STL-10
FFHQ
Priority Mixing
3.19
11.43
3.22
Uniform Mixing
3.44
11.75
2.83
We further conduct ablation study whether the T needs to be adaptively adjusted. As shown in Figure 7 we observe with adaptive diffusion strategy, the training curves of FIDs converge faster and reach lower final FIDs.
J MORE GAN VARIANTS
To further validate our noise injection via diffusion-based mixtures, we add our diffusion-based training into two more representative GAN variants: DCGAN (Radford et al.) 2015) and SNGAN (Miyato et al. 2018b). which have quite different GAN architectures compared to StyleGAN2. We provide the FIDs for CIFAR-10 in Table g We observe that both Diffusion-DCGAN and Diffusion- SNGAN clearly outperform their corresponding baseline GANs.
Table FIDs on CIFAR- 10 DCGAN, Diffusion DCGANN, SNGAN, and Crrtuusion-NGAAR
DCGAN Radford et al.
2015
Iiffusion-DGGAN
SNGAN Miyato et al. 2018h
CIFAR-10
28.65
24.67
20.76
17.23
22

---

Published as a conference paper at ICLR 2023
Method
FID L
Recall 1
NTE
DDPM Ho
2020d
9.46
3.21
0.57
1000
DDIM Song ecsal
2020
4.67
50
Denoising Dirfustonn GAN
Xino et al
2021
9.63
3.75
0.57
4
StyleGAN2
al.
StyleGAN2 DiitAug (Zhno of a
2020
9.40
5.79
0.42
StyleGAN2
Diffusion StyleGANZ
9.94
3.19
0.58
Table Inception Score CIFAR- 10. sampling time, we use the number of function evaluations (NFE).
K INCEPTION SCORE FOR CIFAR-10
We report the Inception Score (IS) (Salimans et al. 2016) Diffusion StyleGAN2 for CIFAR-10 dataset in Table 9 and also include other state-of-the-an art GANs and diffussin models baselines. Note CIFAR-10 is a well-known dataset and tested by almost all baselines, so we pick CIFAR-10 here and we reference the reported IS values from their original papers for fair comparison.
L MORE GENERATED IMAGES
We provide randomly generated images for LSUN-Bedroom, LSUN-Church, AFHQ, and FFHQ datasets in Figure Figure 9 and Figure
23

---

Published as a conference paper at ICLR 2023
erstock
Figure 8: More generated images for LSUN-Bedroom (FID 1.43, Recall 0.58) and LSUN-Church (FID 1.85,
Recall 0.65) from Diflusion ProjectedGAN.
24

---

Published as a conference paper at ICLR 2023
Figure 9: More generated images for AFHQ-Cat (FID 2.40), AFHQ-Dog (FID 4.83) and AFHQ-Wild (FID
1.51) from Diffusion InsGen.
25

---

Published as a conference paper at ICLR 2023
Figure 10: More generated images for FFHQ from Diffusion StyleGAN2 (FID 3.71, Recall 0.43).
26


Reviews:
[{"review_id": "Vim7KGI9WNi", "reviewer": "Reviewer_YnnY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the idea of diffusion-GAN by incorporating diffusion process with GANs. In the framework, an adaptive diffusion process for sampling the noise and a diffusion timestep-dependent discriminator are added to a traditional GAN generator. The main contributions are:\n1. Propose a diffusion-GAN framework which combines the technique of diffussion models and GAN;\n2. Theoretically and empirically show the effectiveness of the method and the ability of stabling GAN training;\n3. According to the results reported, the quality of the generative images are better than the state-of-the-art GANs.", "summary_of_the_review": "This paper studies a combination of two state-of-the-art generative models and theoretically shows its ability to stable the GAN training. According to the results, the images are of better quality than current GANs. I recommend accepting it.", "strength_and_weaknesses": "The main strength:\n1. Proposed a new architecture which combines two state-of-the-art generative models and theorectially showed the ability of acheiving a better GAN training and also sampling;\n2. The experiments are well-designed and the results are significant and interesting;\n3. The paper is well-written and easy to follow\n\nConcerns/questions:\n1. While the FID and Recall are reported, is it possible to also include other metrics(IS, LPIPS, ..);\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IGhWT-Waf9", "reviewer": "Reviewer_jaon", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a diffusion-based approach to stabilise the learning process of GANs in a instance noise fashion.", "summary_of_the_review": "Overall I think this is a good paper tackling a well motivated task.", "strength_and_weaknesses": "Strength:\n- The abstract and introduction give a nice overview and motivation for the problem.\n- The method is supported by a theoretical analysis.\n- The results are good and evaluation is reasonable.\n- The objective is still differentiable wrt the generator using the classical rewritting of the perturbed samples.\n\nWeaknesses:\n- 25-gaussians example (Fig. 5): the purpose of the figure is to exhibit the fact that the propose method is not prone to mode collapse but I am curious to know if the proposed method has shown some mesurable improvement wrt SotA methods on this \"toy example\"? \n- The authors did an ablation study on the adaptive diffusion process. But, reading Section 3.3, it feels like the adaptative diffusion strategy is key to the method whereas the ablation study does not clearly support it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7SRsTbNUL7", "reviewer": "Reviewer_dVvW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an improved GAN training method that involves a diffusion chain that generates Gaussian mixture distributed instance noise. The diffusion level is dependent on timestep, and the discrimination task is made increasingly difficult for the discriminator over the course of training. The approach improves generation quality in terms of FID measurements and recall capacity, while allegedly retaining stability.", "summary_of_the_review": "The paper contributes a non-trivial new approach to GAN training. It could prove a useful addition to the GAN models and provides one possible bridge between diffusion and GAN methods.", "strength_and_weaknesses": "Strengths:\n- The proposed approach makes intuitive sense and it is explained with sufficient clarity.\n- Generation performance is superior to relevant baselines.\n- Computational costs are on a par with relevant baselines (StyleGAN-ADA).\n- Some theoretical support is provided to ensure the injection of extra noise does not hamper the basic GAN learning properties (though it is difficult to assess the ultimate significance of the provided theorems)\n\nWeaknesses:\n- Claims of stable training were made, but I did not easily find where this was justified. Perhaps the authors could point me to where it is shown that the training is especially stable, in comparison to baselines?\n- It was not clear which hyper-parameters need to be addressed while switching the datasets. Again, a simple summary clarification from the authors might help here.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2VDi8nskeJA", "reviewer": "Reviewer_xwap", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to stabalize GAN training by training Discriminator with adapitvely noised images. The method shows improved performance over GAN baselines across low-resolution and high-resolution image generation benchmarks as well as low and high data regime.", "summary_of_the_review": "The paper is well structured with theoretical analysis and convincing experiments. Introducing adaptive noise as formulated in diffusion process to GAN Discriminator training is somewhat a natural extension of previous works but could be a go-to solution to stabilize GAN  training in the future.", "strength_and_weaknesses": "Strength\n1. The paper is clearly written with theoretical analysis to support proposed method. \n2. Experiments are well designed and demonstrated clear improvments over baseline methods, ranging from low-reoslution to high-resolution and from low-diversity to high-diversity\n\nWeakness\nThe method improvment is somewhat marginal: introducing noise to discriminator training has been explored before, introducing different levels of noises as defined in diffusion process seems like a minor improvement. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster