Real or Not Real, that is the Question
Authors: Yuanbo Xiangli*
Yubin Deng*
Bo Dai*
Chen Change Loy
Dahua Lin
Venue:  (2020)
Keywords: GAN
generalization
realness
loss function

Abstract: While generative adversarial networks (GAN) have been widely adopted in various topics, in this paper we generalize the standard GAN to a new perspective by treating realness as a random variable that can be estimated from multiple angles. In this generalized framework, referred to as RealnessGAN, the discriminator outputs a distribution as the measure of realness. While RealnessGAN shares similar theoretical guarantees with the standard GAN, it provides more insights on adversarial learning. More importantly, compared to multiple baselines, RealnessGAN provides stronger guidance for the generator, achieving improvements on both synthetic and real-world datasets. Moreover, it enables the basic DCGAN architecture to generate realistic images at 1024*1024 resolution when trained from scratch.

Published as a conference paper at ICLR 2020
REAL OR NOT REAL, THAT IS THE QUESTION
Yuanbo Xiangli'; : Yubin Deng': Bo D Chen Change Loy², Dahua Lin' The Chinese University of Hong Kong [xy019, 9, dy 015,, bdai, , dh collinneion eie. cuhk. edu.hk
Nanyang Technological University ccloyêntu.edu..
ABSTRACT
While generative adversarial networks (GAN) have been widely adopted var- ious topics, in this paper generalize the standard GAN to new perspective by treating realness as a random variable that can be estimated from multiple angles. In this generalized framework, referred criminator outputs a distribution as the measure of realness. While RealnessGAN shares similar theoretical guarantees with the standard GAN, it provides more in- sights adversarial learning. Compared to multiple baselines, RealnessGAN provides stronger guidance for the generator, achieving improvements on both synthetic and real-world datasets. Moreover, it enables the basic DCGAN (Rad- ford al.] 2015) architecture to generate realistic images at 1024* 1024 resolution when trained from scratch.
RealnessGANT the dis-
INTRODUCTION
The development of generative adversarial network (GAN) (Goodfellow al., 2014 Radford al., 2013 Arjovsky et al. (017) is one the most important topics in machine learning since its first appearance in (Goodfellow al. 2014). It learns discriminator along with the target generator in an adversarial manner, where the discriminator distinguishes generated samples from real ones. Due its flexibility when dealing with high dimensional data, GAN has obtained remarkable progresses on realistic image generation (Brock et al. 2019).
In the standard formulation (Goodfellow al. 2014), the realness of an input sample is estimated by the discriminator using a single scalar. However, for high dimensional data such as images, we naturally perceive them from more than one angles and deduce whether it life-like based on multiple criteria. As shown in Fig when portrait is given, one might focus on its facial structure, skin tint, hair texture and even details like iris and teeth if allowed, each of which indicates a different aspect of realness. Based on this observation, the single scalar could be viewed as an abstract or summarization of multiple measures, which together reflect the overall realness of an image. Such a concise measurement may convey insufficient information to guide the generator, potentially leading to well-known issues such as mode-collapse and gradient vanishing.
In this paper, propose generalize the standard framework (Goodfellow et al. 2014) by treating realness as a random variable, represented as a distribution rather than a single scalar. We refer to
"Equal contribution.
'Code will he available at https://giiii https://github.com com/ kam1 107 /Rea 1 InessGAI
(a)
(b)
Figure 1: The perception of realness depends various aspects. (a) Hluann-percevve flawless. (b) Potentially reduced realness due to: inharmonious facial structure/comppnents, unnatural back- ground, abnormal style combination and texture distortion.

---

Published as a conference paper at ICLR 2020
such generalization RealnessGAN. The learning process of RealnessGAN abide by the stan- dard setting, but in distributional form. While the standard GAN can be viewed as a special case of RealnessGAN, RealnessGAN and the standard GAN share similar theoretical guarantees. Lee. Re- alnessGAN converges to a Nash-equilibrium where the generator and the discriminator reach their optimalities. Moreover, by expanding the scalar realness score into distributional one, the dis- criminator 1) naturally provides stronger guidance to the generator G where G needs to match not only the overall realness (as in the standard GAN), but the underlying realness distribution as well. Consequently, RealnessGAN facilitates G to better approximate the data manifold while generat- ing decent samples. As shown the experiments, based on rather simple DCGAN architecture, RealnessGAN could successfully learn from scratch to generate realistic images at 1024* 1024 res- olution.
2
REALNESSGAN
2.1 GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial network jointly learns a generator G and discriminator D, where G attempts to generate samples that are indistinguisaable from the real ones, and D) classifies generated and real samples. In the original work of (Goodfellow et al., 2014), the learning process of D and G follows a minimax game with value function V(G, D);
min max (G, = Eerpan oog D (x)) + + Exrp, [log(1 - D(G(z)))), D
(1)
E lo (xx) 0)] + En-p, [lo((1 D(æ))),
(2)
where the approximated data distribution Pg is defined by a prior p= on input latent variables and G. As proved by Goodfellow et al.) (2014). under such a learning objective, the optimal D satisfies
for a fixed G. Fixing D at its optimal, the optimal G satisfies Ps = Pdata- The theoretical guarantees provide strong supports for GAN's success in many applications (Rad-
(x))
Pasx (a)
ford et al. 2015 Yu et al., 2017; Zhu et al. 2017 Dai et al. 2017), and inspired multiple variants (Arjovsky et al. 2017 Mao et al. 2017: Zhao et al. 2017: Berthelot et al. 2017) to improve the
original design. Nevertheless, a single sealar is constantly adopted as the measure of realness, while
the concept of realness is essentially a random variable covering multiple factors, e.g. texture and overall configuration in the case of images. In this work, we intend to follow this observation, encouraging the discriminator D to learn realness distribution.
2.2 DISTRIBUTIONAL VIEW ON REALNESS
We start by substituting the scalar output of discriminator D with distribution Prealness so that for an input sample x. D(x) = {Prealness (z, u); u E 2), where Q is the set of outcomes of Preatness Each outcome
be viewed 1 in equation 2 are used as two virtual ground-truth scalars that respectively represent the realness of real and fake images, we also need two virtual ground-truth distributions to stand for the realness distributions of real and fake images. We refer to these two distributions as A, (real) and Ao (fake), which are also defined on Q. As in the standard GAN where 0 and can be replaced with other scalars such as -1 and 1, there are various choices for A1 and Ag. Factors lead to good pair of Ai and Ao will be discussed later. Accordingly, the difference between two scalars replaced with the Libblrr Kullback- eibler (KI.) divergence. The minimax game between generator G and distributional discriminator D thus becomes
potential realness measure, estimated via some criteria. While 0 and
max min (GG, D) E [DKL (AIlD((()) + Ex (A) G D
An immediate observation is that if we let Preainess be a discrete distribution with two outcomes (ug. uz }, and set Ao(up) = A1 (u1) = 1 and Au(uz) A1 ) 0, the updated objective in equa- tion 3 can be explicitly converted to the original objective in equation 2 suggesting RealnessGAN is generalized version of the original GAN.
Following this observation, we then extend the thcoretical analysis in Goodfellow et al. (2014) to the case of RealnessGAN. Similar to Goodfellow et al. (2014), our analysis concerns the space of
2

---

Published as a conference paper at ICLR 2020
probability density functions, where D and G are assumed to have infinite capacities. We start from finding the optimal realness discriminator for any given generator G.
Theorem 1. When G fixed, for any outcome u and imput sample æ, the optimal discriminator D satisfies
Da u) Ai(u)páano(t (at) + An(u)p((a))
(4)
Pdina (3) + Py(æ))
Proof. Given fixed G, the objective of D is:
min V(G, D) Ree-pon
+ Bamp, [DKL (Aol)
(5)
/ (dana(x)) (2)) / Ai(w))ogg Al(u) du + po(x)) / Ao(u) log D(x, u)
Aa(u)
D(0, du
dæ.
(6)
/ (Ptna())))(A)) P Pg(())((AD)) dr
[ (ptam((:)) (I) + Pa (o)) log D(x, u)dudx,
where ((A1) and h(AA)) are their entropies. Marking the first term in equation as C1 since it is irrelevant to D, the objective thus equivalent to:
min V(G,D) / (Pdaul (x) Pg(x)) /.
Pulata (a: A1 + (p())) log D(x, u)dudr + C1.
Pdata x)) + P2(x)
(8)
where Pa (u)
Peass (s)) (c) Po
is distribution defined on ft Let C2 Pdata (az) + Py (2)),
we then have
min vi (- Pa Pa(u) log D(x, u)du + h(pw) - h(pww) dx,
(9)
= / CoDia(Pal + Coh(p=)da.
(10)
Observing equation 10 one can see that for any valid æ, when DKL. (p D(x)) achieves its mini- mum, D obtains its optimal D*, leading (* Prs which concludes the proof.
Next, we move on to the conditions for G to reach its optimal when D = DG
Theorem 2. When D D'c and there exists an outcome u Q such that Ai(u) # Ao(u), the maximum of (G, is achieved if and only Pg
Pdata-
Proof. When Pg = -data- D2r u) Ar(u))AAlv)) we have:
(A)
2.A0(u) du. A1(u) + An(u)
V*(G,D') = A log A1(u) + Ao(u) + +oll)logg
(11)
Subtracting V*GG from V(G,DE) gives:
V'G V(G, (3) V*(
/ (ptmaa A1 (u) + (2x) log
(piau (x) + Py (x)) + An(u)) dudæ,
2( (z) Ai (u) Pg xz) An()))
(12)
Pdataa (aaaa(x))AA An + (2(x))Ao(u)
Paz A1
2
log
duda,
2
(pazz (z) 1 A Anlu))
(13)
A1 + Po
(Pdata + (o) + Ao),
=-2DKI
(14)
2
4
3

---

Published as a conference paper at ICLR 2020
Since (*, D2) ing G' (G) The optimal (% is achieved if and only the KL divergence reaches its minimum, where:
) is constant with respect to G, maximizing V(G, Dī) is equivalent to maximiz-
Pdaia + Pg
(Pdata + +Pg)(A + Ao)
(15)
2 (Pdata Py)(A1 - -A))= =0,
4
(16)
for any valid æ and u. Hence, as long as there exists valid a that Ai(u) + Ao(u), we have Pdata = Pg for any valid
2.3 DISCUSSION
The theoretical analysis gives more insights on RealnessGAN.
Number of outcomes: according to equation I6, each u E Ω with Au(u) * A1 may work as a constraint, pushing Pg towards Pdata In the case of discrete distributions, along with the increment of the number of outcomes, the constraints imposed on G' accordingly become more rigorous and can cost G more effort learn. This is due to the fact that having more outcomes suggests grained shape of the realness distribution for G to match. In S we verified that it is beneficial to update G an increasing number of times before Dss update the number of outcomes grows.
fine-
Effectiveness of anchors: view equation 16 as a cost function to minimize, when Pdata * Pg- for some u E 92, the larger the difference between A1 (u) and Ao(u) is, the stronger the constraint on G becomes, Intuitively. RealnessGAN can be more efficiently trained if we choose Ao and Ai to be adequately different.
Objective of G: according to equation 3 the best way to fool D is to increase the KL divergence between D(x) and the anchor distribution Ag of fake samples, rather than decreasing the KL diver- gence between ((x) and the anchor distribution A, of real samples. It's worth noting that these two original work (Goodfellow et al.) 2014). An intuitive explanation that, in the distributional view of realness, realness distributions of real samples are not necessarily identical. is possible that each them corresponds to a distinct one. While A1 only serves as an anchor, it is ineffective to drag all generated samples towards the same target.
objectives are equivalent
Flexibility of RealnessGAN: as a generalization of the standard framework, it is straightforward to integrate RealnessGAN with different GAN architectures, such as progressive GANs (Karras al., 2018 2019) and conditional GANs (Zhu et al.) [017]] Ledig et al.] 2017). Moreover, one may also combine the perspective of RealnessGAN with other reformulations of the standard GAN, such as replacing the KL divergence in equation] with the Earth Mover's Distance.
2.4 IMPLEMENTATION
In our implementation, the realness distribution Prealness characterized as a discrete distribution over N outcomes Ω {uo, U1, ..., -. Given input sample x, the discriminator D returns N probabilities on these outcomes, following:
(x))
Preatness (x, u.) E, E
(17)
where db = (4b, vh....,, ..., 11) are the parameters of D. Similarly, Ai and An are discrete distri- butions defined
As shown in the theoretical analysis, the ideal objective for G is maximizing the KL divergence between ((x) of generated samples and An:
(Gobjectivel)
min Ezzpa PPL(
(18)
However, as the discriminator D is not always at its optimal, especially in the early stage, directly applying this objective in practice could only lead generator with limited generative power. Con- sequently, a regularizer is needed to improve G. There are several choices for the regularizer, such as the relativistic term introduced in Molicoeur-Martinaal 2019) that minimizes the KI. divergence
4

---

Published as a conference paper at ICLR 2020
between (x)) of generated samples and random real samples, or the term that minimizes the KL divergence between A1 and ((x) of generated samples, each of which leads to a different objective:
(Gohjective2)
min G
Ex-penx--p ( ( - Exrpa (Kol (Rr
(19)
(Gohjectives)
min G
Ezmp, (PkL(A) Ezmp, (okL(()l
(20)
In Sec 4 these objectives are compared. And the objective in equation T9 is adopted the default choice.
Feature resampling. In practice, especially in the context of images, we are learning from a limited number of discrete samples coming from a continuous data manifold. We may encounter issues caused by insufficient data coverage during the training process. Inspired by conditioning augmen- tation mentioned in (Zhang et al., 2016). we introduce a resampling technique performed on the realness output to augment data variance. Given
{æo, .... 2M-1} of size M, a Gaus- sian distribution, N (fss, o:) is fitted on { (mo), (b;())) (bM(MM))), which are logits computed by D on i-th outcome. We then resample M new logits (1b, -m (m 1); NN(,,,)) for i-th outcome and use them succeedingly.
The randomness introduced by resampling benefits the training of RealnessGAN in two aspects. First of all, it augments data by probing instances around the limited training samples, leading to more robust models. Secondly, the resampling approach implicitly demands instances of (a)( to be homologous throughout the mini-batch such that each outcome reflects realness consistently across samples. We empirically found the learning curve of RealnessGAN is more stable if feature resampling is utilized, especially in the latter stage, where models are prone to overfit.
RELATED WORK
Generative adversarial network (GAN) was first proposed in (Goodfellow et al. 2014), which jointly learns discriminator D and generator G in an adversarial manner. Due its outstanding learning ability. GANs have been adopted in various generative tasks (Radford et al.] 2015: Yu et al. 2017] Zhu et al. 2017). among which Deep Convolutional GAN (DCGAN) (Radford et al. 2015) has shown promising results in image generation.
Although remarkable progress has been made. GAN is known to suffer from gradient diminish- ing and mode collapse. Variants of GAN have been proposed targeting these issues. Specifically, Wasserstein GAN (WGAN) Arjovsky et al. (2017) replaces JS-divergence with Earth-Mover's 's Dis- tance, and Least-Square GAN (LSGAN) (Mao et al. 2017) transforms the objective of G to Pearson divergence. Energy-based GAN (EBGAN) (Zhao et 2017) and Boundary Equilibrium GAN (BE- GAN) (Berthelot et al., 2017) employ a pre-trained auto outo-encoder as the discriminator, learning to distinguish between real and generated samples via reconstruction. Besides adjusting the objective of GAN, alternative approaches include more sophisticated architectures and training paradigms. Generally, ProgressiveGAN (Karras et al. (0018) and StyleGAN (Karras et al., (2019) propose pro- gressive paradigm, which starts from a shallow model focusing on low resolution, and gradually grows into a deeper model to incorporate more details as resolution grows. On the other hand, COCO-GAN (Lin al., 2019) tackles high resolution image generation in divide-and-conque strategy. It learns to produce decent patches at corresponding sub-regions, and splices the patches to produce higher resolution image.
It's worth noting that many works on generative adversarial networks have discussed 'distributions' (Goodfellow et al., 22144; Radford et al., 2015) Arjovsky et al., [017]. which usually refers to the underlying distribution of samples. Some of the existing works aim to improve the original objec- tive using different metrics to measure the divergence between the learned distribution P, and the real distribution Paata Nevertheless, single scalar is constantly adopted represent the concept of realness. In this paper, we propose complementary modification that models realness as a random variable follows the distribution Prealness. In the future work, may study the combination of real- ness discriminator and other GAN variants to enhance the effectiveness and stability of adversarial learning.
5

---

Published as a conference paper at ICLR 2020
Real Data
Std Loss
WGMMGP
LSGAN HingeGAN
Reainess
Figure 2: Left: real data sampled from the mixture of Gaussian distributions. Right: samples generated by Std-GAN, WGAN-GP, LSGAN, HingeGAN and RealnessGAN.
EXPERIMENTS
In this section we study RealnessGAN from multiple aspects. Specifically, 1) we firstly focus on RealnessGANNs mode coverage ability on synthetic dataset. 2) Then we evaluate RealnessGAN on CIFARIO 10 (32*32) (Krizhevsky) 2209) and CelebA (256#256) (Liu et al. (0015) datasets qualitatively and quantitatively. 3) Finally we explore RealnessGAN on high-resolution image generation task, which is known to be challenging for unconditional non proggeesee architectures. Surprisingly, the FFHQ dataset (Karras et al. [2019), RealnessGAN managed to generate images at the 1024* 1024 non-progressive architecture. We compare RealnessGAN
resolution based
other popular
objectives in generative adversarial learning. including the standard GAN (Sid-GAN) (Radford et al., 2015), WGAN-GP (Arjovsky et al., 2017), HingeGAN (Zhao al. 2017) and ISGAN (Mao al. 2017
For experiments on synthetic dataset, we use a generator with four fully-connected hidden layers, each of which has 400 units, followed by batch normalization and ReLU activation. The discrimina- tor has three fully-connected hidden layers, with 200 units each layer. LinearMaxout with 5 maxout pieces are adopted and no batch normalization is used in the discriminator. The latent input x is 32-dimensional vector sampled from Gaussian distribution ((, I). All models are trained using Adam (Kingma & Ba 2015) for 500 iterations.
On worldworld datasets, the network architecture is identical the DCGAN architecture in Radford et al. (0055). with the prior p= (=(z)
188dimmensionll Gaussian distribution 1). Models are trained using Adam (Kingma & Ba 2015) for 520k iterations. To guarantee training stability, we adopt settings that proved be effective for baseline methods. Batch normalization (loffe & Szegedy 2015) is used in G. and spectral normalization (Miyato et al., 2018 is used in D. For WGAN-GP we use ir le 4, 8 0.5, B2 = 0.9, updating D for 5 times per G's update Gulrajani et al. 2017); for the remaining models, we use Ir = 2e - 4, 31 = =0.,, 32 0.999, time per G's update (Radford et al., 2015). Fréchet Inception Distance (FID) (Heusel et al. 2017) and Sliced Wasserstein Distance (SWD) (Karras et al. 2018) are reported as the evaluation metrics. Unless otherwise stated, A, and Ao are chosen resemble the shapes of two normal distributions with a positive skewness and a negative skewness, respectively. In particular, the number of outcomes are empirically set 51 for CelebA and FFHQ datasets, and 3 for CIFAR10 dataset.
updating 1 for
4.1 SYNTHETIC DATASET
Since Piata is usually intractable on real datasets, we use a toy dataset to compare the learned dis- tribution Pa and the data distribution Pdata- The toy dataset consists of 100, 000 2D points sampled from a mixture of 9 isotropic Gaussian distributions whose means are arranged in a 3 by 3 grid, with variances equal to 0.05. As shown in Fig the data distribution Paat contains 9 welly separated modes, making it a difficult task despite its low-dimensional nature.
To evaluate pg. we draw 10, 000 samples and measure their quality and diversity. As suggested in (Dumoulin et al., 2016). we regard a sample as of high quality if it is within 4a from the of its nearest Gaussian. When a Gaussian assigned with more than 100 high quality samples, we consider this mode of Pdaaa is recovered in Pg- Fig visualizes the sampled points of different methods, where LSGAN and HingeGAN suffer from significant mode collapse, recovering only a single mode. Points sampled by WGAN-GP are overly disperse, and only 0.03% of them are of high quality. While Sıd-GAN recovers 4 modes in Paa with 32.4% high quality samples, 8 modes are recovered by RealnessGAN with 60.2% high quality samples. The average of these high quality samples in Std-GAN and RealnessGAN are respectively 0.083 and 0.043. The results suggest that treating realness random variable rather than single scalar leads to more strict discriminator

---

Published as a conference paper at ICLR 2020
Number of Outcomes
out=2, Giar=]
outes, Giturel
cut=10 Gier=i
gut=20, Sitr-1
out-30. Gitr=1
cut=600 Gitees
k_G=1. K_D=1
out=2 Gttr=l
outeS, Gite=]]
out=10. Gtor-s
Gtor-1d
36
Converge
1.0
#recoveree ed modes
samples
0.8
0.7
0.6
0.4
10
20
30
#outcomes
First row: the results of RealnessGAN when fixing ke kp = and increasing the number of outcomes. Second row: the results of RealnessGAN when kg is properly increased. Bottom curves: under the settings of second row, the ratio of high quality samples and the number of recovered modes.
Figure 3:
that criticizes generated samples from various aspects, which provides more informative guidance. Consequently, Pg learned by RealnessGAN is more diverse and compact.
We further study the effect of adjusting the number of outcomes the realness distribution Prealness on this dataset. To start with, we fix kg and kn to be 1, which are the number of updates for G and D in one iteration, and adjust the number of outcomes of Prealness Ag and AA, As shown in the first row of Fig it can be observed that in general G' recovers less modes as the number of outcomes grows, which a direct result of 7) becoming increasingly rigorous and imposing more constraints on An intuitive solution is to increase kG such that G is able to catch up with current D). The second row Fig demonstrates the converged cases achieved with suitable kgs, suggesting RealnessGAN effective when sufficient learning capacity is granted to G. The ratio of high quality samples ThQ and the number of recovered modes 7odde these cases are plotted Fig 3 The two curves imply that besides kg, T'H and Roode all positively related the number of outcomes, validating that measuring realness from more aspects leads to a better generator.
4.2 REAL-WORLD DATASETS
As GAN has shown promising results when modeling complex data such as natural images, we evaluate RealnessGAN on real-world datasets, namely CIFARIO. CelebA and FFHQ, which re- spectively contains images at 32*32, 256*256 and 1024*1024 resolutions. The training curves of baseline methods and RealnessGAN on CelebA and CIFAR10 are shown in Fig The qualitative results measured in FID and SWD are listed in Tab We report the minimum, the maximum, the mean and the standard deviation computed along the training process. On both datasets, compared to baselines, RealnessGAN obtains better scores in both metrics. Mcantime, the learning process of RealnessGAN smoother and steadier (see SD in Tab and curves in Fig Samples of generated images on both datasets are included in Fig
On FFHQ, we push the resolution of generated images to 1024*1024 which known to be challeng- ing especially for non-progressive architecture. As shown in Fig despite building relatively simple DCGAN architecture, RealnessGAN able to produce realistic samples from scratch at such high resolution. Quantitatively, RealnessGAN obtains FID score of 17.18. For reference,

---

Published as a conference paper at ICLR 2020
LEGAN HEReSAA
LSG4N Ringeldn
227
0
Postt
mrantie (354)
FTD CelehA
(b) SWD CelebA
510 LOC4N KingoiA
LEGAN
227
30
Portt
Irovisns
FID CIFARIO
SWD CIFARIO
Figure 4: Training curves of different methods in terms of FID and SWD on both CelebA and CIFARIO, where the raise of curves the later stage indicate mode collapse. Best viewed color.
Table 1: Minimum (min), maximum (max), mean and standard deviation (SD) of FID and SWD on CelebA and CIFAR1O, calculated 20k, 30k, iterations. The best indicators in baseline methods are underlined.
FID
SWD (x (x10))
Method
Min
Max
Mean
SD
Min
Max
Mean
SD
Std-GAN
27.02
70.43
34.85
9.40
14.81
68.06
30.58
15.39
WGAN-GP
70.28
104.60
81.15
8.27
17.85
30.56
22.09
2.93
CelebA
LSGAN
30.76
57.97
34.99
5.15
16.72
23.99
20.39
2.25
HingeGAN
25.57
75.03
33.89
10.61
14.91
54.30
28.86
10.34
RealnessGAN
23.51
81.3
30.82
7.61
12.72
31.39
17.11
3.59
Sid-GAN
38.56
88.68
47.4
15.96
28.76
57.71
37.55
7.02
WGAN-GP
41.86
79.25
46.96
5.57
28.17
36.04
30.98
1.78
CIFARIO
LSGAN
42.01
75.06
48.41
7.72
31,99
40.46
34.75
2.34
HingeGAN
42.40
117.49
57.30
20.69
32.18
61.74
41.85
7.31
RealnessGAN
34.59
102.98
42.30
11.84
22.80
53.38
26.98
5.47
re-implemented StyleGAN Karras et al. 2019) trained under similar setting receives FID score
of 16.12. These results strongly support the effectiveness RealnessGAN, as StyleGAN is one of the most advanced GAN architectures far.
4.3 ABLATION STUDY
The implementation of RealnessGAN offers several choices that also worth digging into. On syn- thetic dataset, we explored the relationship between the number of outcomes and G's update fre- quency. On real-worke dataset, apart from evaluating RealnessGAN affect of feature resampling, different settings Ag and Ag and choices of G's objective.
whole, also studied the
Table 2: Minimum (min), maximum (max),
recampingg
mean and standard deviation (SD) of FID on CelebA using different anchor distributions, calculated at 20k, 30k, iterations.
wrrcaumaing
62
DKL. An)
Min
Max
Mean
SD
45-
1.66
31.01
96.11
40.75
11.83
5.11
26.22
87.98
36.11
9.83
7,81
25,98
85.51
36.30
10.04
Figure 5: Training FID curves Realness- GAN with and without feature re-sampling.
11.05
23.51
81.30
30.82
7.61
8

---

Published as a conference paper at ICLR 2020
Figure 6: Samples generated by RealnessGAN trained with the ideal objective (cquation 18 Top- row: samples when DKL (AIl|Au) 11.05. Bottom-row: samples when (x An)) 33.88.
Table 3: FID scores of G on CIFARIO, trained with different objectives.
100
mI KUAA, AJP/SI00 KUIK OIDICIA!! KUDDHIEDORLO 21J.311
160
G Objective
FID
107
Objectivel (equation
36.73
Objective2 (equation
34.59
2%
Objective3 (equation 20
36.21
oerations 110k)
DCGAN
38.56
WGAN-GR
41.86
Figure 7: Training curves of RealnessGAN on CelebA using objective2 (equation 19 and objective3 (equation 20
LSGAN
42.01
HingeGAN
42.40
Feature Resampling, Fig shows the training curves RealnessGAN with and without feature resampling. It can be noticed that despite the results similar, feature resampling stabilizes the training process especially in the latter stage.
Effectiveness of Anchors. Tab reports the results of varying the KL divergence between anchor distributions Ao and A The FID score indicates that, as the KL divergence between Ap and A1 increases, RealnessGAN tends to perform better, which verifies larger difference between anchor distributions imposes stronger constraints on G. To further testify, two different pairs of anchors with similar KL divergences (11.95 and 11.67) are exploited and they yield comparable FID scores (23.98 and 24.22).
discussion in Sec 2.33 that
Objective of G. As mentioned in 2.c theoretically, the objective of G is maxg Eg-p, (Aol need either a pair of Ag and A1 that are drastically different, or an additional constraint to aid this objective. Fig G shows that, with the ideal objective alone, when the KL divergence between Ao and A, is sufficiently large, on CelebA we could obtain generator with limited generative power. On the other hand, by applying constraints as discussed in See 2.4 G can learn to produce more realistic samples as demonstrated Fig Similar results are observed on CIFAR10, where RealnessGAN obtains comparable FID scores with and without constraints, shown in Tab[] 7 also provides the training curves of RealnessGAN on CelebA using these two alternative objectives.
However, in practice, since D is not always optimal, we
5 CONCLUSION
In this paper, we extend the view of realness in generative adversarial networks under a distributional perspective. In proposed extension, RealnessGAN, represent the concept of realness realness distribution rather than single scalar. so that the corresponding discriminator estimates realness from multiple angles, providing more informative guidance the generator. We prove RealnessGAN has theoretical guarantees on the optimality of the generator and the discriminator. datasets, RealnessGAN also demonstrates the ability of effectively
On both synthetic and and steadily capturing the underlying data distribution.

---

Published as a conference paper at ICLR 2020
Figure 8: Images sampled from RealnessGAN, respectielly trained on CIFARIO (top), CelebA (mid- dle) and FFHQ (bottom).
Acknowiedgement We thank Zhizhong Li for helpful discussion on the theoretical analysis. This work is partially supported by the Collaborative Research Grant "Large-scale Multi-modality Analytics" from Sense (CUHK Agreement No. TS1712093), the General Research Funds (GRF) Hong Kong (No. 14209217 and No. 14205719), Singapore MOE AcRF Tier 1, NTU SUG, and NTU NAP.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv: 07877, 2017.
Wasserstein gan.
arXiv preprint
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative ad- versarial networks. arXiv preprint
2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ICLR, 2019.
Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. Towards diverse and natural image descrip- tions via conditional gan. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2970-2973 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky. Olivier Mastropi- etro, and Aaron C. Courville. Adversarially learned inference. ArXiv, abs/1600.00704, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 5779, USA, 2017. Curran Asso- ciates Inc. ISBN 978-1-5100 978-1-5108-6096-1 URL htt.:http//ee /cll acm. org/ c tat ion cfm?ic d=
3295222. 32953227
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference Inter- national Conference on Machine Learning- Volume 37, ICML'15, pp. 448 448-458- JMLR.org, 2015.
URL http: acm. httg/iitatiion.. ctw.iiioonccmmmccc. efm?id-mmii-eii. 3045118. 30411cm..1...... 3045167
Alexia Jolicoeur-Martnnauu The relativistic discriminator: a key element missing from standard gan. In ICLR, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL http////peeeeeeee///////eee net
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), ), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
7/
6980
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009,
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo- realistic single image super-resolution using generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 105-114, July 2017. doi: 10.1109/CVPR.2011..19
Chich Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann- Chen. COCO-GAN: Conditional coordinate generative adversarial network, 2019. https:
//openrevien net. forum? w 4Aas0977

---

Published as a conference paper at ICLR 2020
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks, In ICCV, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018. URL/attps://ooeeeevvves/ net/ -tt.///mmmmm1111122111
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1rX1v:15/1.06444, 2015.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, and Dim- itris N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative ad- versarial networks. 2017 IEEE International Conference Computer Vision (ICCV), 5908- 5916, 2016.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-base based generative adversarial network. In ICLR, 2017.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
12


Reviews:
[{"review_id": "rJldMhpadH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post rebuttal: The authors' responses have addressed most of my concerns, and I've raised my rating from 3 to 6.\n\n----------------------------------------\n\nSummary:\nThis paper extends the discriminator of GAN to use a distributional output (multiple scalars) instead of a single scalar. As a result, the trained GAN becomes robust to the mode collapse.\n\nPros:\n- The proposed method is clearly written and well-justified (e.g., Theorem 2).\n- Extension of the relativistic GAN [1] to the proposed setting is interesting.\n- The authors demonstrate that vanilla DCGAN architecture can generate high-fidelity (1024x1024) images.\n\nCons:\n\n1. An ensemble of discriminators?\n\nThe authors use multiple scalars to consider diverse factors of the realness. However, it is simply an ensemble of discriminators [2] in a spirit. As each discriminator focus on different factors, it is not surprising that the generator becomes robust to the mode collapse. Also, recent work on mode collapse (e.g., [3]) shows better results on the mixture of gaussian experiments even using a single discriminator. At least, the authors should compare their method with the ensemble methods and claim the advantage over them.\n\n2. Choice of the anchor distributions.\n\nThe choice of anchor distributions A_0 and A_1 are not specified. While the authors provide some partial results in Table 2, it would be worthwhile to clarify the experimental details and justify them.\n\n3. Role of each outcome u_i?\n\nThe authors claim that each outcome u_i corresponds to the different factors of realness. However, the role of learned u_i is not investigated. Also, one may enforce u_i to learn different factors by promoting diversity of them, e.g., decrease their cosine similarity [4].\n\nMinor comments:\n- The word \"support\" [5] is misused. The support itself means the set of non-zero elements, hence the authors should use the word \"outcome\" (or \"sample\") instead of \"support\".\n- The notation is not consistent. For example, the authors may use \"x \\sim p_data(x)\" (specify variable) or \"z \\sim p_z\" (omit variable), but not both.\n- Numbering is not consistent. For example, \"Tab.4.2.\" should be changed to \"Tab.2.\" for consistency.\n\n\n[1] Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard GAN. ICLR 2019.\n[2] Durugkar et al. Generative Multi-Adversarial Networks. ICLR 2017.\n[3] Xiao et al. BourGAN: Generative Networks with Metric Embeddings. NeurIPS 2018.\n[4] Elfeki et al. GDPP: Learning Diverse Generations Using Determinantal Point Process. ICML 2019.\n[5] https://en.wikipedia.org/wiki/Support_(mathematics)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJldG7XbcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to improve upon GANs by considering to infer the distribution of realness instead of binary true/false labels in the discriminator side. They carry out this idea with some theoretical arguments, and their method is shown to perform very well in empirical experiments on one synthetic dataset, and three real world datasets:CelebA, CIFAR-10, and FFHQ. Overall I feel this is a well presented paper with a simple yet interesting idea and solid results. The authors are encouraged to share their code and results to public.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eQF85RYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I raised my score from 3 to 6 after the authors addressed most of my comments.\n\n====================================================\n\nThis paper propose a new GAN formulation where the Discriminator outputs a discrete probability distribution instead of a scalar for each inputs. This discrete probability distribution outputted by the discriminator is then compared, using the KL divergence, to two different reference distributions according to if the image is from the dataset or generated. The paper show that the proposed approach is a generalization of the standard GAN. They then prove that under some condition that similarly to GAN at the optimum $p_g = p_data$. In addition the paper propose two tricks 1) they include an additional term in the loss for the generator, such that the generator is also trying to minimize the KL between the discriminator distribution for a generated and the discriminator distribution for a real samples.  2) They propose some procedure to resample the logits of the Discriminator. \nThey show on a toy example how increasing the dimension of the distribution of the Discriminator also increases the performance. They also show that their method can slightly improve performance on CelebA and CIFAR10 and that it can also scale to high resolution images on FFHQ.\n\nI'm in favour of rejecting this paper. In particular I find the method not very well motivated for several reasons. First it's never explained in the text how the reference distribution $A_0$ and $A_1$ are chosen, and so it's not clear what they represent. Second it's not clear why we need the two tricks or wether the method would also work without the proposed tricks.\n\nMain Argument:\n- Please provide an explanation how $A_0$ and $A_1$ are chosen and what are the $A_0$ and $A_1$ chosen in the experiments. This seems very critical to the performance of the method as shown in table 2 but explained nowhere in the paper.\n- What is the effect of the \"Relativistic\" loss ? Does the method work if we remove it ? An extended ablation study would be nice and give more insight on what is really important for the performance of the method ?\n- The author claim in the experimental section that their method perform betters on both datasets, however when looking at table 1, WGAN-GP performs better on CIFAR10 and when looking at the standard deviation we can see that the improvment is not significative.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel GAN formulation where the discriminator outputs discrete distributions instead of a scalar. The objective uses two "anchor" distributions that correspond to real and fake data. There were some concerns about the choice of these distributions but authors have addressed it in their response. The empirical results are impressive and the method will be of interest to the wide generative models community. 
Decision: Accept (Spotlight)