Title: REAL OR NOT REAL, THAT IS THE QUESTION

Abstract: While generative adversarial networks (GAN) have been widely adopted in various topics, in this paper we generalize the standard GAN to a new perspective by treating realness as a random variable that can be estimated from multiple angles. In this generalized framework, referred to as RealnessGAN 1 , the discriminator outputs a distribution as the measure of realness. While RealnessGAN shares similar theoretical guarantees with the standard GAN, it provides more insights on adversarial learning. Compared to multiple baselines, RealnessGAN provides stronger guidance for the generator, achieving improvements on both synthetic and real-world datasets. Moreover, it enables the basic DCGAN (Radford et al., 2015) architecture to generate realistic images at 1024*1024 resolution when trained from scratch.

1. INTRODUCTION:

The development of generative adversarial network (GAN) (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017) is one of the most important topics in machine learning since its first appearance in (Goodfellow et al., 2014) . It learns a discriminator along with the target generator in an adversarial manner, where the discriminator distinguishes generated samples from real ones. Due to its flexibility when dealing with high dimensional data, GAN has obtained remarkable progresses on realistic image generation (Brock et al., 2019) .

In the standard formulation (Goodfellow et al., 2014) , the realness of an input sample is estimated by the discriminator using a single scalar. However, for high dimensional data such as images, we naturally perceive them from more than one angles and deduce whether it is life-like based on multiple criteria. As shown in Fig. 1 , when a portrait is given, one might focus on its facial structure, skin tint, hair texture and even details like iris and teeth if allowed, each of which indicates a different aspect of realness. Based on this observation, the single scalar could be viewed as an abstract or a summarization of multiple measures, which together reflect the overall realness of an image. Such a concise measurement may convey insufficient information to guide the generator, potentially leading to well-known issues such as mode-collapse and gradient vanishing.

In this paper, we propose to generalize the standard framework (Goodfellow et al., 2014) by treating realness as a random variable, represented as a distribution rather than a single scalar. We refer to such a generalization as RealnessGAN. The learning process of RealnessGAN abide by the standard setting, but in a distributional form. While the standard GAN can be viewed as a special case of RealnessGAN, RealnessGAN and the standard GAN share similar theoretical guarantees. i.e. Re-alnessGAN converges to a Nash-equilibrium where the generator and the discriminator reach their optimalities. Moreover, by expanding the scalar realness score into a distributional one, the discriminator D naturally provides stronger guidance to the generator G where G needs to match not only the overall realness (as in the standard GAN), but the underlying realness distribution as well.

Consequently, RealnessGAN facilitates G to better approximate the data manifold while generating decent samples. As shown in the experiments, based on a rather simple DCGAN architecture, RealnessGAN could successfully learn from scratch to generate realistic images at 1024*1024 resolution.

2. REALNESSGAN:



2.1. GENERATIVE ADVERSARIAL NETWORKS:

Generative adversarial network jointly learns a generator G and a discriminator D, where G attempts to generate samples that are indistinguishable from the real ones, and D classifies generated and real samples. In the original work of (Goodfellow et al., 2014) , the learning process of D and G follows a minimax game with value function V (G, D):

EQUATION

EQUATION

where the approximated data distribution p g is defined by a prior p z on input latent variables and G. As proved by Goodfellow et al. (2014) , under such a learning objective, the optimal D satisfies D

* G (x) = pdata(x)

pdata(x)+pg(x) for a fixed G. Fixing D at its optimal, the optimal G satisfies p g = p data . The theoretical guarantees provide strong supports for GAN's success in many applications (Radford et al., 2015; Yu et al., 2017; Zhu et al., 2017; Dai et al., 2017) , and inspired multiple variants (Arjovsky et al., 2017; Mao et al., 2017; Zhao et al., 2017; Berthelot et al., 2017) to improve the original design. Nevertheless, a single scalar is constantly adopted as the measure of realness, while the concept of realness is essentially a random variable covering multiple factors, e.g. texture and overall configuration in the case of images. In this work, we intend to follow this observation, encouraging the discriminator D to learn a realness distribution.

2.2. A DISTRIBUTIONAL VIEW ON REALNESS:

We start by substituting the scalar output of a discriminator D with a distribution p realness , so that for an input sample x, D(x) = {p realness (x, u); u ∈ Ω}, where Ω is the set of outcomes of p realness . Each outcome u can be viewed as a potential realness measure, estimated via some criteria. While 0 and 1 in equation 2 are used as two virtual ground-truth scalars that respectively represent the realness of real and fake images, we also need two virtual ground-truth distributions to stand for the realness distributions of real and fake images. We refer to these two distributions as A 1 (real) and A 0 (fake), which are also defined on Ω. As in the standard GAN where 0 and 1 can be replaced with other scalars such as -1 and 1, there are various choices for A 1 and A 0 . Factors lead to a good pair of A 1 and A 0 will be discussed later. Accordingly, the difference between two scalars is replaced with the Kullback-Leibler (KL) divergence. The minimax game between a generator G and a distributional discriminator D thus becomes

max G min D V (G, D) = E x∼pdata [D KL (A 1 D(x))] + E x∼pg [D KL (A 0 D(x))].

(3)

An immediate observation is that if we let p realness be a discrete distribution with two outcomes {u 0 , u 1 }, and set A 0 (u 0 ) = A 1 (u 1 ) = 1 and A 0 (u 1 ) = A 1 (u 0 ) = 0, the updated objective in equation 3 can be explicitly converted to the original objective in equation 2, suggesting RealnessGAN is a generalized version of the original GAN.

Following this observation, we then extend the theoretical analysis in Goodfellow et al. (2014) to the case of RealnessGAN. Similar to Goodfellow et al. (2014) , our analysis concerns the space of probability density functions, where D and G are assumed to have infinite capacities. We start from finding the optimal realness discriminator D for any given generator G.

Theorem 1. When G is fixed, for any outcome u and input sample x, the optimal discriminator D satisfies

EQUATION

Proof. Given a fixed G, the objective of D is:

EQUATION

EQUATION

= -

EQUATION

where h(A 1 ) and h(A 0 ) are their entropies. Marking the first term in equation 7 as C 1 since it is irrelevant to D, the objective thus is equivalent to:

EQUATION

where

p x (u) = pdata(x)A1(u)+pg(x)A0(u) pdata(x)+pg(x)

is a distribution defined on Ω. Let C 2 = p data (x) + p g (x), we then have

EQUATION

EQUATION

)

Observing equation 10, one can see that for any valid x, when D KL (p x D(x)) achieves its minimum, D obtains its optimal D , leading to D (x) = p x , which concludes the proof.

Next, we move on to the conditions for G to reach its optimal when D = D G . Theorem 2. When D = D G , and there exists an outcome

u ∈ Ω such that A 1 (u) = A 0 (u), the maximum of V (G, D G ) is achieved if and only if p g = p data . Proof. When p g = p data , D G (x, u) = A1(u)+A0(u)

2:

, we have:

EQUATION

EQUATION

= -2

EQUATION

EQUATION

) Since V (G, D G ) is a constant with respect to G, maximizing V (G, D G ) is equivalent to maximiz- ing V (G, D G ).

The optimal V (G, D G ) is achieved if and only if the KL divergence reaches its minimum, where:

EQUATION

EQUATION

for any valid x and u. Hence, as long as there exists a valid u that A 1 (u) = A 0 (u), we have p data = p g for any valid x.

2.3. DISCUSSION:

The theoretical analysis gives us more insights on RealnessGAN.

Number of outcomes: according to equation 16, each u ∈ Ω with A 0 (u) = A 1 (u) may work as a constraint, pushing p g towards p data . In the case of discrete distributions, along with the increment of the number of outcomes, the constraints imposed on G accordingly become more rigorous and can cost G more effort to learn. This is due to the fact that having more outcomes suggests a more finegrained shape of the realness distribution for G to match. In Sec.4, we verified that it is beneficial to update G an increasing number of times before D's update as the number of outcomes grows.

Effectiveness of anchors: view equation 16 as a cost function to minimize, when p data = p g , for some u ∈ Ω, the larger the difference between A 1 (u) and A 0 (u) is, the stronger the constraint on G becomes. Intuitively, RealnessGAN can be more efficiently trained if we choose A 0 and A 1 to be adequately different.

Objective of G: according to equation 3, the best way to fool D is to increase the KL divergence between D(x) and the anchor distribution A 0 of fake samples, rather than decreasing the KL divergence between D(x) and the anchor distribution A 1 of real samples. It's worth noting that these two objectives are equivalent in the original work (Goodfellow et al., 2014) . An intuitive explanation is that, in the distributional view of realness, realness distributions of real samples are not necessarily identical. It is possible that each of them corresponds to a distinct one. While A 1 only serves as an anchor, it is ineffective to drag all generated samples towards the same target.

Flexibility of RealnessGAN: as a generalization of the standard framework, it is straightforward to integrate RealnessGAN with different GAN architectures, such as progressive GANs (Karras et al., 2018; 2019) and conditional GANs (Zhu et al., 2017; Ledig et al., 2017) . Moreover, one may also combine the perspective of RealnessGAN with other reformulations of the standard GAN, such as replacing the KL divergence in equation 3 with the Earth Mover's Distance.

2.4. IMPLEMENTATION:

In our implementation, the realness distribution p realness is characterized as a discrete distribution over N outcomes Ω = {u 0 , u 1 , ..., u N -1 }. Given an input sample x, the discriminator D returns N probabilities on these outcomes, following:

EQUATION

where ψ = (ψ 0 , ψ 1 , ..., ψ N -1 ) are the parameters of D. Similarly, A 1 and A 0 are discrete distributions defined on Ω.

As shown in the theoretical analysis, the ideal objective for G is maximizing the KL divergence between D(x) of generated samples and A 0 :

EQUATION

However, as the discriminator D is not always at its optimal, especially in the early stage, directly applying this objective in practice could only lead to a generator with limited generative power. Consequently, a regularizer is needed to improve G. There are several choices for the regularizer, such as the relativistic term introduced in (Jolicoeur-Martineau, 2019) that minimizes the KL divergence between D(x) of generated samples and random real samples, or the term that minimizes the KL divergence between A 1 and D(x) of generated samples, each of which leads to a different objective:

EQUATION

EQUATION

In Sec.4, these objectives are compared. And the objective in equation 19 is adopted as the default choice.

Feature resampling. In practice, especially in the context of images, we are learning from a limited number of discrete samples coming from a continuous data manifold. We may encounter issues caused by insufficient data coverage during the training process. Inspired by conditioning augmentation mentioned in (Zhang et al., 2016) , we introduce a resampling technique performed on the realness output to augment data variance. Given a mini-batch {x 0 , ..., x M -1 } of size M , a Gaussian distribution N (µ i , σ i ) is fitted on {ψ i (x 0 ), ψ i (x 1 ), ..., ψ i (x M -1 )}, which are logits computed by D on i-th outcome. We then resample M new logits {ψ i (x 0 ), ..., ψ i (x M -1 ); ψ i ∼ N (µ i , σ i )} for i-th outcome and use them succeedingly.

The randomness introduced by resampling benefits the training of RealnessGAN in two aspects.

First of all, it augments data by probing instances around the limited training samples, leading to more robust models. Secondly, the resampling approach implicitly demands instances of ψ i (x) to be homologous throughout the mini-batch, such that each outcome reflects realness consistently across samples. We empirically found the learning curve of RealnessGAN is more stable if feature resampling is utilized, especially in the latter stage, where models are prone to overfit.

3. RELATED WORK:

Generative adversarial network (GAN) was first proposed in (Goodfellow et al., 2014) , which jointly learns a discriminator D and a generator G in an adversarial manner. Due to its outstanding learning ability, GANs have been adopted in various generative tasks (Radford et al., 2015; Yu et al., 2017; Zhu et al., 2017) , among which Deep Convolutional GAN (DCGAN) (Radford et al., 2015) has shown promising results in image generation.

Although remarkable progress has been made. GAN is known to suffer from gradient diminishing and mode collapse. Variants of GAN have been proposed targeting these issues. Specifically, Wasserstein GAN (WGAN) Arjovsky et al. (2017) replaces JS-divergence with Earth-Mover's Distance, and Least-Square GAN (LSGAN) (Mao et al., 2017) transforms the objective of G to Pearson divergence. Energy-based GAN (EBGAN) (Zhao et al., 2017) and Boundary Equilibrium GAN (BE-GAN) (Berthelot et al., 2017) employ a pre-trained auto-encoder as the discriminator, learning to distinguish between real and generated samples via reconstruction. Besides adjusting the objective of GAN, alternative approaches include more sophisticated architectures and training paradigms. Generally, ProgressiveGAN (Karras et al., 2018) and StyleGAN (Karras et al., 2019) propose a progressive paradigm, which starts from a shallow model focusing on a low resolution, and gradually grows into a deeper model to incorporate more details as resolution grows. On the other hand, COCO-GAN (Lin et al., 2019) tackles high resolution image generation in a divide-and-conquer strategy. It learns to produce decent patches at corresponding sub-regions, and splices the patches to produce a higher resolution image.

It's worth noting that many works on generative adversarial networks have discussed 'distributions' (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017) , which usually refers to the underlying distribution of samples. Some of the existing works aim to improve the original objective using different metrics to measure the divergence between the learned distribution p g and the real distribution p data . Nevertheless, a single scalar is constantly adopted to represent the concept of realness. In this paper, we propose a complementary modification that models realness as a random variable follows the distribution p realness . In the future work, we may study the combination of realness discriminator and other GAN variants to enhance the effectiveness and stability of adversarial learning. 

4. EXPERIMENTS:

In this section we study RealnessGAN from multiple aspects. Specifically, 1) we firstly focus on RealnessGAN's mode coverage ability on a synthetic dataset. 2) Then we evaluate RealnessGAN on CIFAR10 (32*32) (Krizhevsky, 2009) and CelebA (256*256) (Liu et al., 2015) datasets qualitatively and quantitatively. 3) Finally we explore RealnessGAN on high-resolution image generation task, which is known to be challenging for unconditional non-progressive architectures. Surprisingly, on the FFHQ dataset (Karras et al., 2019) , RealnessGAN managed to generate images at the 1024*1024 resolution based on a non-progressive architecture. We compare RealnessGAN to other popular objectives in generative adversarial learning, including the standard GAN (Std-GAN) (Radford et al., 2015) , WGAN-GP (Arjovsky et al., 2017) , HingeGAN (Zhao et al., 2017) and LSGAN (Mao et al., 2017) .

For experiments on synthetic dataset, we use a generator with four fully-connected hidden layers, each of which has 400 units, followed by batch normalization and ReLU activation. The discriminator has three fully-connected hidden layers, with 200 units each layer. LinearMaxout with 5 maxout pieces are adopted and no batch normalization is used in the discriminator. The latent input z is a 32-dimensional vector sampled from a Gaussian distribution N (0, I). All models are trained using Adam (Kingma & Ba, 2015) for 500 iterations.

On real-world datasets, the network architecture is identical to the DCGAN architecture in Radford et al. (2015) , with the prior p z (z) a 128-dimensional Gaussian distribution N (0, I). Models are trained using Adam (Kingma & Ba, 2015) for 520k iterations. To guarantee training stability, we adopt settings that are proved to be effective for baseline methods. Batch normalization (Ioffe & Szegedy, 2015) is used in G, and spectral normalization (Miyato et al., 2018) is used in D. For WGAN-GP we use lr = 1e -4, β 1 = 0.5, β 2 = 0.9, updating D for 5 times per G's update (Gulrajani et al., 2017) ; for the remaining models, we use lr = 2e -4, β 1 = 0.5, β 2 = 0.999, updating D for one time per G's update (Radford et al., 2015) . Fréchet Inception Distance (FID) (Heusel et al., 2017) and Sliced Wasserstein Distance (SWD) (Karras et al., 2018) are reported as the evaluation metrics. Unless otherwise stated, A 1 and A 0 are chosen to resemble the shapes of two normal distributions with a positive skewness and a negative skewness, respectively. In particular, the number of outcomes are empirically set to 51 for CelebA and FFHQ datasets, and 3 for CIFAR10 dataset.

4.1. SYNTHETIC DATASET:

Since p data is usually intractable on real datasets, we use a toy dataset to compare the learned distribution p g and the data distribution p data . The toy dataset consists of 100, 000 2D points sampled from a mixture of 9 isotropic Gaussian distributions whose means are arranged in a 3 by 3 grid, with variances equal to 0.05. As shown in Fig. 2 , the data distribution p data contains 9 welly separated modes, making it a difficult task despite its low-dimensional nature.

To evaluate p g , we draw 10, 000 samples and measure their quality and diversity. As suggested in (Dumoulin et al., 2016) , we regard a sample as of high quality if it is within 4σ from the µ of its nearest Gaussian. When a Gaussian is assigned with more than 100 high quality samples, we consider this mode of p data is recovered in p g . Fig. 2 that criticizes generated samples from various aspects, which provides more informative guidance. Consequently, p g learned by RealnessGAN is more diverse and compact.

We further study the effect of adjusting the number of outcomes in the realness distribution p realness on this dataset. To start with, we fix k G and k D to be 1, which are the number of updates for G and D in one iteration, and adjust the number of outcomes of p realness , A 0 and A 1 . As shown in the first row of Fig. 3 , it can be observed that in general G recovers less modes as the number of outcomes grows, which is a direct result of D becoming increasingly rigorous and imposing more constraints on G. An intuitive solution is to increase k G such that G is able to catch up with current D. The second row of Fig. 3 demonstrates the converged cases achieved with suitable k G s, suggesting RealnessGAN is effective when sufficient learning capacity is granted to G. The ratio of high quality samples r HQ and the number of recovered modes n mode in these cases are plotted in Fig. 3 . The two curves imply that besides k G , r HQ and n mode are all positively related to the number of outcomes, validating that measuring realness from more aspects leads to a better generator.

4.2. REAL-WORLD DATASETS:

As GAN has shown promising results when modeling complex data such as natural images, we evaluate RealnessGAN on real-world datasets, namely CIFAR10, CelebA and FFHQ, which respectively contains images at 32*32, 256*256 and 1024*1024 resolutions. The training curves of baseline methods and RealnessGAN on CelebA and CIFAR10 are shown in Fig. 4 . The qualitative results measured in FID and SWD are listed in Tab.1. We report the minimum, the maximum, the mean and the standard deviation computed along the training process. On both datasets, compared to baselines, RealnessGAN obtains better scores in both metrics. Meantime, the learning process of RealnessGAN is smoother and steadier (see SD in Tab.1 and curves in Fig. 4 ). Samples of generated images on both datasets are included in Fig. 8 .

On FFHQ, we push the resolution of generated images to 1024*1024, which is known to be challenging especially for a non-progressive architecture. As shown in Fig. 8 , despite building on a relatively simple DCGAN architecture, RealnessGAN is able to produce realistic samples from scratch at such a high resolution. Quantitatively, RealnessGAN obtains an FID score of 17.18. For reference, our re-implemented StyleGAN (Karras et al., 2019) trained under a similar setting receives an FID score of 16.12. These results strongly support the effectiveness of RealnessGAN, as StyleGAN is one of the most advanced GAN architectures so far.

4.3. ABLATION STUDY:

The implementation of RealnessGAN offers several choices that also worth digging into. On synthetic dataset, we explored the relationship between the number of outcomes and G's update frequency. On real-world dataset, apart from evaluating RealnessGAN as a whole, we also studied the affect of feature resampling, different settings of A 0 and A 1 and choices of G's objective. Effectiveness of Anchors. Tab.2 reports the results of varying the KL divergence between anchor distributions A 0 and A 1 . The FID score indicates that, as the KL divergence between A 0 and A 1 increases, RealnessGAN tends to perform better, which verifies our discussion in Sec.2.3 that a larger difference between anchor distributions imposes stronger constraints on G. To further testify, two different pairs of anchors with similar KL divergences (11.95 and 11.67) are exploited and they yield comparable FID scores (23.98 and 24.22) .

Objective of G. As mentioned in Sec.2.3, theoretically, the objective of G is max G E x∼pg [D KL (A 0 D(x))]. However, in practice, since D is not always optimal, we need either a pair of A 0 and A 1 that are drastically different, or an additional constraint to aid this objective. Fig. 6 shows that, with the ideal objective alone, when the KL divergence between A 0 and A 1 is sufficiently large, on CelebA we could obtain a generator with limited generative power. On the other hand, by applying constraints as discussed in Sec. 

5. CONCLUSION:

In this paper, we extend the view of realness in generative adversarial networks under a distributional perspective. In our proposed extension, RealnessGAN, we represent the concept of realness as a realness distribution rather than a single scalar. so that the corresponding discriminator estimates realness from multiple angles, providing more informative guidance to the generator. We prove RealnessGAN has theoretical guarantees on the optimality of the generator and the discriminator.

On both synthetic and real-world datasets, RealnessGAN also demonstrates the ability of effectively and steadily capturing the underlying data distribution. 

:



Figures and Tables:

Figure 1: The perception of realness depends on various aspects. (a) Human-perceived flawless. (b) Potentially reduced realness due to: inharmonious facial structure/components, unnatural background, abnormal style combination and texture distortion.

Figure 2: Left: real data sampled from the mixture of 9 Gaussian distributions. Right: samples generated by Std-GAN, WGAN-GP, LSGAN, HingeGAN and RealnessGAN.

Figure 3: First row: the results of RealnessGAN when fixing k G = k D = 1 and increasing the number of outcomes. Second row: the results of RealnessGAN when k G is properly increased. Bottom curves: under the settings of second row, the ratio of high quality samples and the number of recovered modes.

Figure 4: Training curves of different methods in terms of FID and SWD on both CelebA and CIFAR10, where the raise of curves in the later stage indicate mode collapse. Best viewed in color.

Figure 5: Training FID curves of Realness-GAN with and without feature re-sampling.

Figure 6: Samples generated by RealnessGAN trained with the ideal objective (equation 18). Toprow: samples when D KL (A 1 A 0 ) = 11.05. Bottom-row: samples when D KL (A 1 A 0 ) = 33.88.

Figure 7: Training curves of RealnessGAN on CelebA using objective2 (equation 19) and objective3 (equation 20).

2.4, G can learn to produce more realistic samples as demonstrated in Fig.8. Similar results are observed on CIFAR10, where RealnessGAN obtains comparable FID scores with and without constraints, as shown in Tab.3. Fig.7 also provides the training curves of RealnessGAN on CelebA using these two alternative objectives.

Figure 8: Images sampled from RealnessGAN, respectively trained on CIFAR10 (top), CelebA (middle) and FFHQ (bottom).

Minimum (min), maximum (max), mean and standard deviation (SD) of FID and SWD on CelebA and CIFAR10, calculated at 20k, 30k, ... iterations. The best indicators in baseline methods are underlined.

Table content: <table><tr><td/><td>Method</td><td/><td colspan="2">FID ↓</td><td/><td/><td colspan="2">SWD (×10 3 ) ↓</td></tr><tr><td/><td/><td>Min</td><td>Max</td><td>Mean</td><td>SD</td><td>Min</td><td>Max</td><td>Mean</td><td>SD</td></tr><tr><td/><td>Std-GAN</td><td>27.02</td><td>70.43</td><td>34.85</td><td>9.40</td><td colspan="3">14.81 68.06 30.58 15.39</td></tr><tr><td/><td>WGAN-GP</td><td colspan="3">70.28 104.60 81.15</td><td>8.27</td><td colspan="3">17.85 30.56 22.09</td><td>2.93</td></tr><tr><td>CelebA</td><td>LSGAN</td><td>30.76</td><td>57.97</td><td>34.99</td><td>5.15</td><td colspan="3">16.72 23.99 20.39</td><td>2.25</td></tr><tr><td/><td>HingeGAN</td><td>25.57</td><td>75.03</td><td colspan="2">33.89 10.61</td><td colspan="3">14.91 54.30 28.86 10.34</td></tr><tr><td/><td colspan="2">RealnessGAN 23.51</td><td>81.3</td><td>30.82</td><td>7.61</td><td colspan="3">12.72 31.39 17.11</td><td>3.59</td></tr><tr><td/><td>Std-GAN</td><td>38.56</td><td>88.68</td><td colspan="2">47.46 15.96</td><td colspan="3">28.76 57.71 37.55</td><td>7.02</td></tr><tr><td/><td>WGAN-GP</td><td>41.86</td><td>79.25</td><td>46.96</td><td>5.57</td><td colspan="3">28.17 36.04 30.98</td><td>1.78</td></tr><tr><td>CIFAR10</td><td>LSGAN</td><td>42.01</td><td>75.06</td><td>48.41</td><td>7.72</td><td colspan="3">31.99 40.46 34.75</td><td>2.34</td></tr><tr><td/><td>HingeGAN</td><td colspan="4">42.40 117.49 57.30 20.69</td><td colspan="3">32.18 61.74 41.85</td><td>7.31</td></tr><tr><td/><td colspan="5">RealnessGAN 34.59 102.98 42.30 11.84</td><td colspan="3">22.80 53.38 26.98</td><td>5.47</td></tr></table>



Table content: <table><tr><td colspan="5">: Minimum (min), maximum (max),</td><td/><td/></tr><tr><td colspan="5">mean and standard deviation (SD) of FID on</td><td/><td/></tr><tr><td colspan="5">CelebA using different anchor distributions,</td><td/><td/></tr><tr><td colspan="4">calculated at 20k, 30k, ... iterations.</td><td/><td/><td/></tr><tr><td>DKL(A1 A0)</td><td>Min</td><td>Max</td><td>Mean</td><td>SD</td><td/><td/></tr><tr><td>1.66</td><td colspan="4">31.01 96.11 40.75 11.83</td><td>0</td><td>10</td><td>20 Iterations (10k) 30</td><td>40</td><td>50</td></tr><tr><td>5.11</td><td colspan="3">26.22 87.98 36.11</td><td>9.83</td><td/><td/></tr><tr><td>7.81</td><td colspan="4">25.98 85.51 36.30 10.04</td><td/><td/></tr><tr><td>11.05</td><td colspan="3">23.51 81.30 30.82</td><td>7.61</td><td/><td/></tr></table>

FID scores of G on CIFAR10, trained with different objectives.

Table content: <table><tr><td>G Objective</td><td>FID</td></tr><tr><td colspan="2">Objective1 (equation 18) 36.73</td></tr><tr><td colspan="2">Objective2 (equation 19) 34.59</td></tr><tr><td colspan="2">Objective3 (equation 20) 36.21</td></tr><tr><td>DCGAN</td><td>38.56</td></tr><tr><td>WGAN-GP</td><td>41.86</td></tr><tr><td>LSGAN</td><td>42.01</td></tr><tr><td>HingeGAN</td><td>42.40</td></tr></table>

Peer review (iclr2020):
Post rebuttal: The authors* responses have addressed most of my concerns, and I*ve raised my rating from 3 to 6. ---------------------------------------- Summary: This paper extends the discriminator of GAN to use a distributional output (multiple scalars) instead of a single scalar. As a result, the trained GAN becomes robust to the mode collapse. Pros: - The proposed method is clearly written and well-justified (e.g., Theorem 2). - Extension of the relativistic GAN [1] to the proposed setting is interesting. - The authors demonstrate that vanilla DCGAN architecture can generate high-fidelity (1024x1024) images. Cons: 1. An ensemble of discriminators? The authors use multiple scalars to consider diverse factors of the realness. However, it is simply an ensemble of discriminators [2] in a spirit. As each discriminator focus on different factors, it is not surprising that the generator becomes robust to the mode collapse. Also, recent work on mode collapse (e.g., [3]) shows better results on the mixture of gaussian experiments even using a single discriminator. At least, the authors should compare their method with the ensemble methods and claim the advantage over them. 2. Choice of the anchor distributions. The choice of anchor distributions A_0 and A_1 are not specified. While the authors provide some partial results in Table 2, it would be worthwhile to clarify the experimental details and justify them. 3. Role of each outcome u_i? The authors claim that each outcome u_i corresponds to the different factors of realness. However, the role of learned u_i is not investigated. Also, one may enforce u_i to learn different factors by promoting diversity of them, e.g., decrease their cosine similarity [4]. Minor comments: - The word *support* [5] is misused. The support itself means the set of non-zero elements, hence the authors should use the word *outcome* (or *sample*) instead of *support*. - The notation is not consistent. For example, the authors may use *x sim p_data(x)* (specify variable) or *z sim p_z* (omit variable), but not both. - Numbering is not consistent. For example, *Tab.4.2.* should be changed to *Tab.2.* for consistency. [1] Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard GAN. ICLR 2019. [2] Durugkar et al. Generative Multi-Adversarial Networks. ICLR 2017. [3] Xiao et al. BourGAN: Generative Networks with Metric Embeddings. NeurIPS 2018. [4] Elfeki et al. GDPP: Learning Diverse Generations Using Determinantal Point Process. ICML 2019. [5] https://en.wikipedia.org/wiki/Support_(mathematics)