Title: ROBUST GENERATIVE ADVERSARIAL NETWORK

Abstract: Generative adversarial networks (GANs) are powerful generative models, but usually suffer from instability which may lead to poor generations. Most existing works try to alleviate this problem by focusing on stabilizing the training of the discriminator, which unfortunately ignores the robustness of both generator and discriminator. In this work, we consider the robustness of GANs and propose a novel robust method called robust generative adversarial network (RGAN). Particularly, we design a robust optimization framework where the generator and discriminator compete with each other in a worst-case setting within a small Wasserstein ball. The generator tries to map the worst input distribution (rather than a specific input distribution, typically a Gaussian distribution used in most GANs) to the real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We have provided theories showing that the generalization of the new robust framework can be guaranteed. A series of experiments on CIFAR-10, STL-10 and CelebA datasets indicate that our proposed robust framework can improve consistently on four baseline GAN models. We also provide ablation analysis and visualization showing the efficacy of our method on both generator and discriminator quantitatively and qualitatively.

INTRODUCTION:

Generative adversarial networks (GANs) (Goodfellow et al., 2014) have been enjoying much attention recently due to their great success on different tasks and datasets (Radford et al., 2015) (Salimans et al., 2016) (Ho & Ermon, 2016) (Li et al., 2017) (Chongxuan et al., 2017) . The framework of GANs can be formulated as the game between generator and discriminator. The generator tries to produce the fake distribution which approximates the real data distribution, while the discriminator attempts to distinguish the fake distribution from the real distribution. These two players compete with each other iteratively. GANs are also popular for their theoretical value. Training the discriminator is showed to equivalent to training a good estimator for the density ratio between the fake distribution and the real one (Nowozin et al., 2016) (Uehara et al., 2016) (Mohamed & Lakshminarayanan, 2016) .

The discriminator generally measures the departure between the model distribution and real data distribution with certain divergence measure, e.g. Jensen-Shannon divergence or fdivergence (Nowozin et al., 2016) . Arjovsky et al. proved that the supports of the fake and real distributions are typically disjoint on low dimensional manifolds and there is a nearly trivial discriminator which can correctly classify the real and fake data (Arjovsky et al., 2017) . The loss of such discriminator converges quickly to zero which causes the vanishing gradient for generator. To alleviate such problem, Arjovsky et al. proposed the Wasserstein GAN based on Wasserstein metric requiring no joint supports. Since it is inconvenient to minimize directly the Wasserstein distance, they solve the dual problem by clipping the weights to ensure the Lipschitz condition for discriminator. Later Gulrajani et al. proposed the gradient penalty to guarantee the Lipschitz condition (Gulrajani et al., 2017) . Spectral normalization is also proposed to stabilize the training of the discriminator (Miyato et al., 2018) .

Most existing methods try to improve the stability of GANs by controlling the discriminator. However, the robustness of GANs have not adequately been considered. When the discriminator is not robust to noise (i.e., the discriminator cannot measure the distance between the fake and real distribution accurately), some examples might be mis-classified which consequently misleads the training of generator. Meanwhile, the poor generalization performance of generator might cause the "blurry" generated images for some potential input noise. Robust Conditional Generative Adversarial Net-works is proposed to improve the robustness of conditional GAN for noised data. However, this method can merely implement on conditional GAN which improves the ability of generator only to defend the noise (Chrysos et al., 2018) . Some other researchers focus on the robustness of GAN to label noise (Thekumparampil et al., 2018) (Kaneko et al., 2019) .

In this paper, we attempt to improve the robustness of GANs in a systematical way by promoting the robustness of both discriminator and generator. We propose a novel robust method called robust generative adversarial network (RGAN) where the generator and discriminator still compete with each other iteratively, but in a worst-case setting. Specifically, a robust optimization is designed with considering the worst distribution within the small Wasserstein ball. The generator tries to map the worst input distribution (rather than a specific distribution) to the real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We provide some theoretical analysis for the proposed Robust GAN including generalization. We also implement our robust framework on different baseline GANs (i.e., DCGAN, WGAN-GP, and BW-GAN) (Radford et al., 2015) (Adler & Lunz, 2018) , observing substantial improvements consistently on all the datasets used in this paper.

GENERATIVE ADVERSARIAL NETWORK:

The principle of GAN is a game between two players: generator and discriminator, both of which are usually formulated as the deep neural networks. The generator tries to generate a fake example to fool discriminator, while the discriminator attempts to distinguish between fake and real images. Formally, the training procedure of GAN can be formulated as:

EQUATION

where, x and x = G(z) are real and fake examples sampled from the real data distribution P r and generation distribution P g respectively. The generation distribution is defined by G(z) where z ∼ P z (P z is a specific input noise distribution). The minmax problem cannot be solved directly since the expectation of the real and generation distribution is usually intractable. Therefore, the approximation problem is defined as:

EQUATION

where m examples of x i and z i are sampled from distributions P r and P z and the mean value of loss is used to approximate the original problem. However, such a way might not ensure a good robustness of discriminator and generator. Some noised images might not be classified correctly and potential input noise points will cause degraded generation. In this paper, for alleviating such problem, we design a distributionally robust optimization. Particularly, we consider the worst distribution (rather than a specific single distribution) within the small range.

ROBUST GENERATIVE ADVERSARIAL NETWORK:

As we discussed in the previous sections, although most existing GAN methods can stabilize the training of the discriminator, the robustness might not be adequately considered. In other words, the discriminator might not perform well on some noised data which consequently misleads the training of generator. Similarly, the generator might produce poor generations for certain input noise points if its robustnessis not good. To alleviate such problem, we design the distributionally robust optimization on GAN. Before we discuss how we can achieve this, we first elaborate the distributionally robust optimization.

DISTRIBUTIONALLY ROBUST OPTIMIZATION:

Let d : X × X → R + ∪ {∞}. The departure between x and x 0 can then be represented by d(x, x 0 ). For distributionally robust optimization, the robustness region P = {P : D(P, P 0 ) ≤ ρ} is considered, a ρ -neighborhood of the distribution P 0 under the divergence D(., .) instead of a single distribution1 . The distributionally robust optimization can be formulated as (Sinha et al., 2017) :

EQUATION

where l(.) is a loss function parameterized by θ. The problem of (3) is typically intractable for arbitrary ρ.

In order to solve this problem, we first present a proposition:

Proposition 0.1 Let l: θ × X → R and d: X × X → R + be continuous. Then, for any distribution P 0 and ρ > 0 we have

EQUATION

)

(Proof is provided in (Sinha et al., 2017) ).

With Proposition 0.1, we can reformulate (3) with the Lagrangian relaxation as follows:

EQUATION

where the second term d(x, x 0 ) is to restrict the distance between two points.

ROBUST TRAINING OVER GENERATOR:

With the distributionally robust optimization, we first discuss how we can perform robust training over generator. The generator of GAN tries to map a noise distribution P z to the image distribution P r . The objective of generator is described as follows:

min G 1 m m i=1 [log(1 -D(G(z i )))],

where

EQUATION

Typically, P z is a Gaussian distribution. For improving the robustness, we consider all the possible distributions within the robust region P z = {P : W (P, P z ) ≤ ρ z } rather than a single specific distribution (typically a Gaussain in most existing GANs). Here we use the Wasserstein metric to measure the distance between P and P z , where P is the ρ z -neighbor of the original distribution P z . However, it is difficult to consider all the distributions in this small region, the alternative way is to consider their upper bound (the worst distribution). The robust optimization problem for G is then described as follows:

min G sup P ∈Pz 1 m m i=1 [log(1 -D(G(z i )))],

where z i ∼ P

According to Proposition 0.1, we can relax (7) as:

min G max r 1 m m i=1 [log(1 -D(G(z i + r i ))) -λ z r 2 2 ],

where

z i ∼ P z (8)

Different from those previous methods, our method attempts to map the worst distribution (in the ρ z -neighborhood of the original distribution P z ) to the image distribution. Intuitively, we sample the noise points which are most likely (or the worst) to generate the blurry images and optimize the generator based on these risky points. Therefore, such generator would be robust against poor input noises and might be less likely to generate the low-quality images.

ROBUST TRAINING OVER DISCRIMINATOR:

In traditional GANs described by ( 2), the generator attempts to generate a fake distribution to approximate the real data distribution, while the discriminator tries to learn the decision boundary to separate real and fake distributions. Apparently, a discriminator with a poor robustness would inevitably mislead the training of generator. In this section, we utilize the popular adversarial learning method and propose the robust optimization method to improve the discriminator's robustness both for clean and noised data.

Specifically, we define the robust regions for both the fake distribution P g = {P : W (P, P g ) ≤ ρ g } and real distribution P r = {P : W (P, P r ) ≤ ρ r }. The generator tries to reduce the distance between the fake distribution P g and real distribution P r . The discriminator attempts to separate the worst distributions in P g and P r . Intuitively, the worst distributions are closer to decision boundary (less discriminative) and they are able to guide the training of discriminator to perform well on "confusing" data points near the classification boundary (such discriminator can be more robust than original one). We can reformulate (2) in the robust version:

EQUATION

where z i ∼ P z , x i ∼ P 1 and G ∼ P 2 . Using Proposition 0.1, we can relax the alternate problem as:

EQUATION

Here r 1 = {r i 1 } m i=1 is the set of small perturbations for the points sampled from real distribution P d which tries to make the real distribution closer to the fake distribution. r 2 = {r i 2 } m i=1 tries to make fake distribution closer to real one. Intuitively, these perturbations try to enhance the difficulty of classification task for discriminator by making real and fake data less distinguishable and it can help promote the robustness of discriminator.

OVERALL OPTIMIZATION:

We now integrate the robust training of generator and discriminator into a single framework:

min G max D V (G, D) (1 -λ)S(G, D) + sup P :W (P,Pr)≤ρr λE x∼P [logD(x)]

+ sup

EQUATION

where G (z i ) = G(z i ) + r i 2 and z i ∼ p λ z . p λ z is the mixture distribution defined by p λ z = (1 -λ)p z + λp z and p z is the worst distribution defined by p z = arg max P :W

(P,Pz)≤ρz E x∼P [1 -logD(G(x))]. r i

2 is arbitrary perturbation. It is noted that we also combine the original GAN into the framework, allowing a more flexible training. The specific algorithm is given as below:

Algorithm 1 Algorithm for RGAN.

1: for number of training iterations do 2:

Sample a batch of input noise z i ∼ P z of size m, a batch of real data x i ∼ P r of size m. λ is the trade-off parameter for original objective and our objective. 1 and 2 are amplitude of perturbation for input and images respectively.

3: find the worst perturbation {r zadv , r dadv1 , r dadv2 } by maximizing the objective of generator and minimizing the objective of discriminator: 4:

r i zadv = arg min r i : r i 2 =1 [log(1 -D(G(z i + r i ))) + λ z r i 2 2 ]

5:

r i dadv1 = arg min r i : r i 2=1 [logD(x i + r i ) + λ d r i 1 2 2 ]

6:

r i dadv2 = arg min r i : r i 2=1 [log(1 -D(G(z i ) + r i )) + λ d r i 2 2 2 ] 7:

Update G by descending along its stochastic gradient:

8: ∇ θg [ 1 m m i=1 [log(1 -D(G(z i ))) + λ m m i=1 [log(1 -D(G(z i + 1 r i zadv ))) 9:

Update D by descending along its stochastic gradient: 10:

∇ θ d [S m (G, D)+ λ m m i=1 [logD(x i + 2 r i dadv1 )]+ λ m m i=1 [log(1-D(G(z i )+ 2 r i dadv2 ))]] 11: end for

THEORETICAL ANALYSIS:

In this section, we provide theoretical analysis for the RGAN but leave the proof details in appendix. We now show that the optimal discriminator of RGAN balances the mixture of real distributions and the mixture of fake distributions as Lemma 0.2. Lemma 0.2 For arbitrary fixed G, the optimal D of the game defined by the utility function V (G, D) is:

EQUATION

where, p λ r (x) = (1 -λ)p r + λp r is the mixture distribution for real data with λ ∈ [0, 1]. p r is the worst distribution defined by p r = arg min P :W (P,Pr)≤ρr E x∼P [logD(x)]. p λ g (x) = (1λ)p g + λp g is the mixture distribution for fake data. The worst distribution p g is defined by p g = arg min P :W (P,Pg)≤ρg

E G ∼P [1 -logD(G (z))].

We further show the optimum point of the utility function V (G, D) as Lemma 0.3. Lemma 0.3 When the optimum discriminator D * is achieved, the utility function reaches the global minimum if and only if p λ g (x) = p λ r (x).

The min-max problem of ( 11) is computationally intractable due to the expectations over real and fake distributions. An alternate way is to approximate the original problem with the empirical average of finite examples:

EQUATION

where x i ∼ p r , G ∼ p g and z i ∼ p λ z . p λ z is the mixture distribution defined by p λ z = (1-λ)p z +λp z and p z is the worst distribution defined by p z = arg max P :W (P,Pz)≤ρz E x∼P [1 -logD(G(x))].

We now provide the analysis for generalization ability as Lemma 0.4. First, we give some assumptions:

Assumption 1 We provide the following assumptions for RGAN:

1. The discriminator logD θ (x) is k θ -Lipschitz in its parameter θ, i.e., |logD θ (x) -logD θ (x)| ≤ k θ θ -θ . 2. The discriminator logD θ (x) is k x -Lipschitz in its x, i.e., |logD θ (x) -logD θ (x )| ≤ k x x -x .

3. The distance between two arbitrary samples is bounded, i.e., x -x ≤ ∆ B .

The generalization ability of discriminator is defined as in (Qi, 2017) (Arora et al., 2017) and it describes if and how fast the difference

|V θ m -V θ | converges, where, V θ = max D V (G * , D) and V θ m = max D V m (G * , D).

Lemma 0.4 Under Assumption 1, with at least probability 1 -η, we have:

|V θ m -V θ | ≤ (14)

when the number of samples

EQUATION

)

where C is a sufficiently large constant, and N is the number of parameters of the discriminator function.

Similarly, the generalizability of the generator can be defined as convergence of difference

|Q φ m - Q φ |, where, Q φ = min G V (G, D * ) and Q φ m = min G V m (G, D * ).

We first give the assumptions:

Assumption 2 We provide the following assumptions for RGAN:

1. The generator G φ (z) is k φ -Lipschitz in its parameter φ, i.e., |G φ (z) -G φ (z)| ≤ k φ φ -φ . 2. The discriminator G φ (z) is k z -Lipschitz in its z, i.e., |G φ (z) -G φ (z )| ≤ k z z -z .

3. The distance between two arbitrary samples is bounded, i.e., z -z ≤ ∆ B z .

Lemma 0.5 Under Assumption 2, with at least probability 1 -η, we have:

|Q φ m -Q φ | ≤ (16)

when the number of samples

EQUATION

)

where C g is a sufficiently large constant, and N g is the number of parameters of the generator function.

EXPERIMENTS:

We present a series of experiments in this section. First, we show that our proposed RGAN can improve the performance of different kinds of baseline models including WGAN-GP, DCGAN, WGAN-GP (resnet), and BWGAN. Inception score and FID are used to evaluate the quality of generations. Following many previous relevant work, we mainly conduct on CIFAR-10 and STL-10 the comparison between our proposed method and various baseline models quantitatively while visualizing different models qualitatively on both CIFAR-10 and CelebA. In addition, we plot the bar charts for different baseline models and RGANs on two datasets (CIFAR-10 and STL-10). We also perform the ablation analysis to examine closely our proposed framework. Furthermore, we provide visualizations showing that the performance of baseline models may degrade given some specific input noises (sampled from the worst distribution). In comparison, our proposed method is more robust and can still perform fairly well. In the third part, we provide the visualizations of T-SNE embedding for original and worst distributions. Moreover, we show some images generated by baseline models and our proposed model.

QUANTITATIVE COMPARISON:

To evaluate the performance of our proposed method, we follow the previous works (Gulrajani et al., 2017; Adler & Lunz, 2018) on robustness and mainly conduct experiments on the CIFAR-10 and STL-10 dataset. There are 4 baseline models including WGAN-GP, WGAN-GP (resnet), DCGAN, and BWGAN. We implemented our proposed robust strategy on these baselines and would like to check if the robust training could indeed improve the performance. The structures and settings of our method are the same as baseline models. We train WGAN-GP, DCGAN, BWGAN and our proposed RGANs with 50, 000 training samples for 200, 000 epochs. For WGAN-GP (resnet) and our corresponding model, we found that 100, 000 epochs appear sufficient. For each 500 epochs, we calculate the inception score for 50, 000 generated images. For training RGANs, there are three hyper-parameters λ (trade off our objective and original one), 1 and 2 . We set λ = 0.1 which is searched from {0.001, 0.01, 0.1, 0.5, 1, 2}. We also set 1 = 0.01 and 2 = 4 which was searched from {0.001, 0.01, 0.1, 0.2, 0.5, 1, 2, 4, 5, 10}. For STL-10, we train our models and corresponding baselines with 80w training samples with size 48 × 48. The training settings are totally the same with settings for CIFAR-10. Note that we do not need to adjust hyper-parameters for achieving better performance on the second dataset.

We list the performance for different models in Table 1 . Clearly, our proposed RGAN (which is based on WGAN-GP-res) achieves the best result among all the methods in terms of both the criteria, i.e., Inception Score and FID. In order to check if the proposed robust strategy can indeed improve over different baselines, we also detail the performance in Figure 1 where we plot the bar charts for different baseline models and their robust version with RGAN on two datasets (CIFAR-10 and STL-10).2 It is noted that the robust strategy can consistently improve the baselines on the two datasets in terms of both the criteria. In addition, we also show the convergence curves in Figure 2 . Clearly, when our robust strategy is applied on the baseline GANs, an obvious increase of the inception scores can be observed (though the convergence speed is similar to that of baseline models). All these experiments indicate that the robust training is indeed necessary and useful. (Berthelot et al., 2017) 5.62 DCGAN (Radford et al., 2015) 5.77 ± 0.021 7.36 ± 0.06 42.18 53.23 Improved GAN (-L+HA) (Salimans et al., 2016) 6.86 ± 0.06 EGAN-Ent-VI (Dai et al., 2017) 7.07 ± 0.10 DFM (Warde-Farley & Bengio, 2016) 7.72 ± 0.13 CT GAN (Wei et al., 2018) 8.12 ± 0.12 SNGAN (Miyato et al., 2018) 8.22 ± 0.05 9.10 ± 0.04 21.70 40.1 ± 0.04 BWGAN (Adler & Lunz, 2018) 8.08 ± 0.05 25.67 WGAN-GP-res (Gulrajani et al., 2017) 7 Each subfigure shows the comparison between a different baseline model (blue curve) and its corresponding robust version (by applying the RGAN strategies, red curve). Robust GANs consistently achieve much better performance though they converge in a similar speed to baseline models.

or discriminator can consistently improve the performance of all baseline models, while a joint robust training on both generator and discriminator can further boost the performance. It is interesting to note that robust training on discriminator only could lead to more performance gain than on generator only, implying that a robust discriminator may be more important. This would be investigated as future work.

Table 2 : Ablation analysis for RGAN on different baselines on CIFAR-10. In addition, taking again CIFAR-10 as one illustrative dataset, we also show that our proposed robust method can perform robust on some potential input noise which might lead to poor generations (input noise sampled from the worst input distribution). Specifically, we generate 50, 000 images with RGAN and various baseline models from the original distribution and worst distribution for five times. Then, we compute the inception score and their corresponding standard deviation. The results are showed in Figure 3 . As observed, without the robust training, those baseline models perform consistently worst in the case of the worst noises input than that of the original input noises. This shows that the traditional GANs may not be robust and may lead to worse performance in case of certain poor input noise. In comparison, when the robust training is implemented, RGAN leads to similar performance even if the worst input noise is given.

VISUALIZATION:

We present visualization results to compare various methods qualitatively.

VISUALIZATION ON CIFAR-10:

In this subsection, we present a series of visualization trying to understand visually why the robust GAN could lead to better performance than the traditional GANs. To this end, we sample 500 data points from the original input distribution and worst input distribution respectively. We then plot the 2-dimensional T-SNE embedding of these points. We also would like to plot the real data distribution and the generated data from the traditional GAN as well as our robust GAN. For clarity, we take WGAN-GP as one example but we should bear in mind that the conclusion is basically the same for other traditional GANs like DCGAN. These plots are made in Figure 4 where one can inspect the meaning of each subfigure in the caption. We highlight some remarks as follows. First, Figure 4 (a) indicates that the worst distribution covers wider range of areas, especially low density areas of the original distribution; this might cause poor generations since the worst input noise distribution is significantly different from the original input noise. Second, (b) shows that the worst real distribution (red) actually looks much similar to the worst generation distribution. It may be more robust and meaningful to minimize in the worst-case setting the departure of the real data distribution and the fake data distribution, which is conducted in our RGAN. Third, (c) shows that the real data distribution varies largely from the generated data points obtained by traditional GANs, indicating the poor generalization of the traditional GAN; in comparison, with a robust optimization in the worst-case setting, (d) demonstrates that the generated data look very close to the real data. 

CONCLUSION:

In this paper, we consider the generalization issue of GANs and propose a robust model called robust generative adversarial network (RGAN). We have designed a robust optimization framework where the generator and discriminator compete with each other in a worst-case setting within a small Wasserstein ball. The generator tries to map the worst input distribution (rather than a specific input distribution) to real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We have provided theories showing that the generalization of the new robust framework can be guaranteed. We also have conducted extensive experiments on CIFAR-10, STL-10 and CelebA datasets with two criteria (Inception score and FID) indicating that our proposed robust framework can improve consistently on several baseline GAN models.

Ablation analysis and visualization have demonstrated the advantages of RGAN both quantitatively and qualitatively.

Proof::

Given the optimal D * , we can reformulate the function V (G, D):

V (G, D * ) = p λ r (x)log( p λ r (x) p λ r (x) + p λ g (x)

)dx + p λ g (x)log( p λ g (x) p λ r (x) + p λ g (x)

)dx Then, V (G, D * ) can be rewritten as:

EQUATION

)

where JSD is the Jensen-Shannon divergence, which is always non-negative and the unique optimum is achieved if and only if p λ r (x) = p λ g (x). Assumption 1 We provide following assumptions for RGAN:

1. The discriminator D θ (x) is k θ -Lipschitz in its parameter θ, i.e., |logD θ (x)-logD θ (x)| ≤ k θ θθ .

2. The discriminator D θ (x) is k x -Lipschitz in its x, i.e., |logD θ (x) -logD θ (x )| ≤ k x x -x .

3. The distance between two arbitrary samples is bounded, i.e., x -x ≤ ∆ B . Lemma 0.4 Under assumption 1, with at least probability 1 -η, we have:

|V θ m -V θ | ≤ (22)

when the number of samples

EQUATION

)

where C is a sufficiently large constant, and N is the number of parameters of the discriminator function, V θ = max D V (G * , D) and V θ m = max D V m (G * , D). Proof: To prove the bound, we need to apply the McDiarmid's inequality. We first bound the change of function V θ m (D, G * ) when a sample is changed. When i-th samples are replaced by x 1i , x 1i , G z1i and G z1i , the function changes to V θi m (D, G * ). Then, we have 24) 

EQUATION

:

Normally, the Wasserstein metric W (., .) is used and corresponding d(x, x0) = x -x0

p where p > 0

BWGAN appears not to converge in STL-10 in our experiments. For fair comparison, we did not report the performance when BWGAN is used as the baseline in STL-10.

Figures and Tables:

Figure 1: Performance (Inception score: the bigger the better, and FID: the lower the better) of different baselines (blue bars) and corresponding RGANs (orange bar). Our methods consistently perform better than baselines on different datasets and criteria. 2

Figure 2: Inception score versus training step.Each subfigure shows the comparison between a different baseline model (blue curve) and its corresponding robust version (by applying the RGAN strategies, red curve). Robust GANs consistently achieve much better performance though they converge in a similar speed to baseline models.

Figure 3: Inception score of baselines and RGANs on both the original input noise and the worst input noise on CIFAR-10. Performance of baselines are almost consistently degraded in the worst input noise (compared from the original input noise), while their robust versions (trained with RGAN) perform similar and stable for both worst and original input noise.

Figure 4: Visualization and T-SNE embedding on CIFAR-10. (a): Red points are the input noise points sampled from the original Gaussian distribution, and blue points are sampled from the worst distribution. The worst distribution covers a wider range of area, especially low density area of original distribution which might cause poor generation. (b): The worst real distribution (red) and worst generation distribution (blue). It can be noted that the worst data distributions are more similar to each other which is more difficult to be classified. (c): Red points are the images sampled from real distributionand blue points are generated by WGAN-GP. (d): Red points are the images sampled from real distributionand blue points are generated by RGAN. The data distribution generated by our method is apparently closer to the real distribution.

VISUALIZATION ON CELEBATo clearly examine the visual quality, we demonstrate some images generated by WGAN-GP, DC-GAN and their corresponding RGANs on the CelebA dataset. These generated images are shown in Figure5. As we can observe from these examples, the existing GANs may sometimes lead to very bad generations as circled in (a) and (c). In comparison, with the robust training under the worstcase distribution, such very bad examples can hardly be seen in RGAN. This clearly demonstrates the advantages of the proposed model.

Figure 5: Face images generated by WGAN-GP, DCGAN and corresponding RGANs. In (a), WGAN-GP generates two obviously strange faces highlighted with red circles. In (c), several repeated low quality faces are generated by DCGAN highlighted by red circles. Our method achieves better results.

Performance of different models on CIFAR-10 and STL-10

Table content: <table><tr><td>Methods</td><td colspan="2">Inception Score</td><td>FID</td><td/></tr><tr><td/><td>CIFAR-10</td><td>STL-10</td><td>CIFAR-10</td><td>STL-10</td></tr><tr><td>Real data</td><td colspan="2">11.24 ± 0.12 26.08 ± 0.26</td><td>7.8</td><td>7.9</td></tr><tr><td>Weight clipping</td><td>6.41 ± 0.11</td><td>7.57 ± 0.10</td><td>42.6</td><td>64.2</td></tr><tr><td>Layer norm</td><td>7.19 ± 0.12</td><td>7.61 ± 0.12</td><td>33.9</td><td>75.6</td></tr><tr><td>Weight norm</td><td>6.84 ± 0.07</td><td>7.16 ± 0.10</td><td>34.7</td><td>73.4</td></tr><tr><td>Orthonormal</td><td>7.40 ± 0.12</td><td>8.56 ± 0.07</td><td>29.0</td><td>46.7</td></tr><tr><td>ALI</td><td/><td/><td/><td/></tr><tr><td colspan="2">(Warde-Farley &amp; Bengio, 2016) 5.34 ± 0.05</td><td/><td/><td/></tr><tr><td>BEGAN</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
Summary The present work proposes to combine GANs with adversarial training replacing the original GAN lass with a mixture of the original GAN loss and an adversarial loss that applies an adversarial perturbation to both the input image of the discriminator, and to the input noise of the generator. The resulting algorithm is called robust GAN (RGAN). Existing results of [Goodfellow et al 2014] (characterizing optimal generators and discriminators in terms of the density of the true data) are adapted to the new loss functions and generalization bounds akin to [Arora et al 2017] are proved. Extensive experiments show a small but consistent improvement over a baseline method. Decision The authors do a thorough job at characterizing the proposed method using both theoretical analysis and wide ranging experimental studies. My main criticism of the paper in its present form is the lack of motivation for the proposed method. Why, out of the many possible ways to impose additional regularization should one use adversarial training to regularize GANs? While it is remarkable that the experimental results seem to be improving consistently, the improvement is quite small. Similarly, while theoretical results are provided, a discussion of what they mean for the performance of RGAN is sorely lacking, leaving me unconvinced that adversarial training leads to an improvement over GANs when compared with simpler methods of regularization. Therefore I vote to reject the paper in its present form. Suggestions for improvement on the experiments My main concern with the experiments is that a similar small improvement over the baseline could be achieved by tuning the hyperparameters in an alternative simpler regularization method. For instance, instead of using an adversarial perturbation, one could simply use a random perturbation applied to both the random noise and the discriminator input at testing time. The former would amount to a variance of the truncation trick [Brock et al 2019], while the latter would amount to using instance noise. These are established methods to improve GAN performance and to make a case for adversarial training of GANs one would need to show improvements compared to these simpler strategies, in my opinion. My main suggestion for the theoretical part is to make a stronger case of what (if anything) these theoretical results say about the performance of RGAN compared to the usual GAN. In particular, the generalization bound does not seem to depend on lambda, (which interpolates between the original GAN and RGAN). What is to be inferred from these results regarding the performance of RGAN? Questions to the authors (1) I assume you perform adversarial training in practice by backpropagating in image/noise space? How does this affect performance? How would the convergence plots look like if wall-clock time, or the number of model evaluations were used on the x-axis? (2) Did you try investing a similar computational budget to tune hyperparameters for simpler regularization methods as mentioned above and compare the resulting improvement? (3) Is the value (I presume, standard deviation) given after each the inception score computed for different multiple iterations of the same run or multiple runs with different initialization and random seed?