ROBUST GENERATIVE ADVERSARIAL NETWORK
Authors: Shufei Zhang
Zhuang Qian
Kaizhu Huang
Rui Zhang
Jimin Xiao
Venue:  (2020)
Keywords: Generative Adversarial Network
Robustness
Deep Learning

Abstract: Generative adversarial networks (GANs) are powerful generative models, but usually suffer from instability which may lead to  poor generations. Most existing works try to alleviate this problem by focusing on stabilizing the training of the discriminator, which unfortunately ignores the robustness of generator and discriminator. In this work, we consider the robustness of GANs and propose a novel robust method called robust generative adversarial network (RGAN). Particularly, we design a robust optimization framework where the generator and discriminator compete with each other in a worst-case setting within a small Wasserstein ball. The generator tries to map the worst input distribution (rather than a specific input distribution, typically a Gaussian distribution used in most GANs) to the real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We have provided theories showing that the generalization of the new robust framework can be guaranteed. A series of experiments on CIFAR-10, STL-10 and CelebA datasets indicate that our proposed robust framework can improve consistently on four baseline GAN models. We also provide  ablation analysis and visualization showing the efficacy of our method on both generator and discriminator  quantitatively and qualitatively.

Under review as a conference paper at ICLR 2020
ROBUST GENERATIVE ADVERSARIAL NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) are powerful generative models, but usu- ally suffer from instability which may lead to poor generations. Most existing works try to alleviate this problem by focusing on stabilizing the training of the discriminator, which unfortunately ignores the robustness of both generator and discriminator. In this work, we consider the robustness of GANs and propose a novel robust method called robust generative adversarial network (RGAN). Par- ticularly, we design robust optimization framework where the generator and dis- criminator compete with each other in worst-case setting within small Wasser- stein ball. The generator tries to map the worst inpul distribution (rather than a specific input distribution, typically Gaussian distribution used in most GANs) to the real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We have provided theories showing that the generalization of the new robust framework can be guaranteed. A series of experiments on CIFAR-10, STL-10 and CelebA datasets indicate that our proposed robust framework can improve consistently on four baseline GAN models. We also provide ablation analysis and visualization showing the efficacy of our method on both generator and discriminator quantitatively and qualitatively.
INTRODUCTION
Generative adversarial networks (GANs) Goodfellow al. 2014] have been enjoying much atten- tion recently due to their great success on different tasks and datasets (Radford et al., 2015 (Sllmaass ctt al., (H & Ermon 2016) (Li al. (Chongxunn et al. 2017). The framework of GANs can be formulated as the game between generator and discriminator. The generator tries to produce the fake distribution which approximates the real data distribution, while the discriminator attempts to distinguish the fake distribution from the real distribution. These two players compete with each other iteratively. GANs are also popular for their thcoretical value. Training the discriminator is showed to equivalent to training good estimator for the density ratio between the fake distribution and the real one (Nowozin et al. 20h6aa et al. (ohhaeed & Lakshminarayanan 2016).
The discriminator generally measures the departure between the model distribution and real data distribution with certain divergence measure, c.g. Jensen-Shannon divergence or f- divergence (Nowozin al. 2016). Arjovsky et al. proved that the supports of the fake and real distributions are typically disjoint on low dimensional manifolds and there is a nearly trivial dis- criminator which can correctly classify the real and fake data (Arjovsky et al. (007). The loss of such discriminator converges quickly to zero which causes the vanishing gradient for generator. To alleviate such problem, Arjovsky et al. proposed the Wasserstein GAN based on Wasserstein metric requiring no joint supports. Since it is inconvenient to minimize directly the Wasserstein dis- tance, they solve the dual problem by clipping the weights to ensure the Lipschitz condition for discriminator. Later Guirajani et al. proposed the gradient penalty to guarantee the Lipschitz condi- tion (Gulrajani et al. (017). Spectral normalization is also proposed to stabilize the training of the discriminator (Miyato et al. 2018).
Most existing methods try to improve the stability of GANs by controlling the discriminator, How- ever, the robustness of GANs have not adequately been considered. When the discriminator is not robust to noise (i.e., the discriminator cannot measure the distance between the fake and real distri- bution accurately), some examples might be mis- classified which consequently misleads the training of generator. Meanwhile, the poor generalization performance of generator might cause the "blury" generated images for some potential input noise. Robust Conditional Generative Adversarial Net-

---

Under review as a conference paper at ICLR 2020
works is proposed to improve the robustness of conditional GAN for noised data. However, this method can merely implement on conditional GAN which improves the ability of generator only to defend the noise (Chrysos et al.] 2018). Some other researchers focus on the robustness of GAN to label noise (Thekumparampil et al. 2018 (eri)KKanee Kaneko et al. 2019).
In this paper, attempt improve the robustness of GANs in systematical way by promoting the robustness of both discriminator and generator. We propose a novel robust method called robust generative adversarial network (RGAN) where the generator and discriminator still compete with each other iteratively, but in a worst-case setting. Specifically, robust optimization is designed with considering the worst distribution within the small Wasserstein ball. The generator tries to map the worst input distribution (rather than specific distribution) to the real data distribution, while the discriminator attempts distinguish the real and fake distribution with the worst perturbation. We provide some theoretical analysis for the proposed Robust GAN including generalization. We also implement our robust framework on different baseline GANs (i.e., DCGAN, WGAN-GP, and BW- GAN) (Radford et al. 2015 (Adler & Lunz 2018), observing substantial improvements consistently on all the datasets used in this paper.
GENERATIVE ADVERSARIAL NETWORK
The principle of GAN game between two players: generator and discriminator, both of which are usually formulated as the deep neural networks. The generator tries to generate fake example fool discriminator, while the discriminator attempts to distinguish between fake and real images. Formally, the training procedure of GAN can be formulated as:
min max S(G, D) A EgsP, [I)) + Eg-P, [(1 (oggD((()))) G
(1)
D
where, I and T = G(2) are real and fake examples sampled from the real data distribution P. and generation distribution Pa respectively. The generation distribution is defined by G(z) where z ~ P2 (P& is a specific input noise distribution). The minmax problem cannot be solved directly since the expectation of the real and generation distribution is usually intractable. Therefore, the approximation problem is defined as:
min max Sm (G, D) G
Sm,G., D
A
(x:)( + E (ogg=)))))(()
(2)
where examples of and 2; are sampled from distributions P. and P. and the mean value of loss used to approximate the original problem. However, such a way might not ensure a good robustness of discriminator and generator. Some noised images might not be classified correctly and potential input noise points will cause degraded generation. In this paper, for alleviating such prob- lem, design distributionally robust optimization. Particularly, we consider the worst distribution (rather than a specific single distribution) within the small range.
ROBUST GENERATIVE ADVERSARIAL NETWORK
As we discussed the previous sections, although most existing GAN methods can stabilize the training of the discriminator, the robustness might not be adequately considered. In other words, the discriminator might not perform well on some noised data which consequently misleads the training of generator. Similarly, the generator might produce poor generations for certain input noise points its robustnessis not good. To alleviate such problem, we design the distributionally robust opti- mization on GAN. Before discuss how we can achieve this, first elaborate the distributionally robust optimization.
DISTRIBUTIONALLY ROBUST OPTIMIZATION
Let d: X x X R4 (oo). The departure between and ze can then be represented by d(æ, xr)- For distributionally robust optimization, the robustness region P = {P : D(P, Pb) & p} is con- sidered, P -nighhoorhood of the distribution Po under the divergence D(.,-) instead of single
2

---

Under review as a conference paper at ICLR 2020
distributior |
The distributionally robust optimization can be formulated (Sinha et al. 2017
min sup Ep
(3)
0 PEP
where I(.) is loss function parameterized by The problem of is typically intractable for arbitrary
In order to solve this problem, we first present proposition:
Proposition 0.1 Let x X R and d: X' x X - R+ be continuous. Then, for any distribution
Po and > 0 we have
sup (Ep[[(X; 0)] (o) Ep [sup 0) (d(xx PEP
(4)
(Proof is provided in (Snha et al. 2017))
With Proposition 0.1 we can reformulate with the Lagrangian relaxation as follows:
min Ep. sup [up]((<;0))) 0) Ad(2, xo)]
(5)
0 me
where the second term d(x, mo) is to restrict the distance between two points.
ROBUST TRAINING OVER GENERATOR
With the distributionallyy robust optimization, we first discuss how we can perform robust training over generator. The generator of GAN tries to map noise distribution Pa to the image distribution P.. The objective of generator described as follows:
1
Σ [log(1 - D(G())))) where 21 ~ Pa
min G 171
(6)
1=1
Typically, Ps Gaussian distribution. For improving the robustness, consider all the possible distributions within the robust region P {P W (P. P.) S p=} rather than a single specific distribution (typically Gaussain in most existing GANs). Here we use the Wasserstein metric to measure the distance between P and P., where P is the p--neighbor of the original distribution P.. However, difficult to consider all the distributions this small region, the alternative way is to consider their upper bound (the worst distribution). The robust optimization problem for G is then described as follows:
min sup PEP,
>
:log(1 D(G(z)))) where 2;
According to Proposition 0.1 we can relax
as:
min max
llo(1 - -
D(G +())))))())))))) A₂ [r1)), where 24 P=
Different from those previous methods, our method attempts to map the worst distribution (in the p--neighborhood of the original distribution Pa) the image distribution. Intuitively, sample the noise points which are most likely (or the worst) to generate the blurry images and optimize the generator based on these risky points. Therefore, such generator would be robust against poor input noises and might be less likely to generate the low-quality images.
ROBUST TRAINING OVER DISCRIMINATOR
In traditional GANs described by (2. the generator attempts to generate a fake distribution to ap- proximate the real data distribution, while the discriminator tries to learn the decision boundary to separate real and fake distributions. Apparently, a discriminator with poor robustness would in- evitably mislead the training of generator. In this section, utilize the popular adversarial learning
Normalll, the Wasserstein metric W(--) is used and corresponding d(z, = - -roll where p

---

Under review as a conference paper at ICLR 2020
method and propose the robust optimization method to improve the discriminator's robustness both for clean and noised data.
Specifically, define the robust regions for both the fake distribution P. {P: W(P.P.) pg) and real distribution IP, = {P' W.P.P.) & or The generator tries to reduce the distance between the fake distribution Pg and real distribution P.. The discriminator attempts to separate the worst distributions in P, and P,. Intuitively, the worst distributions are closer to decision boundary (less discriminative) and they are able guide the training of discriminator perform well on "confusing" data points near the classification boundary (such discriminator can be more robust than original one). We can reformulate (2)
in the robust version:
m
m E D(
> [log
max sup D PIEF, TTL
sup PLE
m
where ~ Pz, a1 Pi and G' Ps. Using Proposition 0.1 as:
we can relax the alternate problem
1
Σ [logD(x; + +
Σ loo D(G(2) +2)) -
Ad
m
1 + 1r
max min
(10)
1=1
with 2 Ps, I; Pr
Here ri = (ri) is the set of small perturbations for the points sampled from real distribution Pa which tries to make the real distribution closer to the fake distribution. 72 = {rr) tries to make fake distribution closer to real one. Intuitively, these perturbations try to enhance the difficulty of classification task for discriminator by making real and fake data less distinguishable and it can help promote the robustness of discriminator.
OVERALL OPTIMIZATION
We now integrate the robust training of generator and discriminator into single framework:
min maxV
D) & (1- D)) sup (lggploo
G
D
P.W(P.P.).p
(11)
+
sup P.W(P.P))Ppp
NEG PLo--((L-lggD(((( (ogD(G
where G²xz)) = G(2,) + and 24 p.. p. is the mixture distribution defined by p. (1 A)p=+ + Xp'_ and p is the worst distribution defined by Çl= arg maxp: ri is arbitrary perturbation. It is noted that we also combine the original GAN into the framework, allowing more flexible training. The specific algorithm is given as below:
Algorithm Algorithm for RGAN.
1: for number of training iterations do
2: Sample batch of input noise 2, P. of size m, batch of real data x; Pr of size
À is the trade-off parameter for original objective and our objective. 61 and é2 are amplitude of perturbation for input and images respectively.
3: find the worst perturbation {rzades ddadel, (dado2) } by maximizing the objective of generator
and minimizing the objective of discriminator:
4:
rindo = arg -unn,: [log(1 D(G + r')) + A
5:
Tdado1 = arg [= + Aa Arirri
6:
rdade2 arg min,;:]]|||=
D r")) + Aellri
7:
Update G by descending along its stochastic gradient:
8: Va, [ Σ i= [ [ D(G(2))) AM 1 [log(L1 D(G
+ t +(rr)d)))))
9: Update D by descending along its stochastic gradient:
10: Vo,, [Smm (G, D)+ 1 - Dlog E2t à (og(( tor (edo2)))
end for

---

Under review as a conference paper at ICLR 2020
THEORETICAL ANALYSIS
In this section, we provide theoretical analysis for the RGAN but leave the proof details in appendix. We now show that the optimal discriminator of RGAN balances the mixture of real distributions and the mixture of fake distributions as Lemma 0.2
Lemma 0.2 For arbitrary fixed G, the optimal D of is:
the the game defined by the utility function V(G, D)
(x) 1x(x) + Ds
Dz(x)=
(12)
where, = (= (1- + Xp', is the mixture distribution for real data with À E 00, 1]. p/+
the worst distribution defined by pl arg minp:w
p2 Ee-pplogD(x>))
A)ps + Xply is the mixture distribution for fake data. The worst distribution ply is defined by p'g arg mi BG'=P[1 logD(G
We further show the optimum point of the utility function V(G, D) as Lemma 0.3
Lemma 0.3 When the optimum discriminator D is achieved, the utility function reaches the global minimum if and only if D2 (x) (2))
The man-max problem of 111 is computationally intractable due to the expectations over real and fake distributions. An alternate way to approximate the original problem with the empirical aver- age of finite examples:
min maxV (oaavVnnG,in D) A) +
log
G
D
m
(13)
m
Dog
m
where Ii. ~ p,, G' ~ and 2, and p', is the worst distribution defined by p'_ arg max.:w(..n.SSp. Ex-p -
p.. p. is the mixture distribution defined by p. = (1- +Xp
We now provide the analysis for generalization ability Lemma 0.4 First, we give some assump- tions:
Assumption
We provide the following assumptions for RGAN:
1. The discriminator log Do(()( is ke-Liphchitz in its parameter 0, ie., (logDa(x) - (z) s
ke | - 0'
The discriminator log Do (x) is Lipschit in x, ie., (oog - log Do(a') S kalla x'|l.
The distance between two arbitrary samples bounded, i.e., 1| 2'1l S Ag.
The generalization ability of discriminator is defined in Qi 2017 Arora al. 2017 and it describes and how fast the difference VV= - v converges, where, V pax V, (G" D).
maxp VG" D) and
Lemma 0.4 Under Assumption with at least probability 1 T, we have:
IV
v0 SE
(14)
when the number of samples
(k
(Nlog Ko +
1oggt)
(15)
where is a sufficiently large constant, and is the number of parameters of the discriminator function.
5

---

Under review as a conference paper at ICLR 2020
Similarly, the generalizability of the generator be defined convergence of difference !Q Q%, where, Q = ming V(G,D" and Q = ming Vn (n, D*). We first give the assumptions:
Assumption We provide the following assumptions for RGAN:
The generator Gz(z) is kgg -ippchiht in its parameter 6, i.e., (G - G2 = k 0'1l.
The discriminator Gz(z) is ---iisschitz in its z, i.e., (G(()) Gz(z')l S kz 2'11.
The distance between two arbitrary samples bounded, i.e., ||| - z/11 S B=
Lemma 0.5 Under Assumption 2, with at least probability 1 17, we have:
10th 5e
(16)
when the number of samples
C c R2R2 (N,log NokiN + +log
m
(17)
where Cg is sufficiently large constant, and Ng is the number of parameters of the generator function.
EXPERIMENTS
We present a series of experiments in this section. First, we show that our proposed RGAN can improve the performance of different kinds of baseline models including WGAN-GP, DCGAN, WGAN-GP (resnet), and BWGAN. Inception score and FID are used to evaluate the quality of generations. Following many previous relevant work, we mainly conduct on CIFAR-10 and STL- 10 the comparison between our proposed method and various bascline models quantitatively while visualizing different models qualitatively both CIFAR-10 and CclebA. In addition, plot the bar charts for different baseline models and RGANS on two datasets (CIFAR-10 10 and STL-10). We also perform the ablation analysis examine closely proposed framework. Furthermore, provide visualizations showing that the performance baseline models may degrade given some specific input noises (sampled from the worst distribution). In comparison, our proposed method is more robust and can still perform fairly well. In the third part, we provide the visualizations of T-SNE embedding for original and worst distributions. Moreover, we show some images generated by baseline models and our proposed model.
QUANTITATIVE COMPARISON
To evaluate the performance of our proposed method, we follow the previous works (Gulrajani et al. 2017 Adler & Lunz 2018) robustness and mainly conduct experiments the CIFAR- 10 and STL- 10 dataset. There are 4 baseline models including WGAN-GP, WGAN-GP (resnet), DCGAN, and BWGAN. We implemented our proposed robust strategy these baselines and would like to check if the robust training could indeed improve the performance. The structures and settings of our method are the same as baseline models. We train WGAN-GP, DCGAN, BWGAN and our proposed RGANs with 50, 000 training samples for 200, 000 epochs. For WGAN-GP (resnet) and our corresponding model, we found that 100, 000 epochs appear sufficient. For each 500 epochs, we calculate the inception score for 50, 000 generated images. For training RGANs, there are three hyper-parameters X (trade off our objective and original one), 6j and E2. We set = 0.1 which is which was searched
searched from (0.001, 0.01, 011, 0.5, 1,2], 2]. We also set 1,2, 4,5, 10). For S0L-10, we train our models and corresponding baselines with 80w training samples with size 48 48. The training settings totally the same with settings for CIFAR-10, Note that we do not need to adjust hyper-parameters for achieving better performance on the second dataset.
0.01 and
from (0.001, 0.01, 011, 0.1, 0.5, 0,
We list the performance for different models in Table Clearly, our proposed RGAN (which is based on WGAN-GP-res achieves the best result among all the methods in terms of both the criteria, i.e., Inception Score and FID. In order to check if the proposed robust strategy can indeed improve over different baselines, also detail the performance in Figure T where we plot the bar charts for

---

Under review as a conference paper at ICLR 2020
Tammncee or Shoal ver a0
Bock
(a) Inception Score
(b) FID on CIFAR-I0
(c) Inception score
(d) FID on STL-10
CIFAR-I0
STL-10
Figure Performance (Inception score: the bigger the better, and FID: the lower the better) of (blue bars) and corresponding RGANs (orange bar). Our methods consistently perform better than baselines on different datasets and criteria.
different baselines
different baseline models and their robust version with RGAN two datasets (CIFAR-10 and STL- 10) is noted that the robust strategy can consistently improve the baselines on the two datasets terms of both the criteria. In addition, we also show the convergence curves Figure 2 Clearly, when our robust strategy is applied on the baseline GANs, an obvious increase of the inception scores can be observed (though the convergence speed is similar to that of baseline models). All these experiments indicate that the robust training is indeed necessary and useful.
Table 1: Performance of different models CIFAR-10 10 and STL-10
Inception Score
FID
Methods
CIFAR-10
STL-10
CIFAR-10
STL-10
Real data
11.24 0.12
26.08 + 0.26
7.8
7.9
Weight clipping Layer norm Weight norm Orthonormal ALI Warde-Farley & Bengio
6.41 + 0.11
7.57 0.10
42.6
64.2
7.19 =19=00 0.12
7.61 + 0.12
33.9
75.6
6.84 + 0.07
7.16 + 0.10
34.7
73.4
7.40 0.12
8.56 + 0.07
29.0
46.7
2016)
1.05
BEGAN (Berthelot et al.
2017
5.62
DCGAN (Radford et al.
2015)
5.77 = 0.021
7.36 0.06
42.18
53.23
Improved GAN
HA)
Salimans et al.
2016
6.86 0.06
(Da et al.
2017
7.07 0.10
DFM
WWarde-Farles & Bengio
2016)
7.12
CAGANN (Wei et al. 2018)
8.12 0.12
SNGAN
(Myytoo al..
2018)
8.22 0.05
9.10 + 0.04
21.70
40.1 + 0.04
BWGAN
Adler & Lunz
2018
8.08 + 0.05
25.67
GPAN-GP-ree
Gulrajani et al.
2017
7.76
9.06 + 0.03
22.19
42.60
RGAN (WGAN-GP-res)
8.25 - 0.013
9.16 - 0.015
19.79
39.62
ABLATION ANALYSIS
We conduct the ablation analysis in this subsection. Specifically, we experiment on CIFAR-10 10 with robust training over generator only, robust training over discriminator only, and robust training both the generator and discriminator, trying to see if robust training is necessary on both generator and discriminator. The results listed in Table 2
As observed, robust training either generator
2BWGAN appears not to converge in STL-10 in experiments. For fair comparison, did not report the performance when BWGAN used the baseline in STT-110.

---

Under review as a conference paper at ICLR 2020
2000
(a) WGAN-GP RGAN (b) WGAN-GP (res) (c) DCGAN RGAN RGAN
(d) BWGAN VS RGAN
Figure 2: Inception score versus training step. Each subfigure shows the comparison between a dif- ferent baseline model (blue curve) and its corresponding robust version (by applying the RGAN strategies, red curve). Robust GANs consistently achieve much better performance though they con- verge in similar speed to baseline models.
or discriminator can consistently improve the performance of baseline models, while a joint ro- bust training on both generator and discriminator can further boost the performance. It is interesting to note that robust training on discriminator only could lead to more performance gain than on gen- erator only, implying that robust discriminator may be more important. This would be investigated as future work.
Table 2: Ablation analysis for RGAN on different baselines on CIFAR-10.
WGAN-GP
WGAN-GP (res)
DCGAN
BWGAN
Baseline (without robust training)
5.77 1 0.021
7.76
5.70 + 0.045
8.08
Robust training generator only
5.89 + 0.020
7.86
5.80 + 0.022
8.11
Robust training discriminator only
5.87 0.025
8.01
6.02 + 0.019
8.23
Robust training on generator discriminator
5.91 0.018
8.25
6.11 + 0.017
8.40
(a) WGAN-GP-res
(b) DCGAN
(c) WGAN-GP
(d) RGAN(WGAN-GPPres)
(e) RGAN(DCGAN)
(1) RGAN(WGAN-GP)
Figure 3: Inception score of baselines and RGANs on both the original input noise and the worst input noise on CIFAR-10. Performance of baselines are almost consistently degraded in the worst in- put noise (compared from the original input noise), while their robust versions (trained with RGAN) perform similar and stable for both worst and original input noise.
In addition, taking again CIFAR-10 one illustrative dataset, we also show that our proposed robust method can perform robust on some potential input noise which might lead to poor generations (input noise sampled from the worst input distribution). Specifically, with RGAN and various baseline models from the original distribution and worst distribution for five times. Then, compute the inception score and their corresponding standard deviation. The
generate 50, 000 images
8

---

Under review as a conference paper at ICLR 2020
(a) Input noise
(b) Worst distribution
(c) Data by WGAN-GP
(d) Data by RGAN
Figure 4: Visualization and T-SNE embedding on CIFAR-10. 10. (a): Red points are the imput noise points sampled from the original Gaussian distribution, and blue points are sampled from the worst distribution. The worst distribution covers a wider range of area, especially low density area of original distribution which might cause poor generation. (b): The worst real distribution (red) and worst generation distribution (blue). It can noted that the worst data distributions are more similar to each other which is more difficult to be classified. (c): Red points the images sampled from real distributionand blue points are generated by WGAN-GP. (d): Red points are the images sampled from real distributionand blue points generated by RGAN. The data distribution generated by our method is apparently closer to the real distribution.
results are showed in Figure As observed, without the robust training. those baseline models perform consistently worst in the case of the worst noises input than that of the original input noises. This shows that the traditional GANs may not be robust and may lead to worse performance in case of certain poor input noise. In comparison, when the robust training is implemented, RGAN leads to similar performance if the worst input noise is given.
VISUALIZATION
We present visualization results to compare various methods qualitatively.
VISUALIZATION ON CIFAR-10
In this subsection, we present series of visualization trying to understand visually why the robust GAN could lead to better performance than the traditional GANs. To this end, we sample 500 data points from the original input distribution and worst input distribution respectively. We then plot the 2-dimensional T-SNE embedding of these points. We also would like to plot the real data distribution and the generated data from the traditional GAN as well our robust GAN. For clarity, take WGAN-GP as one example but we should bear in mind that the conclusion basically the same for other traditional GANs like DCGAN. These plots made in Figure 4 where one can inspect the meaning of each subfigure in the caption. We highlight some remarks as follows. First, Figure 3a) indicates that the worst distribution covers wider range of areas, especially low density areas of the original distribution; this might cause poor generations since the worst input noise distribution is significantly different from the original input noise. Second, (b) shows that the worst real distribution (red) actually looks much similar to the worst generation distribution. It may be more robust and meaningful minimize in the worst-case setting the departure of the real data distribution and the fake data distribution, which is conducted in our RGAN. Third, (c) shows that the real data distribution varies largely from the generated data points obtained by traditional GANs, indicating the poor generalization of the traditional GAN; in comparison, with robust optimization in the worst-case setting, (d) demonstrates that the generated data look very close to the real data.
VISUALIZATION ON CELEBA
To clearly examine the visual quality, we demonstrate some images generated by WGAN-GP, DC- GAN and their corresponding RGANs on the CelebA dataset. These generated images are shown in Figure 5 As we can observe from these examples, the existing GANs may sometimes lead to very bad generations
circled in (a) and (c). In comparison, with the robust training under the worst- case distribution, such very bad examples can hardly be seen in RGAN. This clearly demonstrates the advantages of the proposed model.

---

Under review as a conference paper at ICLR 2020
(a) WGAN-GP
(b) RGAN (WGAN-GP)
(c) DCGAN.
(d) RGAN (DCGAN)
Figure Face images generated by WGAN-GP, DCGAN and corresponding RGANs. In (a), WGAN-GP generates two obviously strange faces highlighted with red circles. In (c), several peated low quality faces are generated by DCGAN highlighted by red circles. Our method achieves better results.
CONCLUSION
In this paper, we consider the generalization issue of GANs and propose a robust model called robust generative adversarial network (RGAN). We have designed a robust optimization framework where the generator and discriminator compete with each other in worst-case setting within a small Wasserstein ball. The generator tries to map the worst input distribution (rather than specific input distribution) to real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We have provided theories showing that the generalization of the new robust framework can be guaranteed. We also have conducted extensive experiments on CIFAR- 10, STL-10 and CelebA datasets with two criteria (Inception score and FID) indicating that proposed robust framework improve consistently several baseline GAN models. Ablation analysis and visualization have demonstrated the advantages of RGAN both quantitatively and qualitatively.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jonas Adler and Sebastian Lunz. Banach wasserstein gan. In Advances in Neural Information Processing Systems, pp. 6754-6763, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-1422 224-2223,20 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 224-232. JMLR. org, 2017.
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative ad- versarial networks. arXiv preprint MoXi::7033110111 2017.
LI Chongxuan, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In Advances in neural information processing systems, pp.
2017.
Grigorios G Chrysos, Jean Kossaifi, and Stefanos Zafeiriou. Robust conditional generative adver- sarial networks. arXiv preprint 1805.00800500000 2018.
Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard Hovy, and Aaron Courville. Calibrating energy- generative adversarial networks. arXiv preprint arXiv:1002.01691, 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances Neural Information Processing Systems, pp- 5767-5777, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565-4573, 2016.
Takuhiro Kaneko, Yoshitaka Ushiku, and Tatsuya Harada. Label-noise robust generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2467-2476, 2019.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. arXiv preprint S.xiv:1700.06544/ 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.0590 2018.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv
preprint 11X.v:1610.03483, 2016.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, pp. 271-279, 2016.
Guo-Jun Qi. Loss-sensitive generative adversarial networks on lipschitz densities. arXiv preprint PrXiv:1701.0622 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1eee:://11...444 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2243 2016.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv: 10571, 2017.

---

Under review as a conference paper at ICLR 2020
Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. In Advances in Neural Information Processing Systems, pp. 10271 10221-110 102822 2018.
Masatoshi Uehara, Issei Sato, Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Generative adversarial nets from a density ratio estimation perspective. arXiv preprint arXiv:1610.02920, 2016.
David Warde-Farley and Yoshua Bengio. Improving generative adversarial networks with denoising feature matching. 2016.
Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of wasserstein gans: A consistency term and its dual effect. arXiv preprint arXiv: arXiv: 1803.0154,, 2018.
APPENDIX
A. PROOF
Lemma 0.2 For arbitrary fixed G. the optimal of the game defined by the utility function V D) is:
2
Dz(x)=
(18)
1x + pg
where, (x) = (1- + Xp' is the mixture distribution for real data with À E 00 1]. p/r
the worst distribution defined by pl = arg minp. Eg- Be=p((ogD(z)) (x) = (1
A)pg + Xp'g is the mixture distribution for fake data. The worst distribution v'g is defined by D'g = arg mP.): BG'P[11 [cy-)]]-loD((((t)}
Proof:
Given the classifier and generator, the utility function can be rewritten as
V(G, D) & (1 1log
+ +EGmP, - (ogg
+ sup
AE XB
P.W(P.P))ppp
+ sup
XEo-p[(1- ((gge((())
= -1 / pr(o))og(((z)dd
+ (11 A) (o) D(z))dr
(19)
+ > / pl. (()))))
+ > / ((())oo((1D(xx)))
(()))o
+ (x))oo D(x))dx (D
where (²,2,) G(z:) + rá and P.. r is arbitrary perturbation. Then, it is easy prove that the optimal D is (z)
(m
Lemma 0.3 When the optimum discriminator D* is achieved, the utility function reaches the global minimum if and only if (x) (x).
12

---

Under review as a conference paper at ICLR 2020
Proof:
Given the optimal D", we can reformulate the function V(G,D):
/ pt (z))lo))
(x( + pS (x) )x / pr(x)log((
(x)
(x) (1) pz) ) dr
V(G, D²)
/ (1))oog
(x)
(z) + PS (1:) da
(20)
+ / ( (p)(x) + pt (z))
P/ (x)
dd - 2log2
-2oog + (x) (z) + (z)) KUG))ellloO())+
Then, V(G, D*) can be rewritten as:
+ V(G,D") + 2.
(21)
where JSD is the achieved and only if p² (x) = (x))
divergence, which is always non-negative and the unique optimum
Assumption We provide following assumptions for RGAN:
The discriminator Do(x) kg- Bippiischitz in its parameter i.e., [logDy (ooggg(())) s koll0 0'|l.
2. The discriminator Do(x)) is Lipsschcii in its x, i.e., [logD((z)) loogoooo')))) s ke 1| æ'1l
The distance between two arbitrary samples bounded, i.e., || - I'll S A B.
Lemma 0.4 Under assumption 1, with at least probability - we have:
Ie
SE
(22)
when the number of samples
(kz)
2 22
(Nlog No
+ 1og²)
(23)
where C' is sufficiently large constant, and N is the number of parameters of the discriminator function, V maxp (G"', D) and Va maxp Vnn(G", D).
Proof:
To prove the bound, we need apply the McDiarmid's inequality. We first bound the change of function V (D, G*) when sample is changed. When i-th samples are replaced by Tis. Gz and G GZ the function changes V (D, G*). Then, we have
(D.G*) r0i V (D.G*)
[(1 - A)(og) (ogg(G
AllogD(a() (og
(x))) DGg1.)
Allog((z's) D
(24)
< > kx k -1ill + X
kr (G. Gzall
>
kellar - -x'ell + X ke ke - -Gan
< kg AB
m
13

---

Under review as a conference paper at ICLR 2020
Now apply the McDiarmid's inequality. We have
-VVA,D,G)-VV,,)""12222 c/2)
62m 812
(25)
22cxp(-
The above bound applies a single discriminator Dg. To get the union bound, consider a e/8ki-
net N, i.e. for any Da. there is a 0' E N so that |00 - 0'l < </8ka. This standard net can be constructed contain finite discriminators such that N S ((alog(kaN/e)) parameters of discriminator (we here assume the parameter space of the loss function is bounded, then we can construct such net containing finite points). Therefore, for all 0 N,, we have
is the number of
V - V < s/2
(26)
C (Nlog ANW + lo
when
We further consider the bound beyond and easily obtain the bounds with the first assump- tion:
-" - 2kg 0 0'll
(27)
and
v (D, -
V (D, G < 2k 100 0'll
(28)
The final bound for all discriminator can be obtainged with assumption ||0 - 811 S c/ s/8kg:
(=(
I - (D.
(29)
+ (D,G" - -V'
-D,G")-V'(()
Assumption 2 We provide the following assumptions for RGAN:
1. The generator Go(z) is k Lipschizz in its parameter é. i.e., (Go(2) G)((2) < kollo 0'1l
2. The discriminator Go(z) is ---Lisschitz in its z, i.e., [Go(z) Gz(2')l < kz z'|l.
3. The distance between two arbitrary samples is bounded, ie.., || z'1l < A
Lemma 0.5 Under assumption 2, with at least probability 77, we have:
10t Q.1se
(30)
when the number of samples
m C₂ B. k2k2 (Nylog kokioNNg + 1og-) 17
(31)
where Cg is sufficiently large constant, and Ng is the number of parameters of the generator function, Q = ming V(G.D) and Q = ming Vn, (G. D*).
Proof:
Proof is skipped due to its similarity to Lemma 0.4.
14


Reviews:
[{"review_id": "rkxvKVc0YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe present work proposes to combine GANs with adversarial training replacing the original GAN lass with a mixture of the original GAN loss and an adversarial loss that applies an adversarial perturbation to both the input image of the discriminator, and to the input noise of the generator. The resulting algorithm is called robust GAN (RGAN). Existing results of [Goodfellow et al 2014] (characterizing optimal generators and discriminators in terms of the density of the true data) are adapted to the new loss functions and generalization bounds akin to [Arora et al 2017] are proved. Extensive experiments show a small but consistent improvement over a baseline method.\n\nDecision\nThe authors do a thorough job at characterizing the proposed method using both theoretical analysis and wide ranging experimental studies. My main criticism of the paper in its present form is the lack of motivation for the proposed method. Why, out of the many possible ways to impose additional regularization should one use adversarial training to regularize GANs? While it is remarkable that the experimental results seem to be improving consistently, the improvement is quite small. Similarly, while theoretical results are provided, a discussion of what they mean for the performance of RGAN is sorely lacking, leaving me unconvinced that adversarial training leads to an improvement over GANs when compared with simpler methods of regularization. Therefore I vote to reject the paper in its present form.\n\nSuggestions for improvement on the experiments\nMy main concern with the experiments is that a similar small improvement over the baseline could be achieved by tuning the hyperparameters in an alternative simpler regularization method. For instance, instead of using an adversarial perturbation, one could simply use a random perturbation applied to both the random noise and the discriminator input at testing time. The former would amount to a variance of the truncation trick [Brock et al 2019], while the latter would amount to using instance noise. These are established methods to improve GAN performance and to make a case for adversarial training of GANs one would need to show improvements compared to these simpler strategies, in my opinion.\n\nMy main suggestion for the theoretical part is to make a stronger case of what (if anything) these theoretical results say about the performance of RGAN compared to the usual GAN. In particular, the generalization bound does not seem to depend on lambda, (which interpolates between the original GAN and RGAN). What is to be inferred from these results regarding the performance of RGAN?\n\nQuestions to the authors\n(1) I assume you perform adversarial training in practice by backpropagating in image/noise space? How does this affect performance? How would the convergence plots look like if wall-clock time, or the number of model evaluations were used on the x-axis?\n\n(2) Did you try investing a similar computational budget to tune hyperparameters for simpler regularization methods as mentioned above and compare the resulting improvement?\n\n(3) Is the value (I presume, standard deviation) given after each the inception score computed for different multiple iterations of the same run or multiple runs with different initialization and random seed?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkeJlDz6YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed another way to improve GANs. The method tackled the robotness issue by requiring the generator and discriminator to compete with each other in a worst-case setting. The experiments on three datasets show some improvement. \n\nThe idea is interesting by forcing both G/D to learn the mapping in the worst case. However, the theory analysis to show whether the generalization is better than the original WGAN is not clear to me. The clipping or gradient penalty trick is still needed.  In addition, how this framework can work with other techniques (e.g., better architectures, spectral normalization) orthogonally is unclear. \n\nThe experimental results are not strong. First, the improvements are somehow marginal (no gain if compared with SN-GANs). Only three small benchmarks are included. It would be good to see how it works on large datasets. In the meanwhile,  the ablation study to investigate the effect over WGAN-gp is not obvious. Finally, I could not get any insight from the visualization analysis. It is not reasonable to only list several failed cases in un-conditioning setting and do the comparison. \n\nOverall, I think the idea to improve GANs is interesting. I made my recommendation mainly considering the experimental results and the insight analysis. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Syx7zrDiYH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Developing stable GAN training method has gained much attention these years.  This paper propose to tackle this issue via involving distributionally robust optimization into GAN training. Its main contribution is to combine Sinha et al with GAN, proposing a new GAN training method on the basis of vanilla GAN. Relative theory results are proved and detailed experiments are conducted. \n\nSome comments:\n1 The proposition 0.1 is not quite clear.  In fact it is correct only when the distribution discrepancy is Wasserstein. This paper reads “here we use the wasserstein metric” in “robust training over generator” subsection, the reviewer is not sure if the authors are aware of this point.\n\n2. There seems to be a lack of novelty except combining Sinha et al’s theoretical result with GAN training objective. And there seems not much explanation about the reasons behind this combination. \n\n3. The proof in this paper shares similar analysis with that in vanilla GAN paper so theoretically there is also not much novelty. There seems not much insight can one get from the theory results.\n\nOverall, the proposed method is evaluated under elaborate and detailed experiments and enjoys promising results, but lacks novelty and theoretical contribution.  Therefore, the reviewer tends to reject this paper.\n\n### reference\nAman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571, 2017.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a robust variant of GAN, in which the generator and discriminator compete with each other in a worst-case setting within a small Wasserstein ball. Unfortunately, the reviewers have raised some critical concerns in terms of theoretical analysis and empirical support. The authors did not submit rebuttals in time. We encourage the authors to improve the work based on reviewer's comments. 
Decision: Reject