Year: 2019

Abstract:
Generative adversarial networks (GANs) are powerful generative models, but usually suffer from instability which may lead to poor generations. Most existing works try to alleviate this problem by focusing on stabilizing the training of the discriminator, which unfortunately ignores the robustness of both generator and discriminator. In this work, we consider the robustness of GANs and propose a novel robust method called robust generative adversarial network (RGAN). Particularly, we design a robust optimization framework where the generator and discriminator compete with each other in a worst-case setting within a small Wasserstein ball. The generator tries to map the worst input distribution (rather than a specific input distribution, typically a Gaussian distribution used in most GANs) to the real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We have provided theories showing that the generalization of the new robust framework can be guaranteed. A series of experiments on CIFAR-10, STL-10 and CelebA datasets indicate that our proposed robust framework can improve consistently on four baseline GAN models. We also provide ablation analysis and visualization showing the efficacy of our method on both generator and discriminator quantitatively and qualitatively.

heading: INTRODUCTION
text: Generative adversarial networks (GANs) (Goodfellow et al., 2014) have been enjoying much attention recently due to their great success on different tasks and datasets (Radford et al., 2015)(Salimans et al., 2016)(Ho & Ermon, 2016) (Li et al., 2017)(Chongxuan et al., 2017). The framework of GANs can be formulated as the game between generator and discriminator. The generator tries to produce the fake distribution which approximates the real data distribution, while the discriminator attempts to distinguish the fake distribution from the real distribution. These two players compete with each other iteratively. GANs are also popular for their theoretical value. Training the discriminator is showed to equivalent to training a good estimator for the density ratio between the fake distribution and the real one (Nowozin et al., 2016)(Uehara et al., 2016)(Mohamed & Lakshminarayanan, 2016).
The discriminator generally measures the departure between the model distribution and real data distribution with certain divergence measure, e.g. Jensen-Shannon divergence or f - divergence (Nowozin et al., 2016). Arjovsky et al. proved that the supports of the fake and real distributions are typically disjoint on low dimensional manifolds and there is a nearly trivial discriminator which can correctly classify the real and fake data (Arjovsky et al., 2017). The loss of such discriminator converges quickly to zero which causes the vanishing gradient for generator. To alleviate such problem, Arjovsky et al. proposed the Wasserstein GAN based on Wasserstein metric requiring no joint supports. Since it is inconvenient to minimize directly the Wasserstein distance, they solve the dual problem by clipping the weights to ensure the Lipschitz condition for discriminator. Later Gulrajani et al. proposed the gradient penalty to guarantee the Lipschitz condition (Gulrajani et al., 2017). Spectral normalization is also proposed to stabilize the training of the discriminator (Miyato et al., 2018).
Most existing methods try to improve the stability of GANs by controlling the discriminator. However, the robustness of GANs have not adequately been considered. When the discriminator is not robust to noise (i.e., the discriminator cannot measure the distance between the fake and real distribution accurately), some examples might be mis-classified which consequently misleads the training of generator. Meanwhile, the poor generalization performance of generator might cause the “blurry” generated images for some potential input noise. Robust Conditional Generative Adversarial Net-
works is proposed to improve the robustness of conditional GAN for noised data. However, this method can merely implement on conditional GAN which improves the ability of generator only to defend the noise (Chrysos et al., 2018). Some other researchers focus on the robustness of GAN to label noise (Thekumparampil et al., 2018)(Kaneko et al., 2019).
In this paper, we attempt to improve the robustness of GANs in a systematical way by promoting the robustness of both discriminator and generator. We propose a novel robust method called robust generative adversarial network (RGAN) where the generator and discriminator still compete with each other iteratively, but in a worst-case setting. Specifically, a robust optimization is designed with considering the worst distribution within the small Wasserstein ball. The generator tries to map the worst input distribution (rather than a specific distribution) to the real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We provide some theoretical analysis for the proposed Robust GAN including generalization. We also implement our robust framework on different baseline GANs (i.e., DCGAN, WGAN-GP, and BWGAN) (Radford et al., 2015)(Adler & Lunz, 2018), observing substantial improvements consistently on all the datasets used in this paper.

heading: GENERATIVE ADVERSARIAL NETWORK
text: The principle of GAN is a game between two players: generator and discriminator, both of which are usually formulated as the deep neural networks. The generator tries to generate a fake example to fool discriminator, while the discriminator attempts to distinguish between fake and real images. Formally, the training procedure of GAN can be formulated as:
min G max D S(G,D) , Ex∼Pr [logD(x)] + Ex̃∼Pg [(1− logD(G(zi)))] (1)
where, x and x̃ = G(z) are real and fake examples sampled from the real data distribution Pr and generation distribution Pg respectively. The generation distribution is defined by G(z) where z ∼ Pz (Pz is a specific input noise distribution). The minmax problem cannot be solved directly since the expectation of the real and generation distribution is usually intractable. Therefore, the approximation problem is defined as:
min G max D
Sm(G,D) , 1
m m∑ i=1 [logD(xi)] + 1 m m∑ i=1 [(1− logD(G(zi)))] (2)
where m examples of xi and zi are sampled from distributions Pr and Pz and the mean value of loss is used to approximate the original problem. However, such a way might not ensure a good robustness of discriminator and generator. Some noised images might not be classified correctly and potential input noise points will cause degraded generation. In this paper, for alleviating such problem, we design a distributionally robust optimization. Particularly, we consider the worst distribution (rather than a specific single distribution) within the small range.

heading: ROBUST GENERATIVE ADVERSARIAL NETWORK
text: As we discussed in the previous sections, although most existing GAN methods can stabilize the training of the discriminator, the robustness might not be adequately considered. In other words, the discriminator might not perform well on some noised data which consequently misleads the training of generator. Similarly, the generator might produce poor generations for certain input noise points if its robustnessis not good. To alleviate such problem, we design the distributionally robust optimization on GAN. Before we discuss how we can achieve this, we first elaborate the distributionally robust optimization.

heading: DISTRIBUTIONALLY ROBUST OPTIMIZATION
text: Let d : X ×X → R+ ∪{∞}. The departure between x and x0 can then be represented by d(x, x0). For distributionally robust optimization, the robustness region P = {P : D(P, P0) ≤ ρ} is considered, a ρ -neighborhood of the distribution P0 under the divergence D(., .) instead of a single
distribution1. The distributionally robust optimization can be formulated as (Sinha et al., 2017):
min θ sup P∈P EP [l(X; θ)] (3)
where l(.) is a loss function parameterized by θ. The problem of (3) is typically intractable for arbitrary ρ.
In order to solve this problem, we first present a proposition:
Proposition 0.1 Let l: θ ×X → R and d: X ×X → R+ be continuous. Then, for any distribution P0 and ρ > 0 we have
sup P∈P {EP [l(X; θ)]− γW (P, P0)} = EP0 [sup x∈X {l(x; θ)− γd(x, x0)}] (4)
(Proof is provided in (Sinha et al., 2017)).
With Proposition 0.1, we can reformulate (3) with the Lagrangian relaxation as follows:
min θ EP0 sup x∈X [l(x; θ)− λd(x, x0)] (5)
where the second term d(x, x0) is to restrict the distance between two points.

heading: ROBUST TRAINING OVER GENERATOR
text: With the distributionally robust optimization, we first discuss how we can perform robust training over generator. The generator of GAN tries to map a noise distribution Pz to the image distribution Pr. The objective of generator is described as follows:
min G
1
m m∑ i=1 [log(1−D(G(zi)))], where zi ∼ Pz (6)
Typically, Pz is a Gaussian distribution. For improving the robustness, we consider all the possible distributions within the robust region Pz = {P : W (P, Pz) ≤ ρz} rather than a single specific distribution (typically a Gaussain in most existing GANs). Here we use the Wasserstein metric to measure the distance between P and Pz , where P is the ρz-neighbor of the original distribution Pz . However, it is difficult to consider all the distributions in this small region, the alternative way is to consider their upper bound (the worst distribution). The robust optimization problem for G is then described as follows:
min G sup P∈Pz
1
m m∑ i=1 [log(1−D(G(zi)))], where zi ∼ P (7)
According to Proposition 0.1, we can relax (7) as:
min G max r
1
m m∑ i=1 [log(1−D(G(zi + ri)))− λz‖r‖22], where zi ∼ Pz (8)
Different from those previous methods, our method attempts to map the worst distribution (in the ρz-neighborhood of the original distribution Pz) to the image distribution. Intuitively, we sample the noise points which are most likely (or the worst) to generate the blurry images and optimize the generator based on these risky points. Therefore, such generator would be robust against poor input noises and might be less likely to generate the low-quality images.

heading: ROBUST TRAINING OVER DISCRIMINATOR
text: In traditional GANs described by (2), the generator attempts to generate a fake distribution to approximate the real data distribution, while the discriminator tries to learn the decision boundary to separate real and fake distributions. Apparently, a discriminator with a poor robustness would inevitably mislead the training of generator. In this section, we utilize the popular adversarial learning
1Normally, the Wasserstein metric W (., .) is used and corresponding d(x, x0) = ‖x− x0‖2p where p > 0
method and propose the robust optimization method to improve the discriminator’s robustness both for clean and noised data.
Specifically, we define the robust regions for both the fake distribution Pg = {P : W (P, Pg) ≤ ρg} and real distribution Pr = {P : W (P, Pr) ≤ ρr}. The generator tries to reduce the distance between the fake distribution Pg and real distribution Pr. The discriminator attempts to separate the worst distributions in Pg and Pr. Intuitively, the worst distributions are closer to decision boundary (less discriminative) and they are able to guide the training of discriminator to perform well on ”confusing” data points near the classification boundary (such discriminator can be more robust than original one). We can reformulate (2) in the robust version:
max D sup P1∈Pr
1
m m∑ i=1 [logD(x′i)] + sup P2∈Pg 1 m m∑ i=1 [log(1−D(G′(zi)))] (9)
where zi ∼ Pz , x′i ∼ P1 and G′ ∼ P2. Using Proposition 0.1, we can relax the alternate problem as:
max D min r1,r2
1
m m∑ i=1 [logD(xi + r i 1)]+ 1 m m∑ i=1 [log(1−D(G(zi) + ri2))] + λd m m∑ i=1 [‖ri1‖22] + ‖ri2‖22]
with zi ∼ Pz, xi ∼ Pr
(10)
Here r1 = {ri1}mi=1 is the set of small perturbations for the points sampled from real distribution Pd which tries to make the real distribution closer to the fake distribution. r2 = {ri2}mi=1 tries to make fake distribution closer to real one. Intuitively, these perturbations try to enhance the difficulty of classification task for discriminator by making real and fake data less distinguishable and it can help promote the robustness of discriminator.

heading: OVERALL OPTIMIZATION
text: We now integrate the robust training of generator and discriminator into a single framework:
min G max D V (G,D) , (1− λ)S(G,D) + sup P :W (P,Pr)≤ρr λEx∼P [logD(x)]
+ sup P :W (P,Pg)≤ρg
λEG′∼P [(1− logD(G′(z)))] (11)
whereG′(zi) = G(zi)+ri2 and zi ∼ pλz . pλz is the mixture distribution defined by pλz = (1−λ)pz + λp′z and p ′ z is the worst distribution defined by p ′ z = arg maxP :W (P,Pz)≤ρz Ex∼P [1− logD(G(x))]. ri2 is arbitrary perturbation. It is noted that we also combine the original GAN into the framework, allowing a more flexible training. The specific algorithm is given as below:
Algorithm 1 Algorithm for RGAN. 1: for number of training iterations do 2: Sample a batch of input noise zi ∼ Pz of size m, a batch of real data xi ∼ Pr of size m. λ is the trade-off parameter for original objective and our objective. 1 and 2 are amplitude of perturbation for input and images respectively.
3: find the worst perturbation {rzadv, rdadv1, rdadv2} by maximizing the objective of generator and minimizing the objective of discriminator: 4: rizadv = arg minri:‖ri‖2=1[log(1−D(G(zi + r i))) + λz‖ri‖22] 5: ridadv1 = arg minri:‖ri‖2=1[logD(xi + r i) + λd‖ri1‖22] 6: ridadv2 = arg minri:‖ri‖2=1[log(1−D(G(zi) + r i)) + λd‖ri2‖22] 7: Update G by descending along its stochastic gradient: 8: ∇θg [ 1m ∑m i=1[log(1−D(G(zi))) + λ m ∑m i=1[log(1−D(G(zi + 1rizadv)))
9: Update D by descending along its stochastic gradient: 10: ∇θd [Sm(G,D)+ λm ∑m i=1[logD(xi+ 2r i dadv1)]+ λ m ∑m i=1[log(1−D(G(zi)+ 2ridadv2))]] 11: end for

heading: THEORETICAL ANALYSIS
text: In this section, we provide theoretical analysis for the RGAN but leave the proof details in appendix. We now show that the optimal discriminator of RGAN balances the mixture of real distributions and the mixture of fake distributions as Lemma 0.2.
Lemma 0.2 For arbitrary fixed G, the optimal D of the game defined by the utility function V (G,D) is:
D∗G(x) = pλr (x)
pλr (x) + p λ g (x)
(12)
where, pλr (x) = (1 − λ)pr + λp′r is the mixture distribution for real data with λ ∈ [0, 1]. p′r is the worst distribution defined by p′r = arg minP :W (P,Pr)≤ρr Ex∼P [logD(x)]. pλg (x) = (1 − λ)pg + λp ′ g is the mixture distribution for fake data. The worst distribution p ′ g is defined by p ′ g = arg minP :W (P,Pg)≤ρg EG′∼P [1− logD(G′(z))].
We further show the optimum point of the utility function V (G,D) as Lemma 0.3.
Lemma 0.3 When the optimum discriminatorD∗ is achieved, the utility function reaches the global minimum if and only if pλg (x) = p λ r (x).
The min-max problem of (11) is computationally intractable due to the expectations over real and fake distributions. An alternate way is to approximate the original problem with the empirical average of finite examples:
min G max D Vm(G,D) , (1− λ)Sm(G,D) +
λ
m m∑ i=1 [logD(x′i)]
+ λ
m m∑ i=1 [(1− logD(G′(zi)))] (13)
where x′i ∼ p′r,G′ ∼ p′g and zi ∼ pλz . pλz is the mixture distribution defined by pλz = (1−λ)pz+λp′z and p′z is the worst distribution defined by p ′ z = arg maxP :W (P,Pz)≤ρz Ex∼P [1− logD(G(x))].
We now provide the analysis for generalization ability as Lemma 0.4. First, we give some assumptions:
Assumption 1 We provide the following assumptions for RGAN:
1. The discriminator logDθ(x) is kθ-Lipschitz in its parameter θ, i.e., |logDθ(x) − logD′θ(x)| ≤ kθ‖θ − θ′‖.
2. The discriminator logDθ(x) is kx-Lipschitz in its x, i.e., |logDθ(x)− logDθ(x′)| ≤ kx‖x− x′‖.

heading: 3. The distance between two arbitrary samples is bounded, i.e., ‖x− x′‖ ≤ ∆B .
text: The generalization ability of discriminator is defined as in (Qi, 2017)(Arora et al., 2017) and it describes if and how fast the difference |V θm − V θ| converges, where, V θ = maxD V (G∗, D) and V θm = maxD Vm(G ∗, D).
Lemma 0.4 Under Assumption 1, with at least probability 1− η, we have:
|V θm − V θ| ≤ (14)
when the number of samples
m ≥ C∆ 2 B(kx) 2
2 (Nlog
kθN
+ log
1 η ) (15)
where C is a sufficiently large constant, and N is the number of parameters of the discriminator function.
Similarly, the generalizability of the generator can be defined as convergence of difference |Qφm − Qφ|, where, Qφ = minG V (G,D∗) and Qφm = minG Vm(G,D∗). We first give the assumptions:
Assumption 2 We provide the following assumptions for RGAN:

heading: 1. The generator Gφ(z) is kφ-Lipschitz in its parameter φ, i.e., |Gφ(z)−G′φ(z)| ≤ kφ‖φ− φ′‖.
text: 2. The discriminator Gφ(z) is kz-Lipschitz in its z, i.e., |Gφ(z)−Gφ(z′)| ≤ kz‖z − z′‖.

heading: 3. The distance between two arbitrary samples is bounded, i.e., ‖z − z′‖ ≤ ∆Bz.
text: Lemma 0.5 Under Assumption 2, with at least probability 1− η, we have:
|Qφm −Qφ| ≤ (16)
when the number of samples
m ≥ Cg∆ 2 Bz k2xk 2 z
2 (Nglog
kθkφNg + log 1
η ) (17)
where Cg is a sufficiently large constant, and Ng is the number of parameters of the generator function.

heading: EXPERIMENTS
text: We present a series of experiments in this section. First, we show that our proposed RGAN can improve the performance of different kinds of baseline models including WGAN-GP, DCGAN, WGAN-GP (resnet), and BWGAN. Inception score and FID are used to evaluate the quality of generations. Following many previous relevant work, we mainly conduct on CIFAR-10 and STL10 the comparison between our proposed method and various baseline models quantitatively while visualizing different models qualitatively on both CIFAR-10 and CelebA. In addition, we plot the bar charts for different baseline models and RGANs on two datasets (CIFAR-10 and STL-10). We also perform the ablation analysis to examine closely our proposed framework. Furthermore, we provide visualizations showing that the performance of baseline models may degrade given some specific input noises (sampled from the worst distribution). In comparison, our proposed method is more robust and can still perform fairly well. In the third part, we provide the visualizations of T-SNE embedding for original and worst distributions. Moreover, we show some images generated by baseline models and our proposed model.

heading: QUANTITATIVE COMPARISON
text: To evaluate the performance of our proposed method, we follow the previous works (Gulrajani et al., 2017; Adler & Lunz, 2018) on robustness and mainly conduct experiments on the CIFAR-10 and STL-10 dataset. There are 4 baseline models including WGAN-GP, WGAN-GP (resnet), DCGAN, and BWGAN. We implemented our proposed robust strategy on these baselines and would like to check if the robust training could indeed improve the performance. The structures and settings of our method are the same as baseline models. We train WGAN-GP, DCGAN, BWGAN and our proposed RGANs with 50, 000 training samples for 200, 000 epochs. For WGAN-GP (resnet) and our corresponding model, we found that 100, 000 epochs appear sufficient. For each 500 epochs, we calculate the inception score for 50, 000 generated images. For training RGANs, there are three hyper-parameters λ (trade off our objective and original one), 1 and 2. We set λ = 0.1 which is searched from {0.001, 0.01, 0.1, 0.5, 1, 2}. We also set 1 = 0.01 and 2 = 4 which was searched from {0.001, 0.01, 0.1, 0.2, 0.5, 1, 2, 4, 5, 10}. For STL-10, we train our models and corresponding baselines with 80w training samples with size 48 × 48. The training settings are totally the same with settings for CIFAR-10. Note that we do not need to adjust hyper-parameters for achieving better performance on the second dataset.
We list the performance for different models in Table 1. Clearly, our proposed RGAN (which is based on WGAN-GP-res) achieves the best result among all the methods in terms of both the criteria, i.e., Inception Score and FID. In order to check if the proposed robust strategy can indeed improve over different baselines, we also detail the performance in Figure 1 where we plot the bar charts for
different baseline models and their robust version with RGAN on two datasets (CIFAR-10 and STL10).2 It is noted that the robust strategy can consistently improve the baselines on the two datasets in terms of both the criteria. In addition, we also show the convergence curves in Figure 2. Clearly, when our robust strategy is applied on the baseline GANs, an obvious increase of the inception scores can be observed (though the convergence speed is similar to that of baseline models). All these experiments indicate that the robust training is indeed necessary and useful.

heading: ABLATION ANALYSIS
text: We conduct the ablation analysis in this subsection. Specifically, we experiment on CIFAR-10 with robust training over generator only, robust training over discriminator only, and robust training over both the generator and discriminator, trying to see if a robust training is necessary on both generator and discriminator. The results are listed in Table 2. As observed, robust training on either generator
2BWGAN appears not to converge in STL-10 in our experiments. For fair comparison, we did not report the performance when BWGAN is used as the baseline in STL-10.
or discriminator can consistently improve the performance of all baseline models, while a joint robust training on both generator and discriminator can further boost the performance. It is interesting to note that robust training on discriminator only could lead to more performance gain than on generator only, implying that a robust discriminator may be more important. This would be investigated as future work.
In addition, taking again CIFAR-10 as one illustrative dataset, we also show that our proposed robust method can perform robust on some potential input noise which might lead to poor generations (input noise sampled from the worst input distribution). Specifically, we generate 50, 000 images with RGAN and various baseline models from the original distribution and worst distribution for five times. Then, we compute the inception score and their corresponding standard deviation. The
results are showed in Figure 3. As observed, without the robust training, those baseline models perform consistently worst in the case of the worst noises input than that of the original input noises. This shows that the traditional GANs may not be robust and may lead to worse performance in case of certain poor input noise. In comparison, when the robust training is implemented, RGAN leads to similar performance even if the worst input noise is given.

heading: VISUALIZATION
text: We present visualization results to compare various methods qualitatively.

heading: VISUALIZATION ON CIFAR-10
text: In this subsection, we present a series of visualization trying to understand visually why the robust GAN could lead to better performance than the traditional GANs. To this end, we sample 500 data points from the original input distribution and worst input distribution respectively. We then plot the 2-dimensional T-SNE embedding of these points. We also would like to plot the real data distribution and the generated data from the traditional GAN as well as our robust GAN. For clarity, we take WGAN-GP as one example but we should bear in mind that the conclusion is basically the same for other traditional GANs like DCGAN. These plots are made in Figure 4 where one can inspect the meaning of each subfigure in the caption. We highlight some remarks as follows. First, Figure 4(a) indicates that the worst distribution covers wider range of areas, especially low density areas of the original distribution; this might cause poor generations since the worst input noise distribution is significantly different from the original input noise. Second, (b) shows that the worst real distribution (red) actually looks much similar to the worst generation distribution. It may be more robust and meaningful to minimize in the worst-case setting the departure of the real data distribution and the fake data distribution, which is conducted in our RGAN. Third, (c) shows that the real data distribution varies largely from the generated data points obtained by traditional GANs, indicating the poor generalization of the traditional GAN; in comparison, with a robust optimization in the worst-case setting, (d) demonstrates that the generated data look very close to the real data.

heading: VISUALIZATION ON CELEBA
text: To clearly examine the visual quality, we demonstrate some images generated by WGAN-GP, DCGAN and their corresponding RGANs on the CelebA dataset. These generated images are shown in Figure 5. As we can observe from these examples, the existing GANs may sometimes lead to very bad generations as circled in (a) and (c). In comparison, with the robust training under the worstcase distribution, such very bad examples can hardly be seen in RGAN. This clearly demonstrates the advantages of the proposed model.

heading: CONCLUSION
text: In this paper, we consider the generalization issue of GANs and propose a robust model called robust generative adversarial network (RGAN). We have designed a robust optimization framework where the generator and discriminator compete with each other in a worst-case setting within a small Wasserstein ball. The generator tries to map the worst input distribution (rather than a specific input distribution) to real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We have provided theories showing that the generalization of the new robust framework can be guaranteed. We also have conducted extensive experiments on CIFAR-10, STL-10 and CelebA datasets with two criteria (Inception score and FID) indicating that our proposed robust framework can improve consistently on several baseline GAN models. Ablation analysis and visualization have demonstrated the advantages of RGAN both quantitatively and qualitatively.

heading: APPENDIX

heading: A. PROOF
text: Lemma 0.2 For arbitrary fixed G, the optimal D of the game defined by the utility function V (G,D) is:
D∗G(x) = pλr (x)
pλr (x) + p λ g (x)
(18)
where, pλr (x) = (1 − λ)pr + λp′r is the mixture distribution for real data with λ ∈ [0, 1]. p′r is the worst distribution defined by p′r = arg minP :W (P,Pr)≤ρr Ex∼P [logD(x)]. pλg (x) = (1 − λ)pg + λp ′ g is the mixture distribution for fake data. The worst distribution p ′ g is defined by p ′ g = arg minP :W (P,Pg)≤ρg EG′∼P [1− logD(G′(z))]. Proof: Given the classifier and generator, the utility function can be rewritten as
V (G,D) , (1− λ)[Ex∼Pr [logD(x)] + EG∼Pg [(1− logD(G(zi)))]] + sup P :W (P,Pr)≤ρr λEx∼P [logD(x)]
+ sup P :W (P,Pg)≤ρg
λEG′∼P [(1− logD(G′(z)))]
= (1− λ) ∫ pr(x)log(D(x))dx
+ (1− λ) ∫ pg(x)log(1−D(x))dx
+ λ ∫ p′r(x)log(D(x))dx
+ λ ∫ p′g(x)log(1−D(x))dx
= ∫ pλr (x)log(D(x))dx
+ ∫ pλg (x)log(1−D(x))dx
(19)
where G′(zi) = G(zi) + ri2 and zi ∼ Pz . ri2 is arbitrary perturbation. Then, it is easy to prove that the optimal D is D∗G(x) = pλr (x) pλr (x)+p λ g (x) .
Lemma 0.3 When the optimum discriminatorD∗ is achieved, the utility function reaches the global minimum if and only if pλg (x) = p λ r (x).

heading: Proof:
text: Given the optimal D∗, we can reformulate the function V (G,D):
V (G,D∗) = ∫ pλr (x)log(
pλr (x)
pλr (x) + p λ g (x)
)dx+ ∫ pλg (x)log(
pλg (x)
pλr (x) + p λ g (x)
)dx
= ∫ pλr (x)log(
pλr (x)
(pλr (x) + p λ g (x))/2
)dx
+ ∫ pλg (x)log(
pλg (x)
(pλr (x) + p λ g (x))/2
)dx− 2log2
= −2log2 +KL(pλr (x)||(pλr (x) + pλg (x))/2) +KL(pλg (x)||(pλr (x) + pλg (x))/2)
(20)
Then, V (G,D∗) can be rewritten as:
V (G,D∗) = −2log2 + 2JSD(pλr (x)||pλg (x)) (21)
where JSD is the Jensen-Shannon divergence, which is always non-negative and the unique optimum is achieved if and only if pλr (x) = p λ g (x).
Assumption 1 We provide following assumptions for RGAN:
1. The discriminatorDθ(x) is kθ-Lipschitz in its parameter θ, i.e., |logDθ(x)−logD′θ(x)| ≤ kθ‖θ− θ′‖.
2. The discriminator Dθ(x) is kx-Lipschitz in its x, i.e., |logDθ(x)− logDθ(x′)| ≤ kx‖x− x′‖.

heading: 3. The distance between two arbitrary samples is bounded, i.e., ‖x− x′‖ ≤ ∆B .
text: Lemma 0.4 Under assumption 1, with at least probability 1− η, we have:
|V θm − V θ| ≤ (22)
when the number of samples
m ≥ C∆ 2 B(kx) 2
2 (Nlog
kθN
+ log
1 η ) (23)
where C is a sufficiently large constant, and N is the number of parameters of the discriminator function, V θ = maxD V (G∗, D) and V θm = maxD Vm(G ∗, D).

heading: Proof:
text: To prove the bound, we need to apply the McDiarmid’s inequality. We first bound the change of function V θm(D,G
∗) when a sample is changed. When i-th samples are replaced by x1i, x1i, Gz1i and G′z1i , the function changes to V θi m (D,G ∗). Then, we have
|V θm(D,G∗)− V θim (D,G∗)|
= 1
m |(1− λ)[logD(xi) + logD(Gzi)]
+ λ[logD(x′i) + logD(G ′ zi)] − (1− λ)[logD(x1i) + logD(Gz1i)] − λ[logD(x′1i) + logD(G′z1i)]| ≤ 1− λ m kx‖xi − x1i‖+ 1− λ m kx‖Gzi −Gz1i‖
+ λ
m kx‖x′i − x′1i‖+
λ m kx‖G′zi −G ′ z1i‖
≤ 2 m kx∆B
(24)
Now we can apply the McDiarmid’s inequality. We have
P (|V θm(D,G∗)− V (D,G∗)| ≥ /2)
≤ 2exp(− 2m
8k2x∆ 2 B
) (25)
The above bound applies to a single discriminatorDθ. To get the union bound, we consider a /8kθnet N , i.e. for any Dθ, there is a θ′ ∈ N so that ‖θ − θ′‖ ≤ /8kθ. This standard net can be constructed to contain finite discriminators such that N ≤ O(Nlog(kθN/ )). N is the number of parameters of discriminator (we here assume the parameter space of the loss function is bounded, then we can construct such a net containing finite points). Therefore, for all θ ∈ N , we have
|V θm − V θ| ≤ /2 (26)
when m ≥ C∆ 2 B(kx) 2 2 (Nlog kθN + log 1 η ).
We further consider the bound beyond θ and we can easily obtain the bounds with the first assumption:
|V θ(D,G∗)− V θ ′ (D,G∗)| ≤ 2kθ‖θ − θ′‖ (27)
and
|V θm(D,G∗)− V θ ′ m (D,G ∗)| ≤ 2kθ‖θ − θ′‖ (28)
The final bound for all discriminator can be obtainged with assumption ‖θ − θ′‖ ≤ /8kθ:
|V θm(D,G∗)−V θ(D,G∗)|
≤ |V θm(D,G∗)− V θ ′ m (D,G ∗)| + |V θ ′
m (D,G ∗)− V θ ′ (D,G∗)|
+ |V θ ′ (D,G∗)− V θ(D,G∗)| ≤
(29)
Assumption 2 We provide the following assumptions for RGAN:

heading: 1. The generator Gφ(z) is kφ-Lipschitz in its parameter φ, i.e., |Gφ(z)−G′φ(z)| ≤ kφ‖φ− φ′‖.
text: 2. The discriminator Gφ(z) is kz-Lipschitz in its z, i.e., |Gφ(z)−Gφ(z′)| ≤ kz‖z − z′‖.

heading: 3. The distance between two arbitrary samples is bounded, i.e., ‖z − z′‖ ≤ ∆Bz .
text: Lemma 0.5 Under assumption 2, with at least probability 1− η, we have:
|Qφm −Qφ| ≤ (30)
when the number of samples
m ≥ Cg∆ 2 Bz k2xk 2 z
2 (Nglog
kθkφNg + log 1
η ) (31)
where Cg is a sufficiently large constant, and Ng is the number of parameters of the generator function, Qφ = minG V (G,D∗) and Qφm = minG Vm(G,D ∗).

heading: Proof:
text: Proof is skipped due to its similarity to Lemma 0.4.

Review summaries:
Developing stable GAN training method has gained much attention these years.  This paper propose to tackle this issue via involving distributionally robust optimization into GAN training. Its main contribution is to combine Sinha et al with GAN, proposing a new GAN training method on the basis of vanilla GAN. Relative theory results are proved and detailed experiments are conducted. 

The present work proposes to combine GANs with adversarial training replacing the original GAN lass with a mixture of the original GAN loss and an adversarial loss that applies an adversarial perturbation to both the input image of the discriminator, and to the input noise of the generator. The resulting algorithm is called robust GAN (RGAN). Existing results of [Goodfellow et al 2014] (characterizing optimal generators and discriminators in terms of the density of the true data) are adapted to the new loss functions and generalization bounds akin to [Arora et al 2017] are proved. Extensive experiments show a small but consistent improvement over a baseline method.