BRIDGING ADVERSARIAL SAMPLES AND ADVERSARIAL NETWORKS
Authors: Faqiang Liu
Mingkun Xu
Guoqi Li
Jing Pei
Luping Shi
Venue:  (2020)
Keywords: ADVERSARIAL SAMPLES
ADVERSARIAL NETWORKS

Abstract: Generative adversarial networks have achieved remarkable performance on various tasks but suffer from sensitivity to hyper-parameters, training instability, and mode collapse. We find that this is partly due to gradient given by non-robust discriminator containing non-informative adversarial noise, which can hinder generator from catching the pattern of real samples. Inspired by defense against adversarial samples, we introduce adversarial training of discriminator on real samples that does not exist in classic GANs framework to make adversarial training symmetric, which can balance min-max game and make discriminator more robust. Robust discriminator can give more informative gradient with less adversarial noise, which can stabilize training and accelerate convergence. We validate the proposed method on image generation tasks with varied network architectures quantitatively. Experiments show that training stability, perceptual quality, and diversity of generated samples are consistently improved with small additional training computation cost.

Under review as a conference paper at ICLR 2020
BRIDGING ADVERSARIAL SAMPLES AND ADVERSAR- IAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial networks have achieved remarkable performance on vari- ous tasks but suffer from sensitivity to hyper-parameters, training instability, and mode collapse. We find that this is partly due to gradient given by non-robust dis- criminator containing uninformative adversarial noise, which can hinder generator from catching the pattern of real samples. Inspired by defense against adversarial samples, we introduce adversarial training of discriminator on real samples that does not exist in classic GANs framework to make adversarial training symmet- ric, which can balance min-max game and make discriminator more robust. Ro- bust discriminator can give more informative gradient with less adversarial noise, which stabilize training and accelerate convergence. We validate the pro- posed method on image generation tasks with varied network architectures quan- titatively. Experiments show that FID score of generated samples is improved by 10% 50% relative to the baseline with additional 25% computation cost. Train- ing stability is improved and mode collapse is alleviated.
INTRODUCTION
Generative adversarial networks (GANs) have been applied successfully in various research fields such as natural image modeling (Radford et al. 2013). image translation (Isola et al.] 2016 Zhu
let al.
2017), cross-modal image generation (Dash
2017), image super-resolution (Ledig al.
2016).
semi-supervised learning (Odena 2016) and sequential data modeling (Mogren 2016:1 Yu
al.
2016). Different from explicit density estimation based models (Kingma et al.J [014] Oord
2016 Hinton| 2012). GANs are implicit generative models with two neural networks playing min-max game to find a map from random noise to target distribution, which the generator tries to generate fake samples to fool discriminator and the discriminator tries to distinguish them from real samples (Goodfellow et al.] 2014). In original GANs formula, optimal discriminator measures the Jensen-Shannon divergence between real data distribution and generated distribution. The discrep- ancy measure can generalized to f-divergence (Nowozin et al., 2016) or replaced by earth-mover distance (Arjovsky et al.) (017)). Despite the success, GANs notoriously difficult to KrdinlKodali ct al.] 2018; Arjovsky & Bottou| 2017). which are very sensitive to hyper- parameters. When the support of these two distributions are approximately disjoint, gradient given by discriminator with standard objective may vanish, and training becomes unstable (Arjovsky et al. (017)). More seri- ously, generated distribution can fail cover the whole data distribution and collapse to a single mode in some cases (Dumoulin et al. 2017 Che et al. 2016).
itt al.
hyper-parameters
[017]Che
The condition of discriminator determines the training stability and performance great extent. On the one hand, representation capacity of discriminator realized by neural network is not infinite. Meanwhile, the discriminator is usually not optimal to measure true discrepancy when trained in an alternative manner practically. On the other hand, discriminator as classifier is also vulnerable to adversarial samples (Appendix D benign samples added by imperceptible perturbation can mis- lead classifier to give wrong prediction (Szegedy et al., 201 GGooffeloow et al.) 2015). Adversarial samples can be easily crafted by gradient-base method such as Fast Gradient Sign Method (FGSM) GGoddfeloo et al. 2015)) or Basic Iterative Method (BIM) (Kurakin al.) 2017). It should be noted that the gradient given by discriminator that guides update of the generator is exactly the same as gradient used to craft adversarial samples of the discriminator. In other words, the gradi- ent contains uninformative adversarial noise which is imperceptible but can mislead the generator. However, generator can still generate meaningful samples in classic GANs training procedure. This

---

Under review as a conference paper at ICLR 2020
is because discriminator is adversarially trained with diverse generated fake samples. Nevertheless, adversarial training on real samples does not exist in classic training framework. As a consequence, training will become unstable when generated distribution approximates target distribution because the gradient given by non-robust discriminator around real samples contains more adversarial noise. To this end, we introduce adversarial training real samples into classic GANs training framework to further improve the robustness of discriminator, which can reduce adversarial noise contained in gradient. can be proved by results shown in Figure empirically that the noise in gradient of adversarially trained discriminator is partly eliminated. Meanwhile, our proposed method can regularize the capability discriminator by performing adversarial training both on real samples and fake samples to alleviate training collapse. We validate the proposed method on image generation tasks with widely adopted DCGAN (Radford et al. 2015) and ResNet (He et al., 2015 5 Gulrajani et al. (017) architecture, which shows consistent improvement of training stability and acceleration of convergence. To our best knowledge, this is the first work to consider GANs from the perspective of adversarial samples, besides which make GANs training scheme symmetric and improve per- formance efficiently with acceptable computation cost. We term the proposed method as adversarial symmetric GAN (AS-GAN).
Figure 1: Visualization of the gradient of DCGAN discriminator with respect input images. The first row shows samples from CelebA dataset. The second row and third row show gradients of adversarially symmetrically trained discriminator and standard discriminator, respectively. We clip gradient to within + 3 standard deviations of their mean and take the average absolute value of three channels for easy visualization. We can see that adversarially trained discriminator can provide more informative gradient with less adversarial noise pattern, which can stabilize GAN training.
2 RELATED WORK
There is large body of work on how to stabilize GANs training and alleviate mode collapse. AAr- jovsky & Bottou, 2017) proved that the widely adopted non-saturating loss function for the gener- ator can be decomposed into Kullback-Leibler divergence minus two Jensen-Shannon divergence when discriminator trained to be optimal, which accounts for training instability and mode dropping during GANs training. (Metz et al.] (0016) proposed to unroll the optimization of discriminator as surrogate objective to guide update of the generator, which shows improvement of training stability with relatively large computation overhead. (Kodali et al. (2018) claimed that the existence of unde- sirable local equilibria is responsible for mode collapse and proposed to regularize the discriminator around real data samples with gradient penalty.
Integral probability metric (IPM) based GANs such as Wasserstein GAN (Arjovsky et al. 2017 and its variants (Gulrajani et al. [017 Wu et al.) 2018) can solve gradient vanishing in GANs s training theoretically but it not simple make discriminator 1-Lipschitz required by the duality conversion practically. Wasserstein GAN (Arjovsky et al.) (017) suggests earth-mover (EM) distance as measure of discrepancy between two distributions and adopts weight clip to make the discriminator I-Lipschitz constrained. WGAN-GP (Gulrajani et al.) 2017) adopts gradient penalty to regularize discriminator in a less rigorous way. but it requires calculation of the second order derivative with remarkable computation overhead. Spectral normalization on the weight of discriminator proposed by (Miyato et al. 2018) can make discriminator I-Lipschitz constrained efficiently, but capacity of discriminator significantly constrained.
2

---

Under review as a conference paper at ICLR 2020
Adversarial vulnerability is intriguing property of neural network- based classifier (Szegedy al. 2014). A well-trained neural network can give totally wrong prediction to adversarial samples that human recognize accurately. Small-magnitude adversarial perturbation added to benign data can be easily calculated based on gradient (Goodffllow et al. 2015 Carlini & Wagner 2017 Dong et al. 2018). (Goodfellow al. 2015) proposed augment training data with adversarial samples to improve the robustness of neural networks, which can smooth the decision boundary of classi- fier around training samples. Gradient of adversarially trained classifier contains more semantic information and less adversarial noise (Tiprs et al. 2018 Kim et al. 2019).
Some work tried craft or defense against adversarial samples using GANs. (Xiao al.) 2018)
proposed to generate adversarial samples efficiently with GANs, in which generator is used to generate adversarial perturbation for target classifier given original samples. (Shen et al., 2017) proposed AE-GAN to eliminate adversarial perturbation in an adversarial training manner, which can output images with better perceptual quality. Different from their motivations, our work aims at improving the robustness of discriminator by introducing adversarial training real samples, which does not exist in classic GANs training framework. Similar to our motivation, (Liu & Hsieh) proposed to perform projected gradient attack in supervised GAN training robustify discriminator. In addition to that, our work clarifies that standard GAN training is approximately equivalent to adversarial training on fake samples. Moreover, incorporate the proposed adversarial training scheme into unsupervised GAN and achieve improvement FID score by a large margin.
G
(2)
D,
Lp
X+ 5
esignVV I.,
Pirst Rorward
Second Forward
Gradiont: for Real Samples
First Backward
Second Bacisward
Figure 2: Schematic of proposed AS-GAN. Standard GAN training is illustrated the first forward pass and the first backward pass. In addition to standard GAN training procedure, we introduce adversarial training of discriminator real samples, illustrated the second forward pass and the second backward pass, which is equivalent to train discriminator with robust optimization.
METHOD
3.1
CLASSIC GAN TRAINING FRAMEWORK
In GAN training framework proposed by Goodffelow et al. 2014), the generator Go parameter- ized by ø tries to generate fake samples to fool discriminator and the discriminator Do(r) param- eterized by tries to distinguish generated samples between real samples. The formulation using min-max optimization is as follows:
min max V(O, Φ)
where V(A, Φ) is the objective function. Equation can be formulated as binary classification problem with cross entropy loss:
V(0, Φ) Ex-ra [log Do + Ez~N())) [log (1 Gs(z))]
where Paata is real data distribution and noise z obeys standard Gaussian distribution. When dis- criminator is trained to be optimal, the training objective for the generator be reformulated as
3

---

Under review as a conference paper at ICLR 2020
divergence, which can measure dissimilarity between two distributions. In practice, we use mini-batch gradient descent to optimize generator and discriminator alternatively.
At each iteration, update rule can be derived follows:
0' = 0 + no V Vm (0. 0.
(3)
o' =
- -
V V Vn (0, (0,
(4)
where To and n/ are the learning rate of discriminator and generator, respectively. Vn (0, o, x, 2) denotes the objective function of mini-batch with m real samples and m fake samples, which
1
V (0, 2,
[og Do(x') + log Do(1 - Go (2)))
After updating parameters of networks, fake samples generated by Gy (z) are adjusted as following equation according to chain rule:
(GO(z)
(z) 2 Go(z) - 170
V Vm Φ, I, x)
(6)
dΦ
(=)
8G5(z)
Gz(zz) V1
)z " Vm,
dΦ
0Φ
where (Ga(*) is a Jacobian matrix. The updated (z) can be seen as adversarial samples of 86
the discriminator at this iteration because 7/ is usually small. These samples will be fed into the discriminator at future iteration to perform adversarial training. From this point of view, the classic training framework mainly includes adversarial training fake samples, which is illustrated as the Nevertheless, adversarial training of discriminator on real samples does not exist in this framework, which makes adversarial training unsymmetric and unbalanced. Adversarial noise contained in the gradient of non-robust discriminator can make training unstable because of the unsmoothed decision boundary of discriminator around real data.
first pass in Figure 2
3.2 ROUBUST OPTIMIZATION
In order to make discriminator more robust, we propose the following objective for robust optimiza- tion of discriminator:
V(0, Φ) EE Pauta
min log Da -6) 6) EZZ.N(0)) [log D& Gz(z))]
(8)
In fact, when training data of real samples is infinite, the above objective is approximately equivalent to the original. However, in practice, training data is always limited, which partly accounts for the existence of adversarial samples. With the proposed objective, discriminator is not only required to classify real data correctly but also should not be vulnerable to small perturbation. This robust optimization training scheme smooth the decision boundary of discriminator and prevent it from being stuck in a local minimum.
3.3 ADVERSARIAL TRAINING ON REAL SAMPLES
In this paper, we select adversarial training as implementation to solve the proposed robust opti- mization problem with Toom constraint. Specifically, we introduce adversarial training on real data that does not exist in the original framework, which can make adversarial training symmetric and stabilize GAN training. Practically, we perform adversarial training after Equation 3 at each iteration as the following equation:
0'= + no VoVn 2,)
where e is an adversarial sample of discriminator, perturbation of which can be obtained by back- ward propagation of Equation 5 with respect to a with negligible computation overhead. The adver- sarial sample can be calculated with constant & as follows:
f x 8 sign( V.V.n (0, 6, x, 2))
(10)
4

---

Under review as a conference paper at ICLR 2020
This adversarial training formulation is adopted from (Goodfellow et al. 2013). which calculates Log -norm constrained perturbation by linearizing objective function. Adversarial training on real samples of discriminator is illustrated the second pass in Figure 2 where Lp denotes the minus of V. (0. p), We make adversarial training procedure symmetric by performing adversarial training both fake samples and real samples. In this way, discriminator can provide more infor- mative gradient with less adversarial noise, which can stabilize training and accelerate convergence. Please refer Algorithm for more details about symmetric adversarial training.
3.4 EFFECTIVE MAGNITUDE OF PERTURBATION
crucial to set an appropriate magnitude of perturbation to make adversarial training effective. When perturbation set to zero, the proposed method degrades to updating discriminator twice on the same real data. When perturbation set to a too large value, real data will be drastically perturbed. Semantic information and quality will be changed, which mislead discriminator recognize degraded samples as real data incorrectly. In sense, adversarial training on real samples regularizes discriminator by augmenting training data, which prevents discriminator from being too strong to alleviate training collapse. The capability of two networks become more balanced with adversarial training both on real samples and fake samples. Noteworthily, the network capacity of discriminator should be larger or at least equivalent to that of generator because discriminator is further constrained with additional adversarial training. In addition, we suggest set pertubation to zero in the begining of training in case discriminator is too weak. We do extensive experiments on how perturbation affects training in the next section.
4 EXPERIMENTS
For the purpose of evaluating our method and investigating the reason behind its efficacy, we test our adversarial training method on image generation tasks on CIFAR-1O, CelebA and LSUN with DCGAN and ResNet architecture. CIFAR-10 10 is well-studied dataset of 32 32 natural images, containing 10 categories of objects and CelebA is a large-scale face attributes dataset with more than 200k images. In addition, We validate the proposed method on 3000k images labeled bedroom in LSUN For fast validation, We resize images in CelebA and LSUN to 64 x 64. The effectiveness of the proposed method can be proved empirically by results on these distinct datasets with two widely adopted architectures.
In section we first conduct unsupervised generation task on CIFAR-10. 10. In order to study transferability on different network structure, we conduct some extensive experiments with multiple architectures and settings in section 4.2 which also demonstrate the advantages of our method that can stabilize training and accelerate convergence in section 4.3 Lastly, we compare our adversarial symmetric training method with some latest state-of-le woffthe-arr art GAN variants. With the same experimental settings of some classic models, our method can produce diverse images with better fidelity.
experiments on the magnitude of perturbation for
In this paper, use standard GAN objective function the adversarial loss. In order to alleviate gradient vanishing at the beginning of training, we adopt non-saturating loss suggested by (Good- fellow et al.] 2014). Furthermore, we Fréchet inception distance (Heusee and inception score (Salianns et al. (0166) to measure model performance, both of which are well- studied metric of image quality. Please refer to Appendix F for more details about implementation. Our source code will be released on Github.
al.) 2017)
4.1 EVALUATION WITH DIFFERENT HYPER-PARAMETTE
In order to find appropriate magnitude of adversarial perturbation, we do unsupervised image generation experiments on CIFAR-I0 with DCGAN architecture at different settings of perturba- tion. Due to the large searching space, we select several typical values for experiments such as (0.1,2,3,43/255. Furthermore, we do ablation study by replacing gradient used to craft perturbation by Gaussian noise. All experiments are run three times independently to reduce randomness.
Figure 3 shows the FID results in the different hyper- parameter setting, Our method performs far better than baseline when e lies in interval 0.5/255. ~ 3/255. However, matter the perturbation
5

---

Under review as a conference paper at ICLR 2020
amplitude is too small (e 0.2/255) model slightly. In addition, when the imposed perturbation is too strong, the model performs even worse than the baseline. This is because samples perturbed drastically can mislead discriminator to recognize degraded samples as real data incorrectly, which hinders discriminator from catching true characteristic of real data distribution. With appropriate perturbation, discriminator can be regularized to balance training, which can alleviate the bothersome collapse of training loss and generate samples in standard GAN. Only when the attacking intensity is suitable, discriminator will become more robust owning to adversarial training, facilitating itself to produce more accurate and informative gradient to generator, which can promote the capability of generator tremendously. In this way, generator obtains more meaningful and reliable update information, accounting for generating better samples. Still and all, too much perturbation is harmful to discriminator when exceeding its endurance, leading discriminator to the ground defeated by generator, which is not a favorable scene. Similarly, when the imposed perturbation is too tiny, on which condition the adversarial sample of real data is almost close to the benign, the effect of adversarial training is limited. In particular, when & is zero, the model degrades into regular GAN algorithm, the only change of which is updating discriminator twice same real data. This setting is only slightly better than the baseline but far worse than the version with appropriate e. Overall, results show that discriminator adversarially trained with appropriate perturbation can stabilize training and improve performance of generator.
little large 4/255), the method improves original
1st
1st
2nd
21d
3rd
3:1
23
28
27
27
26
75
23
bosetw
Calosion
65
3.
0.
0.
6.
(a)
(b)
three independent runs (Lower is better). (b) Mean FID of last 20 epoch. With appropriate perturbation and adversarial updating policy, the quality of generated samples can go beyond baseline by a large margin.
Figure 3: (a) Best FID results under different settings of
Besides, the annotation 'Gaussian' in Figure 3 means that the gradient used to craft adversarial samples is replaced by Gaussian noise, whose poor performance proves that perturbation direction of gradient is important for effectiveness. In the following sections, we use E 1/255 as default option for simplicity.
4.2 EVALUATION WITH DIFFERENT ARCHITECTURES
To explore the transferability and compatibility of the proposed method, we test with widely adopted DCGAN and ResNet architecture on CIFAR-10 10 and CelebA. Figure 4 the comparison results. Additionally, two groups of contrast experiments with different setttings are performed (see Ap- pendix E.. We find that our method can largely accelerate convergence and improve the fidelity of generated samples. By applying our method, stability can also be improved vastly (Appendix I. Even with the setting in which baseline GAN collapse thoroughly, model still converge toward optimization objective.
Generally, GANs collapse when discriminator is too strong or too weak; the former situation is more common during training. WGAN and a series of its variants solve the problem theoretically by using Wasserstein distance measure the discrepancy between two distributions with constraining the

---

Under review as a conference paper at ICLR 2020
discriminator. Different from their approach, address training instability in practical point of view that discriminator as a classifier is not ideal and robust. Adversarial training on real samples introduced into classific GAN to make discriminator more robust. Experimental results prove our method is effective with marginal computation cost and can be applied to different network architectures different datasets.
90
baseline
baseline
80
80
ours
ours
70
70
60
50
50
40
40
30
30
20
20
0
20
40
60
80 100120140
0
10
20
30
40
50
60
70
Iteration (x10*)
Iteration (x10*)
90
90
80
baseline
80
baseline
70
70
60
60
E0
50
40
40
30
30
20
20
10
10
0
20
40
60 80 100120140
0
20
40
60
80
100
120
Iteration (x10")
Iteration (x10²)
Figure 4: Training curves of FID on CIAFR-10 (upper) and CelebA (lower) with DCGAN (left) and ResNet (right). Results show that the proposed method can accelerate convergence and achieve better FID. Meanwhile, it can stabilize training with less sensitivity to network architecture and hyper-parameter setting.
4.3 IMPROVED PERFORMANCE
We perform image generation experiments to verify the effectiveness of our method. As shown in Table[[] through unsupervised training on CIFAR-10, CelebA and bedroom in LSUN, our model can achieve comparable performance to state should be noted that the four rows at the bottom show the results of our implementation. We further validate on ImageNet (Russakovsky et al., (2015) resized to 64 x 64 with unsupervised AS-DCGAN and achieve FID of 60.65 relative 65.78 of baseline. Generated samples shown in Appendix
art on FID and inception score (Higher better).
4.4 ANALYSIS
To show how our method works, we perform some analytical experiments on CIFAR-10. Notice that our algorithm, the only significant change for vanilla GAN adversarial training on real samples during the two-players game. Gradient given by discriminator is the key to update generator. Hence, we visualize the gradient as Figure I which shows that the gradient of adversarially trained discriminator contains more semantic information, eg, profiles of face, but the gradient of standard discriminator looks like uninformative noise. We further show the histogram of the gradient of discriminator with respect real samples Figure Our method can obtain more sparse gradient

---

Under review as a conference paper at ICLR 2020
Table 1: Inception scores and FIDs with unsupervised image generation on CIFAR-10 and CelebA. (Rafford et al., 2013) (experimend by (ang et al.
2017
Miyato et al. 2018). (Wu al. 2017). GGulaajani et al. 2017)
Inception score CIFAR-10
FID CelebA
Method
CIFAR-10
LSUN
(Standard CNN)
DCGAN
6.64
30.9
52.0
61.1
WGAN-GP
6.68
40.21
21.2F
SN-GANT
7.58
25.5
WGAN-GP
7.86#.07*
18.8
18.4
26.81
WGAN-div(ResNet)"
18.1
15.2
15.9
DCGAN
7.05
28.05
20.45
25.36
(SUDS)A
7.21
25.50
10.90
18.08
ResNet
16
22.92
25.72
175.70
(GAN)S/))
7.65
21.84
11.71
45.96
and lower LI norm (Figure 5p) as training iteration increases, which means adversarial noise in gradient is partly climinated. What's more, the proposed method augment training data and smooth the decision boundary discriminator, which can alleviate mode collapse to a large extent shown Appendix C.2 By means of adversarial training evenly both real and fake samples, make the training scheme symmetric and stable.
basdline
40020
curS
25020
10030
25030
20030
(a)
(b)
Figure 5: (a) Histogram of gradient from discriminator with respect to input images. The left is the baseline, and the right the adversarial symmetrically trained discriminator. (b) LI-norm evolution of discriminator gradient during training. Adversarial training will prompt the gradient becoming more sparse
The computation overhead of the proposed method about 25% relative to the baseline, which comparable to that of spectral normalization and much smaller than that of gradient penalty. Comparison of average training time of one epoch of different methods is shown as Table 4.4
Table 2: Average training time of different methods
Setting
DCGAN (OUGS)AS SN-DCGAN
DCGAN-GP
Training time
19.83s
26.40s
24.50s
31.57s
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
The relationship between GANs and adversarial samples has been a open question since both models emerged. In this paper, we analyze that adversarial training on fake samples is already taken into account in standard GAN training framework, but adversarial training on real samples does not exist, which can make training unbalanced and unstable. This is because gradient given by non- robust discriminator contains more adversarial noise, which can mislead update of the generator. In order make training scheme symmetric and discriminator robust, we introduce adversarial training on real samples. We validate the proposed method on image generation tasks CIFAR- 10, CelebA and bedroom in LSUN with varied network architectures. Experiments show that the gradient of discriminator adversarially trained both real and fake samples contains less adversarial noise. Convergence speed and performance are improved with marginal computation overhead. Moreover, mode collapse is alleviated. With simple DCGAN network architecture and standard objective function, we can achieve comparable FID to state of the art on these datasets.
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint 17X1v2170..0462,, 2017. URL//tttss:/accddemii http://ccceemm..mmccoofffttt microooff..
ccmmpaaper/25114500
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan. arXiv preprint
arXiv: 078777 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57, 2017.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. arXiv preprint /6Xiv: 2016. URL http:::ttts/aaateee /atp:////ammmi/i// c.
microsoft
Ayushman Dash, John Cristian Borges Gamboa, Sheraz Ahmed, Muhammad Zeshan Afzal, and Marcus Liwicki. Tac-gan text conditioned auxiliary classifier generative adversarial network. 2017.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9185-9193, 2018
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky, Olivier Mastropi- etro, and Aaron C. Courville. Adversarially learned inference, In ICLR 2017 International Con- ference on Learning Representations 2017, 2017. URL htt.:ttteee//ccccmmcccmmccooo com/ http//paa/22222099999
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Xu Bing, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. International Conference Neural Information Processing Systems, 2014.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. international conference on learning representations, 2015.
Ishaan Guirajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Im- proved training of wasserstein gans. In Advances Neural Information Processing Systems, pp. 5767-5777, 2017. URL /RLLttttss///aaademm
m wwwrosoott..ooomoommmmmoo http://tcccccppppee////22220/000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint urXiv:15/2.033355 2015. URL.
c.mi crosoft.
Com/ httpp//ppp/2229678888
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. 2017.

---

Under review as a conference paper at ICLR 2020
Geoffrey E. Hinton. A practical guide to training restricted boltzmann machines, Neural Networks: Tricks of the Trade (2nd ed.), pp. 599 619, 2012. URL htt.:c/pp///aaaemeii..cccooooooo com/ coomppaprr/48/.0 (4bper/4481576
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A, Efros. Image-to-image translation with conditional adversarial networks. 2016.
Beomsu Kim, Junghoon Seo, and Taegyun Jeon. Bridging adversarial robustness and gradient inter- pretability. 2019.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling, Semi-supervised learning with deep generative models. arXiv preprint arXiv:1406. 1406.5298, 2014. URL https:
7/academic microsoft. com/ httpm//pppr222441142222
Naveen Kodali, James Hays, Jacob Abernethy, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv:1705.072155 2018. URL nttps://accdemii wwt.:emoom.mm...oo.ooooo.mommm..
ccmm/paper22752288112
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. international conference on learning representations, 2017.
Christian Ledig, Lucas Theis, Ferene Huszar, Jose Caballero, Andrew Aitken, Alykhan Tejani, Jo- hannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using generative adversarial network. 2016.
Xuanqing Liu and Cho-Jui Hsich. Rob-gan: Generator, discriminator, and adversarial attacker.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint 10.01166 2016. UBLL/ttppp://aaademlc...ccoooo http://aaaaaeeciiiiccooooooooooe
Ecmmpapprr225524622
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. 2018.
Olof Mogren. C-rnn-gan: Continuous recurrent neural networks with adversarial training. 2016.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. fgan: training generative neural samplers using variational divergence minimization. In NIPS' 16 Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 271-279, 2016. URL https:// academic.miee wwosooffftcrooocccoooooccoco.cce coomppape:/9933000000
Augustus Odena. Semi-supervised learning with generative adversarial networks. 2016.
Aaron Van Den Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel recurrent neural networks. 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. Computer Science, 2015.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-255 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, and Chen Xi. Improved tech- niques for training gans. 2016.
Shiwei Shen, Guoqing Jin, Gao Ke, and Yongdong Zhang. Ae-gan: adversarial eliminating with gan. 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan J. Good- fellow, and Rob Fergus. Intriguing properties of neural networks. international conference on learning representations, 2014.
10

---

Under review as a conference paper at ICLR 2020
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turnrr, and Aleksander Madry. Robustmess may be at odds with accuracy. arXiv 18.. 2018. URL https: // //academic wwcrosooi.iiic.i...ommmo.ooo.... httpttt.cccpppppppp.eeeeee conppaaper/29973553333
www.
Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. Wasserstein diver- gence for gans. 2017.
Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. Wasserstein diver- gence for gans. In Computer Vision ECCV 2018. 15th European Conference, Munich, Ger- many, September 8-14, 2018, Proceedings, Part volume 11209, pp. 673-688, 2018. URL https://acaden: wttp://o.........///////////2220
Chaowei Xiao, Li Bo, Jun Yan Zhu, Warren Hc, and Dawn Song. Generating adversarial examples with adversarial networks. 2018.
Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. Lr-gan: Layered recursive generative adversarial networks for image generation. 2017.
Lantao Yu, Weinan Zhang. Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. 2016.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference Computer Vision, 2017.
A ANALYSIS
A.1 TRAINING STABILITY
We further demonstrate the superiority stability. During training on CIFAR-10, we plot the prediction confidence of discriminator about real data and adversarial samples of real data. The detailed confidence and loss value of discriminator with the training process are depicted as Figure
algorithm over traditional GAN methods about training
clearly exhibits that the discriminator of traditional DCGAN has very unstable output confidence and loss curve during training. With our method, both of the variables Do(z) and err become more smooth and stable. Moreover, when adversarial perturbation confidence about adversarial samples Do will be basically lower than D (x: by large margin. Because in the beginning. the discriminator is non-robust and sensitive to adversarial samples. With more iterations, discriminator will become robust to adversarial samples, thus Do(ā) converges to be higher than before, but still inferior to (x(x) due to non-eliminated perturbation. As shown in Appendix D discriminator with adversarial training is more robust than standard discriminator. Similarly, the loss also stabilized with our algorithm (Figure (p)). leading to a more easy- ttainirain less-sensitive hyper-parametedd and high-performance model.
appropriate for training, the

---

Under review as a conference paper at ICLR 2020
Convergance
AS-GMMcemD
DCGAN:
AS-GAN:
20
40
60
600
100
120
140
20
40
02
80
100
120
3.40
Iteration octot:
ther stttntttonn 122.07
(b)
Figure 6: (a) Confidence of discriminator real data and adversarial samples of real data during training. (b) Evolution of discriminator loss at different settings.
A.2 ABLATION STUDY
In addition, visualize the convergence process through evaluation by inception score and FID. As described in Figure 7 when updating discriminator on real samples twice (e = 0) or giving the perturbation of random direction ('Gaussian'), ), the models achieve similar performance to baseline. Only with the appropriate setting of perturbation and symmetric training scheme, desirable results can be realized.
60
7.-
52
0.5
43
2
35
25
20
100
120
140
20
Dtrration 1020")
(a)
(b)
Figure 7: (a) Training curves of inception with different methods. (b) Training curves of FID with different methods. With the increasing iterations, our algorithm (E 1/255) converges faster and better than other models.
12

---

Under review as a conference paper at ICLR 2020
B ALGORITHM OF AS-GAN
Algorithm Mini-batch stochastic gradient descent training of AS-GAN. We perturbation to 1/ 1/255 as default for image generation tasks.
1: for number of training iterations do
2: Sample mini-batch of m noise samples {2 .... zm from Gaussian distribution (0, I).
3: Sample mini-batch of m data samples {a', ..... xm' from real data distribution Pdata-
4: Update the discriminator by gradient ascent, this step can be regarded as adversarial training of discriminator on fake samples:
#' 0 VoV. 00,0,
(11)
5: Craft adversarial samples of real samples for discriminator based on gradient:
E sign V vn (0,
(12)
6: Perform adversarial training of discriminator on real samples:
8' + 1/0 V (0,
(13)
7: Update generator by gradient descent, this step can be regarded as crafting adversarial sam- ples of fake data:
Φ' = 7p V V 20,
(14)
8: end for
C RESULTS PRESENTATION
C.1 GENERATED SAMPLES
(b)
Figure 8: (a): 32 32 CIFAR-10 samples generated by AS AS-ResNet. (b): 64 64 CelebA samples
generated by AS-ResNet. We believe these samples are at least comparable to the best published results so far.
13

---

Under review as a conference paper at ICLR 2020
(b)
Figure 9: (a): Interpolation results from AS-ResNet on CIFAR-10. (b): Interpolation results from
AS-ResNet CelebA.
(a)
(b)
Figure 10: (a): 64 x 64 results from AS-DCGAN on bedroom in LSUN. (b): 64 x 64 results from unsupervised AS-DCGAN on ImageNet.
14

---

Under review as a conference paper at ICLR 2020
25
Figure 11: 128 x 128 results from AS-DCGAN on CelebA
C.2 ALLEVIATED MODE COLLAPSE
(b)
Figure 12: (a): Collapsed samples generated by standard GAN trained on CelebA. (b): Samples
generated by AS-ResNet trained on CelebA.
15

---

Under review as a conference paper at ICLR 2020
(b)
Figure 13: (a): Collapsed samples generated by standard GAN trained on CIFAR-1O. (b): Samples
generated by AS-ResNet trained CIFAR-10.
D ADVERSARIAL SAMPLES OF DISCRIMINATOR
Figure 14: Benign samples (on odd rows) and adversarial samples of standard discriminator (on even rows). Confidence is depicted at corner. Standard discriminator is extremely vulnerable to imperceptible perturbation. The pertubation level is 1/255.
16

---

Under review as a conference paper at ICLR 2020
Figure 15: Benign samples (on odd rows) and adversarial samples of discriminator adversarially trained on real data (on even rows). Confidence is depicted at corner. After adversarial training on real samples, discriminator is more robust to adversarial perturbation. The pertubation level is 1/255.
E
COMPARISON IN DIFFERENT SETTINGS
Table 3: FIDs with unsupervised image generation on CIFAR-10.
Setting
Standard
Adam(32=0.9)
no BN(G)
BN(D)
BN(Both)
Irp=de-4
Baseline 28.05
29. 14
29.40
69.82
59.15
28.44
AS-GAN(ours)
25.50
27.18
26.72
51.30
56.18
26.57
Here, 'standard' means using same setting with Appendix[F From table and table under different conditions, model always better than baseline.
Table 4: FIDs with unsupervised image gener- ation CelebA.
IrD=2e-4 4 bra=2e-4
IrD=5e-5 5
Setting
Ire=2e- 4
Baseline
34.10
18.73
AS-GAN(ours)
12.14
11.82
F IMPLEMENTATION AND TRAINING DETAILS
Our models are implemented by Pytorch with acceleration of RTX 2080ti GPUs, only when eval- uating inception score we use Tensorflow. Before fed into discriminator, rescale images within []. Dimension of the latent vector is set to 100 for all implementations. We train models on CelebA for 100 epoch, CIFAR-10 for 200 epoch, LSUN for 10 epoch and Imagenet for 10 epoch. FID and inception score are computed from 22400 generated samples, and inception score caleu- lated by 10 independent partitions.
17

---

Under review as a conference paper at ICLR 2020
In ResNet architecture, the residual block is organized BauchNorr-ReeLU-Com
ReLU-Resample-CCn
ReLU-Conv with skip connection. We use bilinear interpolation for upsampling and average pooling for downsampling. Batch normalization (loffe & Szegedy 2015) is used both for generator and discriminator. Parameters network are initialized by Xavier method. We train networks with Adam optimizer with learning rate 2e-4. 81 set to 0.5 and [3 set to 0.999.
In DCGAN architecture, the basic block is organized as Conv- Sww--Bachhoon-Leakyleeee inator or SonyTransposed-Batchhhee BatchNorm-ReLL. for generator. The weights of convolution are initial- ized by normal disitribution with zero mean and 0.02 standard deviation. We do not use bias in convolution. We train DCGAN with Adam optimizer with learning rate 2e-4. $1 set to 0.5 and B2 is set to 0.999. Because training standard GAN on CelebA is unstable, we decrease learning rate of discriminator 5e-5 balance training as TTUR training strategy (Heusel al. 2017).
for discrim-
Table 5: ResNet Generator Gz(z) for 32 x 32
Block
Kernel size
Resample
Output shape
Input z
100
Linear
256 4
Residual block
[3 x 3
x 2
Up
256 x 8 x
Residual block
x
Up
256
x
Residual block
[3 3
x 2
Up
256 x 32 x 32
Conv, tanh
3x3
x 32
Table ResNet Discriminator Dø(x:) for 32 x 32
Block
Kernel size
Resample
Output shape
Residual block
3 x 3]
Down
256 16 16
Residual block
3 x 3
x
Down
256 x 8 8
Residual block
33 x 3
Down
256 x 4
Residual block
[3 x 3
2
Down
256 x 2 2
ReLU, Average pool
256
Linear, Sigmoid
1
Table 7: ResNet Generator Gz(z) for 64 x 64
Block
Kernel size
Resample
Output shape 100
Input
Linear
256 x 4 x
Residual block
3
Up
256 8
Residual block
x 3
x
Up
256 x 16 x 16
Residual block
3
Up
256 32 x
Residual block
[3 3
x 2
Up
256 x 64 x 64
Cony, tanh
3x3
x 64
18

---

Under review as a conference paper at ICLR 2020
Table ResNet Discriminator Do(a) for 64 64
Block
Kernel size
Resample
Ouiput shape
Residual block
3 x 3
Down
256 x 32 32
Residual block
3 x 3
2
Down
256 16
16
Residual block
33 x 3
x 2
Down
256 x 8 8
Residual block
[3 x 3
2
Down
256 x 4 4
Residual block
[3 x 3]
x 2
Down
256 x 2 2
ReLU, Average pool
256
Linear, Sigmoid
Table 9: DCGAN Generator Gø(z) for 32 x 32
Block
Kernel size
Stride
Output shape
Input z
100
Basic block
4 4
512 x 4 4
Basic block
4
256
x
Basic block
4 4
2
128 x 16 16
TonnTranpposed, tanh
32
32
Table 10: DCGAN Discriminator Do(x)) for 32 32
Block
Kernel size
Stride
Output shape
Cony, LeakyReLU
4 x 4
2
128 16 x 16
Basic block
x 4
2
256 x 8 x
Basic block
4 x 4
2
512 x 4 x
Conv, Sigmoid
x
1
Table 11: DCGAN Generator Gø(z) for 64 64
Block
Kernel size
Stride
Output shape 100
Input
Basic block
4 x 4
1024 x 4 x 4
Basic block
4
512 8 x
Basic block
4 x 4
2
256 x 16 x 16
Basic block
x x
Conv'l ConyTransposed tanh
4 x 4
2
3 x 64 64
Table 12: DCGAN Discriminator Do(x:) for 64
Block
Kernel size
Stride
Output shape
Cony, LeakyReLU
x 4
2
128 32 32
Basic block
4 x 4
2
256 16 x 16
Basic block
x
2
512 8
Basic block
4 X 4
2
1024 x 4 x 4
Conv, Sigmoid
X 4
19


Reviews:
[{"review_id": "rJx0AWlwKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting idea based on introducing adversarial noise on real samples during GAN training. This novel approach may improve GAN training and have potentially large impact, but the paper in its current form is slightly below the standard of ICLR due to its lack of clarity.\n\nWhile it is very interesting to apply adversarial noise in real data, this approach is not clearly motivated or explained. For example, at the beginning of page 2, why “As a consequence, training will become unstable when generated distribution approximates target distribution because the gradient given by non-robust discriminator around real samples contains more adversarial noise”? One may think that, on the contrary, the adversarial noise in generated samples would approximate that in real data when the generator distribution approximate the true data distribution. Similarly, after eq. 6, why “Consequently, gradient given by discriminator may vanish when discriminator becomes stronger than generator without capacity constrained.”? I would think the gradient would explode in this case given all the regularisation on gradients.\n\nIn addition, the term “non-robust discriminator” has been used several times in important places, but is not clearly defined. Properties used to justify the approach, such as “symmetric” and “balanced” need to be explained. For example, would it be possible to illustrate or measure the imbalance of discriminator?\n\nThe overhead of the algorithm needs to be stated more precisely. (after eq. 8) It is unclear to me that “backward propagation of Equation 5 with respect to x with negligible computation overhead”. Would this require back-prop through the entire discriminator, which can be very deep thus costly? It would be helpful to provide an estimation of runtime overhead supported by experiments.\n\nIn algorithm 1, why is it necessary to perform discriminator update twice? How about skipping step 4 (eq. 12)? I think this may better mirror the adversarial update for generated samples, which only involves updating generator parameters once.\n\nIn section 3.4, I am not sure it is similar to unrolled GAN when the perturbation is zero. Unrolled GAN required backprop through discriminator update, which I don’t think is the case here\n\nFinally, the experimental results are confusing. In Table 1, why the reproduced results are already much better than those in the original papers? A solid baseline is necessary for any further assessment. Further more, since the DCGAN and ResNet baseline models did not use recent regularisation approaches such as spectral-norm, it is hard to assess whether the proposed method can work without those existing techniques. It would be helpful to use at least a SN-GAN baseline.\n\nThe analysis also need to be improved. It is hard to interpret the histograms and L1 norms in Figure 5 as related to “informative gradient” or “semantic information” as claimed in sectiion 4.4. Quantitative measurements such as correlation or mutual information may justify these claims.\n\nOverall, I think the idea presented is certainly promising, but needs further development to be qualified for acceptance.\n\nNit:\n\nThe 1-line derivation of eq. 6 can be incorporated into the main text.\n\nA transpose in eq.10 is missing.\n\nUpdate:\n\nI read the authors' rebuttal, which addressed many of my concerns and presented additional empirical results. The improvement over SN-GAN baseline is particularly convincing. I therefore believe the final camera-ready version can be a valuable contribution to the field, and would like to recommend accepting this paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HylxPVL3Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to propose a new method to stabilize the training procedure of GAN. To this end, they use adversarial samples of real data to train the discriminator, and claim that it is helpful to reduce the adversarial noise contained in the gradient. Although training GAN with adversarial samples of discriminator is somewhat novel, the method and experiments are not convincing. \nI do not recommend the acceptance based on the limited contribution of this paper. The following is a detailed evaluation. \n\n1. The paper uses vague description such as “This approach can improve the robustness of discriminator and reduce adversarial noise contained in gradient” without convincing justification. Please give a formal description or notation of “adversarial noise contained in gradient”, and discuss how to remove the effect of “adversarial noise” in principle instead of extensively testing adversarial training of discriminator. \n\n2. The experiment is not convincing and the improvement is not significant. The author running adversarial training on CIFAR10 dataset with FGSM and the perturbation is tested from 0.2/255 ~ 4/255. The performance (FID score) is a bit sensitive to the amount of perturbation level. Moreover, this The improvement over DCGAN is quite limited given previous works such as WGAN-GP. Combined together, the result is not convincing (it seems to be a heavy tuning result rather than a principled solution).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByeFqVKpdr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new GAN training that additionally feeds adversarial examples as the real samples to the discriminator D. A key motivation here is to regularize the target real distribution simulated by D to be robust to adversarial perturbations. Experimental results show that the proposed GAN training generally improves the generation performance from the vanilla GAN training in CIFAR-10, CelebA and LSUN datasets. \n\nIn overall, I liked its clear motivation and the simplicity of the method. Experimental results are also presented clearly, and shows a significant improvement. One of my main concerns, however, is that robustifying D in GAN training is not a new idea for some readers [1], so they need more clarification on the novelty of the proposed method, e.g. by discussing about it in related work or by comparing the performance. \n\n- In regarding robust optimization, I think [2] could cover a lot of practices considered in this paper. Some questions listed here are relevant to this point:\n    (a) The paper only considers to use FGSM in adversarial training part, but FGSM training on large epsilon usually leads to overfitting [2] on CIFAR-10. I wonder if the authors have tried PGD counterpart in their method.\n    (b) It seems that the method uses both natural and adversarial examples in adversarial training, as in [3]. Instead, there is another (and perhaps more common) type of adversarial training [2] that uses only adversarial examples for the training. What happens if this training is applied to the method?\n\n- The method makes an additional parameter updates (Ep. 3 and 8) for adversarial training. Could this step make additional gain to the method by training more, i.e. perhaps it is a bit unfair to the vanilla training?\n\n- It would strengthen the claim if the paper could present whether the robustness of D is indeed increased, e.g. by comparing adversarial accuracies?\n\n- Table 1: I feel there should be more discussion about why the reproduced values are so different compared to that of previously reported values, as they might confuse the readers to convince the claimed results.\n\n- Eq. 7: The first term in the right hand side have to be E[max (log D(x - d))] instead of E[(log D(x - d))]?\n\n[1] Liu, X., & Hsieh, C. J. (2019). Rob-gan: Generator, discriminator, and adversarial attacker. CVPR 2019.\n[2] Madry, A. et al. (2017). Towards deep learning models resistant to adversarial attacks. ICLR 2018.\n[3] Goodfellow, I. et al. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes incorporating adversarial training on real images to improve the stability of GAN training. The key idea relies on the observation that GAN training already implicitly does a form of adversarial training on the generated images and so this work proposes adding adversarial training on real images as well. In practice, adversarial training on real images is performed using FGSM and experiments are conducted on CelebA, CiFAR10, and LSUN reporting using standard generative metrics like FID.

Initially all reviewers were in agreement that this work should not be accepted. However, in response to the discussion with the authors Reviewer 2 updated their score from weak reject to weak accept. The other reviewers recommendation remained unchanged. The core concerns of reviewers 3 and 1 is limited technical contribution and unconvincing experimental evidence. In particular, concerns were raised about the overlap with [1] from CVPR 2019. The authors argue that their work is different due to the focus on the unsupervised setting, however, this application distinction is minor and doesn’t result in any major algorithmic changes. With respect to experiments, the authors do provide performance across multiple datasets and architectures which is encouraging, however, to distinguish this work it would have been helpful to provide further study and analysis into the aspects unique to this work -- such as the settings and type of adversarial attack (as mentioned by R3) and stability across GAN variants. 

After considering all reviewer and author comments, the AC does not recommend this work for publication in its current form and recommends the authors consider both additional experiments and text description to clarify and solidify their contributions over prior work.

[1] Liu, X., & Hsieh, C. J. (2019). Rob-gan: Generator, discriminator, and adversarial attacker. CVPR 2019.

Decision: Reject