Title: BRIDGING ADVERSARIAL SAMPLES AND ADVERSAR-IAL NETWORKS

Abstract: Generative adversarial networks have achieved remarkable performance on various tasks but suffer from sensitivity to hyper-parameters, training instability, and mode collapse. We find that this is partly due to gradient given by non-robust discriminator containing uninformative adversarial noise, which can hinder generator from catching the pattern of real samples. Inspired by defense against adversarial samples, we introduce adversarial training of discriminator on real samples that does not exist in classic GANs framework to make adversarial training symmetric, which can balance min-max game and make discriminator more robust. Robust discriminator can give more informative gradient with less adversarial noise, which can stabilize training and accelerate convergence. We validate the proposed method on image generation tasks with varied network architectures quantitatively. Experiments show that FID score of generated samples is improved by 10% -50% relative to the baseline with additional 25% computation cost. Training stability is improved and mode collapse is alleviated.

1. INTRODUCTION:

Generative adversarial networks (GANs) have been applied successfully in various research fields such as natural image modeling (Radford et al., 2015) , image translation (Isola et al., 2016; Zhu et al., 2017) , cross-modal image generation (Dash et al., 2017) , image super-resolution (Ledig et al., 2016) , semi-supervised learning (Odena, 2016) and sequential data modeling (Mogren, 2016; Yu et al., 2016) . Different from explicit density estimation based models (Kingma et al., 2014; Oord et al., 2016; Hinton, 2012) , GANs are implicit generative models with two neural networks playing min-max game to find a map from random noise to target distribution, in which the generator tries to generate fake samples to fool discriminator and the discriminator tries to distinguish them from real samples (Goodfellow et al., 2014) . In original GANs formula, optimal discriminator measures the Jensen-Shannon divergence between real data distribution and generated distribution. The discrepancy measure can be generalized to f-divergence (Nowozin et al., 2016) or replaced by earth-mover distance (Arjovsky et al., 2017) . Despite the success, GANs are notoriously difficult to train (Kodali et al., 2018; Arjovsky & Bottou, 2017) , which are very sensitive to hyper-parameters. When the support of these two distributions are approximately disjoint, gradient given by discriminator with standard objective may vanish, and training becomes unstable (Arjovsky et al., 2017) . More seriously, generated distribution can fail to cover the whole data distribution and collapse to a single mode in some cases (Dumoulin et al., 2017; Che et al., 2016) .

The condition of discriminator determines the training stability and performance to a great extent. On the one hand, representation capacity of discriminator realized by a neural network is not infinite. Meanwhile, the discriminator is usually not optimal to measure true discrepancy when trained in an alternative manner practically. On the other hand, discriminator as a classifier is also vulnerable to adversarial samples (Appendix D): benign samples added by imperceptible perturbation can mislead classifier to give wrong prediction (Szegedy et al., 2014) (Goodfellow et al., 2015) . Adversarial samples can be easily crafted by gradient-based method such as Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015) or Basic Iterative Method (BIM) (Kurakin et al., 2017) . It should be noted that the gradient given by discriminator that guides update of the generator is exactly the same as gradient used to craft adversarial samples of the discriminator. In other words, the gradient contains uninformative adversarial noise which is imperceptible but can mislead the generator. However, generator can still generate meaningful samples in classic GANs training procedure. This is because discriminator is adversarially trained with diverse generated fake samples. Nevertheless, adversarial training on real samples does not exist in classic training framework. As a consequence, training will become unstable when generated distribution approximates target distribution because the gradient given by non-robust discriminator around real samples contains more adversarial noise. To this end, we introduce adversarial training on real samples into classic GANs training framework to further improve the robustness of discriminator, which can reduce adversarial noise contained in gradient. It can be proved by results shown in Figure 1 empirically that the noise in gradient of adversarially trained discriminator is partly eliminated. Meanwhile, our proposed method can regularize the capability discriminator by performing adversarial training both on real samples and fake samples to alleviate training collapse. We validate the proposed method on image generation tasks with widely adopted DCGAN (Radford et al., 2015) and ResNet (He et al., 2015; Gulrajani et al., 2017) architecture, which shows consistent improvement of training stability and acceleration of convergence. To our best knowledge, this is the first work to consider GANs from the perspective of adversarial samples, besides which we make GANs training scheme symmetric and improve performance efficiently with acceptable computation cost. We term the proposed method as adversarial symmetric GAN (AS-GAN).

Figure 1 : Visualization of the gradient of DCGAN discriminator with respect to input images. The first row shows samples from CelebA dataset. The second row and third row show gradients of adversarially symmetrically trained discriminator and standard discriminator, respectively. We clip gradient to within ± 3 standard deviations of their mean and take the average absolute value of three channels for easy visualization. We can see that adversarially trained discriminator can provide more informative gradient with less adversarial noise pattern, which can stabilize GAN training.

2. RELATED WORK:

There is a large body of work on how to stabilize GANs training and alleviate mode collapse. (Arjovsky & Bottou, 2017) proved that the widely adopted non-saturating loss function for the generator can be decomposed into Kullback-Leibler divergence minus two Jensen-Shannon divergence when discriminator trained to be optimal, which accounts for training instability and mode dropping during GANs training. (Metz et al., 2016) proposed to unroll the optimization of discriminator as surrogate objective to guide update of the generator, which shows improvement of training stability with relatively large computation overhead. (Kodali et al., 2018) claimed that the existence of undesirable local equilibria is responsible for mode collapse and proposed to regularize the discriminator around real data samples with gradient penalty.

Integral probability metric (IPM) based GANs such as Wasserstein GAN (Arjovsky et al., 2017) and its variants (Gulrajani et al., 2017; Wu et al., 2018) can solve gradient vanishing in GANs training theoretically but it is not simple to make discriminator 1-Lipschitz required by the duality conversion practically. Wasserstein GAN (Arjovsky et al., 2017) suggests earth-mover (EM) distance as a measure of discrepancy between two distributions and adopts weight clip to make the discriminator 1-Lipschitz constrained. WGAN-GP (Gulrajani et al., 2017) adopts gradient penalty to regularize discriminator in a less rigorous way, but it requires calculation of the second order derivative with remarkable computation overhead. Spectral normalization on the weight of discriminator proposed by (Miyato et al., 2018) can make discriminator 1-Lipschitz constrained efficiently, but capacity of discriminator is significantly constrained.

Adversarial vulnerability is an intriguing property of neural network-based classifier (Szegedy et al., 2014) . A well-trained neural network can give totally wrong prediction to adversarial samples that human can recognize accurately. Small-magnitude adversarial perturbation added to benign data can be easily calculated based on gradient (Goodfellow et al., 2015; Carlini & Wagner, 2017; Dong et al., 2018) . (Goodfellow et al., 2015) proposed to augment training data with adversarial samples to improve the robustness of neural networks, which can smooth the decision boundary of classifier around training samples. Gradient of adversarially trained classifier contains more semantic information and less adversarial noise (Tsipras et al., 2018; Kim et al., 2019) . Some work tried to craft or defense against adversarial samples using GANs. (Xiao et al., 2018) proposed to generate adversarial samples efficiently with GANs, in which a generator is used to generate adversarial perturbation for target classifier given original samples. (Shen et al., 2017) proposed AE-GAN to eliminate adversarial perturbation in an adversarial training manner, which can output images with better perceptual quality. Different from their motivations, our work aims at improving the robustness of discriminator by introducing adversarial training on real samples, which does not exist in classic GANs training framework. Similar to our motivation, (Liu & Hsieh) proposed to perform projected gradient attack in supervised GAN training to robustify discriminator. In addition to that, our work clarifies that standard GAN training is approximately equivalent to adversarial training on fake samples. Moreover, we incorporate the proposed adversarial training scheme into unsupervised GAN and achieve improvement of FID score by a large margin.

G  D  z x (z) G  D L x   sign( ) x D L   First Forward First Backward

Second Forward Second Backward

Gradient for Real Samples 

3. METHOD:



3.1. CLASSIC GAN TRAINING FRAMEWORK:

In GAN training framework proposed by (Goodfellow et al., 2014) , the generator G φ (z) parameterized by φ tries to generate fake samples to fool discriminator and the discriminator D θ (x) parameterized by θ tries to distinguish generated samples between real samples. The formulation using min-max optimization is as follows:

EQUATION

where V (θ, φ) is the objective function. Equation 1 can be formulated as a binary classification problem with cross entropy loss:

EQUATION

where P data is real data distribution and noise z obeys standard Gaussian distribution. When discriminator is trained to be optimal, the training objective for the generator can be reformulated as Jensen-Shannon divergence, which can measure dissimilarity between two distributions. In practice, we use mini-batch gradient descent to optimize generator and discriminator alternatively.

At each iteration, update rule can be derived as follows:

EQUATION

where η θ and η φ are the learning rate of discriminator and generator, respectively. V m (θ, φ, x, z) denotes the objective function of mini-batch with m real samples and m fake samples, which is:

EQUATION

After updating parameters of networks, fake samples generated by G φ (z) are adjusted as following equation according to chain rule:

EQUATION

where

∂G φ (z) ∂φ

is a Jacobian matrix. The updated G φ (z) can be seen as adversarial samples of the discriminator at this iteration because η φ is usually small. These samples will be fed into the discriminator at future iteration to perform adversarial training. From this point of view, the classic training framework mainly includes adversarial training on fake samples, which is illustrated as the first pass in Figure 2 . Nevertheless, adversarial training of discriminator on real samples does not exist in this framework, which makes adversarial training unsymmetric and unbalanced. Adversarial noise contained in the gradient of non-robust discriminator can make training unstable because of the unsmoothed decision boundary of discriminator around real data.

3.2. ROUBUST OPTIMIZATION:

In order to make discriminator more robust, we propose the following objective for robust optimization of discriminator:

EQUATION

In fact, when training data of real samples is infinite, the above objective is approximately equivalent to the original. However, in practice, training data is always limited, which partly accounts for the existence of adversarial samples. With the proposed objective, discriminator is not only required to classify real data correctly but also should not be vulnerable to small perturbation. This robust optimization training scheme can smooth the decision boundary of discriminator and prevent it from being stuck in a local minimum.

3.3. ADVERSARIAL TRAINING ON REAL SAMPLES:

In this paper, we select adversarial training as implementation to solve the proposed robust optimization problem with L ∞ -norm constraint. Specifically, we introduce adversarial training on real data that does not exist in the original framework, which can make adversarial training symmetric and stabilize GAN training. Practically, we perform adversarial training after Equation 3 at each iteration as the following equation:

EQUATION

where x is an adversarial sample of discriminator, perturbation of which can be obtained by backward propagation of Equation 5 with respect to x with negligible computation overhead. The adversarial sample can be calculated with constant ε as follows:

EQUATION

This adversarial training formulation is adopted from (Goodfellow et al., 2015) , which calculates L ∞ -norm constrained perturbation by linearizing objective function. Adversarial training on real samples of discriminator is illustrated as the second pass in Figure 2 , where L D denotes the minus of V m (θ, φ, x, z). We make adversarial training procedure symmetric by performing adversarial training both on fake samples and real samples. In this way, discriminator can provide more informative gradient with less adversarial noise, which can stabilize training and accelerate convergence.

Please refer to Algorithm 1 for more details about symmetric adversarial training.

3.4. EFFECTIVE MAGNITUDE OF PERTURBATION:

It is crucial to set an appropriate magnitude of perturbation to make adversarial training effective.

When perturbation set to zero, the proposed method degrades to updating discriminator twice on the same real data. When perturbation set to a too large value, real data will be drastically perturbed.

Semantic information and quality will be changed, which can mislead discriminator to recognize degraded samples as real data incorrectly. 

4. EXPERIMENTS:

For the purpose of evaluating our method and investigating the reason behind its efficacy, we test our adversarial training method on image generation tasks on CIFAR-10, CelebA and LSUN with DCGAN and ResNet architecture. CIFAR-10 is a well-studied dataset of 32 x 32 natural images, containing 10 categories of objects and CelebA is a large-scale face attributes dataset with more than 200k images. In addition, We validate the proposed method on 3000k images labeled as bedroom in LSUN . For fast validation, We resize images in CelebA and LSUN to 64 x 64. The effectiveness of the proposed method can be proved empirically by results on these distinct datasets with two widely adopted architectures.

In section 4.1, we first conduct hyper-parameter experiments on the magnitude of perturbation for unsupervised generation task on CIFAR-10. In order to study transferability on different network structure, we conduct some extensive experiments with multiple architectures and settings in section 4.2, which also demonstrate the advantages of our method that can stabilize training and accelerate convergence in section 4.3. Lastly, we compare our adversarial symmetric training method with some latest state-of-the-art GAN variants. With the same experimental settings of some classic models, our method can produce diverse images with better fidelity.

In this paper, we use standard GAN objective function as the adversarial loss. In order to alleviate gradient vanishing at the beginning of training, we adopt non-saturating loss suggested by (Goodfellow et al., 2014) . Furthermore, we use Fréchet inception distance (FID) (Heusel et al., 2017) and inception score (Salimans et al., 2016) to measure model performance, both of which are wellstudied metric of image quality. Please refer to Appendix F for more details about implementation.

Our source code will be released on Github.

4.1. EVALUATION WITH DIFFERENT HYPER-PARAMETERS:

In order to find an appropriate magnitude of adversarial perturbation, we do unsupervised image generation experiments on CIFAR-10 with DCGAN architecture at different settings of perturbation. Due to the large searching space, we select several typical values for experiments such as {0,1,2,3,4}/255. Furthermore, we do ablation study by replacing gradient used to craft perturbation by Gaussian noise. All experiments are run three times independently to reduce randomness.

Figure 3 shows the FID results in the different hyper-parameter setting, Our method performs far better than baseline when ε lies in interval 0.5/255 ∼ 3/255. However, no matter the perturbation amplitude is too small (ε = 0.2/255) or a little large (ε = 4/255), the method improves original model slightly. In addition, when the imposed perturbation is too strong, the model performs even worse than the baseline. This is because samples perturbed drastically can mislead discriminator to recognize degraded samples as real data incorrectly, which hinders discriminator from catching true characteristic of real data distribution. With appropriate perturbation, discriminator can be regularized to balance training, which can alleviate the bothersome collapse of training loss and generate samples in standard GAN. Only when the attacking intensity is suitable, discriminator will become more robust owning to adversarial training, facilitating itself to produce more accurate and informative gradient to generator, which can promote the capability of generator tremendously. In this way, generator obtains more meaningful and reliable update information, accounting for generating better samples. Still and all, too much perturbation is harmful to discriminator when exceeding its endurance, leading discriminator to the ground defeated by generator, which is not a favorable scene. Similarly, when the imposed perturbation is too tiny, on which condition the adversarial sample of real data is almost close to the benign, the effect of adversarial training is limited. In particular, when ε is zero, the model degrades into regular GAN algorithm, the only change of which is updating discriminator twice on same real data. This setting is only slightly better than the baseline but far worse than the version with appropriate ε. Overall, results show that discriminator adversarially trained with appropriate perturbation can stabilize training and improve performance of generator. Besides, the annotation 'Gaussian' in Figure 3 means that the gradient used to craft adversarial samples is replaced by Gaussian noise, whose poor performance proves that perturbation direction of gradient is important for effectiveness. In the following sections, we use ε = 1/255 as default option for simplicity.

4.2. EVALUATION WITH DIFFERENT ARCHITECTURES:

To explore the transferability and compatibility of the proposed method, we test with widely adopted DCGAN and ResNet architecture on CIFAR-10 and CelebA. Figure 4 plots the comparison results. Additionally, two groups of contrast experiments with different setttings are performed (see Appendix E). We find that our method can largely accelerate convergence and improve the fidelity of generated samples. By applying our method, stability can also be improved vastly (Appendix A.1). Even with the setting in which baseline GAN collapse thoroughly, our model can still converge toward optimization objective.

Generally, GANs collapse when discriminator is too strong or too weak; the former situation is more common during training. WGAN and a series of its variants solve the problem theoretically by using Wasserstein distance to measure the discrepancy between two distributions with constraining the discriminator. Different from their approach, we address training instability in a practical point of view that discriminator as a classifier is not ideal and robust. Adversarial training on real samples is introduced into classific GAN to make discriminator more robust. Experimental results prove our method is effective with marginal computation cost and can be applied to different network architectures on different datasets. 

4.3. IMPROVED PERFORMANCE:

We perform image generation experiments to verify the effectiveness of our method. As shown in Table 2 , through unsupervised training on CIFAR-10, CelebA and bedroom in LSUN, our model can achieve comparable performance to state of the art on FID and inception score (Higher is better). It should be noted that the four rows at the bottom show the results of our implementation. We further validate on ImageNet (Russakovsky et al., 2015) resized to 64 x 64 with unsupervised AS-DCGAN and achieve a FID of 60.65 relative to 65.78 of baseline. Generated samples are shown in Appendix C.1.

4.4. ANALYSIS:

To show how our method works, we perform some analytical experiments on CIFAR-10. Notice that in our algorithm, the only significant change for vanilla GAN is adversarial training on real samples during the two-players game. Gradient given by discriminator is the key to update generator. Hence, we visualize the gradient as Figure 1 , which shows that the gradient of adversarially trained discriminator contains more semantic information, eg, profiles of face, but the gradient of standard discriminator looks like uninformative noise. We further show the histogram of the gradient of discriminator with respect to real samples as Figure 5a . Our method can obtain more sparse gradient The computation overhead of the proposed method is about 25% relative to the baseline, which is comparable to that of spectral normalization and much smaller than that of gradient penalty. 

A.2 ABLATION STUDY:

In addition, we visualize the convergence process through evaluation by inception score and FID. As described in Figure 7 , when updating discriminator on real samples twice (ε = 0) or giving the perturbation of a random direction ('Gaussian'), the models achieve similar performance to baseline. Only with the appropriate setting of perturbation and symmetric training scheme, desirable results can be realized. 

B ALGORITHM OF AS-GAN:

Algorithm 1 Mini-batch stochastic gradient descent training of AS-GAN. We set perturbation ε to 1/255 as default for image generation tasks.

1: for number of training iterations do 2:

Sample mini-batch of m noise samples {z i , ..., z m } from Gaussian distribution N (0, I).

3::

Sample mini-batch of m data samples {x i , ..., x m } from real data distribution P data .

4::

Update the discriminator by gradient ascent, this step can be regarded as adversarial training of discriminator on fake samples:

EQUATION

Craft adversarial samples of real samples for discriminator based on gradient:

EQUATION

6:

Perform adversarial training of discriminator on real samples:

EQUATION

Update generator by gradient descent, this step can be regarded as crafting adversarial samples of fake data: 

EQUATION

:



Figures and Tables:

Figure 2: Schematic of proposed AS-GAN. Standard GAN training is illustrated as the first forward pass and the first backward pass. In addition to standard GAN training procedure, we introduce adversarial training of discriminator on real samples, illustrated as the second forward pass and the second backward pass, which is equivalent to train discriminator with robust optimization.

Figure 3: (a) Best FID results under different settings of three independent runs (Lower is better). (b) Mean FID of last 20 epoch. With appropriate perturbation and adversarial updating policy, the quality of generated samples can go beyond baseline by a large margin.

Figure 4: Training curves of FID on CIAFR-10 (upper) and CelebA (lower) with DCGAN (left) and ResNet (right). Results show that the proposed method can accelerate convergence and achieve better FID. Meanwhile, it can stabilize training with less sensitivity to network architecture and hyper-parameter setting.

Figure 5: (a) Histogram of gradient from discriminator with respect to input images. The left is the baseline, and the right is the adversarial symmetrically trained discriminator. (b) L1-norm evolution of discriminator gradient during training. Adversarial training will prompt the gradient becoming more sparse.

Figure 6: (a) Confidence of discriminator on real data and adversarial samples of real data during training. (b) Evolution of discriminator loss at different settings.

Figure 7: (a) Training curves of inception score with different methods. (b) Training curves of FID with different methods. With the increasing of iterations, our algorithm (ε = 1/255) converges faster and better than other models.

Figure 8: (a): 32 x 32 CIFAR-10 samples generated by AS-ResNet. (b): 64 x 64 CelebA samples generated by AS-ResNet. We believe these samples are at least comparable to the best published results so far.

Figure 9: (a): Interpolation results from AS-ResNet on CIFAR-10. (b): Interpolation results from AS-ResNet on CelebA.

Figure 11: 128 x 128 results from AS-DCGAN on CelebA.

Figure 13: (a): Collapsed samples generated by standard GAN trained on CIFAR-10. (b): Samples generated by AS-ResNet trained on CIFAR-10.

Figure 15: Benign samples (on odd rows) and adversarial samples of discriminator adversarially trained on real data (on even rows). Confidence is depicted at corner. After adversarial training on real samples, discriminator is more robust to adversarial perturbation. The pertubation level is 1/255.

In a sense, adversarial training on real samples regularizes discriminator by augmenting training data, which prevents discriminator from being too strong to alleviate training collapse. The capability of two networks become more balanced with adversarial training both on real samples and fake samples. Noteworthily, the network capacity of discriminator should be larger or at least equivalent to that of generator because discriminator is further constrained with additional adversarial training. In addition, we suggest to set pertubation to zero in the begining of training in case discriminator is too weak. We do extensive experiments on how perturbation affects training in the next section.

Table content: <table/>

Inception scores and FIDs with unsupervised image generation on CIFAR-10 and CelebA.(Radford et al., 2015)(experimented by(Yang et al.,  2017)), †(Miyato et al., 2018),  ‡ (Wu et al., 2017), *(Gulrajani et al., 2017) Figure5b) as training iteration increases, which means adversarial noise in gradient is partly eliminated. What's more, the proposed method can augment training data and smooth the decision boundary of discriminator, which can alleviate mode collapse to a large extent shown as Appendix C.2. By means of adversarial training evenly on both real and fake samples, we make the training scheme symmetric and stable.

Table content: <table><tr><td>Method</td><td>Inception score CIFAR-10</td><td colspan="3">FID CIFAR-10 CelebA LSUN</td></tr><tr><td>(Standard CNN)</td><td/><td/><td/><td/></tr><tr><td>DCGAN</td><td>6.64±.14</td><td>30.9  ‡</td><td>52.0  ‡</td><td>61.1  ‡</td></tr><tr><td>WGAN-GP</td><td>6.68±.06  †</td><td>40.2  †</td><td>21.2  ‡</td><td/></tr><tr><td>SN-GAN  †</td><td>7.58±.12</td><td>25.5</td><td/><td/></tr><tr><td>WGAN-GP(ResNet)</td><td>7.86±.07  *</td><td>18.8  ‡</td><td>18.4  ‡</td><td>26.8  ‡</td></tr><tr><td>WGAN-div(ResNet)  ‡</td><td/><td>18.1</td><td>15.2</td><td>15.9</td></tr><tr><td>DCGAN</td><td>7.05±.14</td><td>28.05</td><td>20.45</td><td>25.36</td></tr><tr><td>(ours)AS-DCGAN</td><td>7.21±.02</td><td>25.50</td><td>10.90</td><td>18.08</td></tr><tr><td>ResNet</td><td>7.35±.16</td><td>22.92</td><td>25.72</td><td>175.70</td></tr><tr><td>(ours)AS-GAN(ResNet)</td><td>7.65±.15</td><td>21.84</td><td>11.71</td><td>45.96</td></tr><tr><td>and lower L1 norm (</td><td/><td/><td/><td/></tr></table>

Comparison of average training time of one epoch of different methods is shown as Table 4.4. Average training time of different methods relationship between GANs and adversarial samples has been a open question since both models emerged.In this paper, we analyze that adversarial training on fake samples is already taken into account in standard GAN training framework, but adversarial training on real samples does not exist, which can make training unbalanced and unstable. This is because gradient given by nonrobust discriminator contains more adversarial noise, which can mislead update of the generator. In order to make training scheme symmetric and discriminator robust, we introduce adversarial training on real samples. We validate the proposed method on image generation tasks on CIFAR-10, CelebA and bedroom in LSUN with varied network architectures. Experiments show that the gradient of discriminator adversarially trained both on real and fake samples contains less adversarial noise. Convergence speed and performance are improved with marginal computation overhead. Moreover, mode collapse is alleviated. With simple DCGAN network architecture and standard objective function, we can achieve comparable FID to state of the art on these datasets.

Table content: <table><tr><td>Setting</td><td colspan="4">DCGAN (ours)AS-DCGAN SN-DCGAN DCGAN-GP</td></tr><tr><td>Training time</td><td>19.83s</td><td>26.40s</td><td>24.50s</td><td>31.57s</td></tr></table>

FIDs with unsupervised image generation on CIFAR-10.Here, 'standard' means using same setting with Appendix F. From table3 and table 4, under different conditions, our model is always better than baseline.

Table content: <table><tr><td>Setting</td><td colspan="6">Standard Adam(β 2 =0.9) no BN(G) no BN(D) no BN(Both) lr D =4e-4</td></tr><tr><td>Baseline</td><td>28.05</td><td>29.14</td><td>29.40</td><td>69.82</td><td>59.15</td><td>28.44</td></tr><tr><td>AS-GAN(ours)</td><td>25.50</td><td>27.18</td><td>26.72</td><td>51.30</td><td>56.18</td><td>26.57</td></tr></table>

FIDs with unsupervised image generation on CelebA. Our models are implemented by Pytorch with acceleration of RTX 2080ti GPUs, only when evaluating inception score we use Tensorflow. Before fed into discriminator, we rescale images within [-1, 1]. Dimension of the latent vector is set to 100 for all implementations. We train models on CelebA for 100 epoch, CIFAR-10 for 200 epoch, LSUN for 10 epoch and Imagenet for 10 epoch. FID and inception score are computed from 22400 generated samples, and inception score is calculated by 10 independent partitions.

Table content: <table><tr><td>Setting</td><td colspan="2">lr D =2e-4 lr D =5e-5 lr G =2e-4 lr G =2e-4</td></tr><tr><td>Baseline</td><td>34.10</td><td>18.73</td></tr><tr><td>AS-GAN(ours)</td><td>12.14</td><td>11.82</td></tr><tr><td colspan="2">F IMPLEMENTATION AND TRAINING DETAILS</td><td/></tr></table>

Peer review (iclr2020):
This paper presents an interesting idea based on introducing adversarial noise on real samples during GAN training. This novel approach may improve GAN training and have potentially large impact, but the paper in its current form is slightly below the standard of ICLR due to its lack of clarity. While it is very interesting to apply adversarial noise in real data, this approach is not clearly motivated or explained. For example, at the beginning of page 2, why “As a consequence, training will become unstable when generated distribution approximates target distribution because the gradient given by non-robust discriminator around real samples contains more adversarial noise”? One may think that, on the contrary, the adversarial noise in generated samples would approximate that in real data when the generator distribution approximate the true data distribution. Similarly, after eq. 6, why “Consequently, gradient given by discriminator may vanish when discriminator becomes stronger than generator without capacity constrained.”? I would think the gradient would explode in this case given all the regularisation on gradients. In addition, the term “non-robust discriminator” has been used several times in important places, but is not clearly defined. Properties used to justify the approach, such as “symmetric” and “balanced” need to be explained. For example, would it be possible to illustrate or measure the imbalance of discriminator? The overhead of the algorithm needs to be stated more precisely. (after eq. 8) It is unclear to me that “backward propagation of Equation 5 with respect to x with negligible computation overhead”. Would this require back-prop through the entire discriminator, which can be very deep thus costly? It would be helpful to provide an estimation of runtime overhead supported by experiments. In algorithm 1, why is it necessary to perform discriminator update twice? How about skipping step 4 (eq. 12)? I think this may better mirror the adversarial update for generated samples, which only involves updating generator parameters once. In section 3.4, I am not sure it is similar to unrolled GAN when the perturbation is zero. Unrolled GAN required backprop through discriminator update, which I don’t think is the case here Finally, the experimental results are confusing. In Table 1, why the reproduced results are already much better than those in the original papers? A solid baseline is necessary for any further assessment. Further more, since the DCGAN and ResNet baseline models did not use recent regularisation approaches such as spectral-norm, it is hard to assess whether the proposed method can work without those existing techniques. It would be helpful to use at least a SN-GAN baseline. The analysis also need to be improved. It is hard to interpret the histograms and L1 norms in Figure 5 as related to “informative gradient” or “semantic information” as claimed in sectiion 4.4. Quantitative measurements such as correlation or mutual information may justify these claims. Overall, I think the idea presented is certainly promising, but needs further development to be qualified for acceptance. Nit: The 1-line derivation of eq. 6 can be incorporated into the main text. A transpose in eq.10 is missing. Update: I read the authors* rebuttal, which addressed many of my concerns and presented additional empirical results. The improvement over SN-GAN baseline is particularly convincing. I therefore believe the final camera-ready version can be a valuable contribution to the field, and would like to recommend accepting this paper.