Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper tries to propose a new method to stabilize the training procedure of GAN. To this end, they use adversarial samples of real data to train the discriminator, and claim that it is helpful to reduce the adversarial noise contained in the gradient. Although training GAN with adversarial samples of discriminator is somewhat novel, the method and experiments are not convincing. 
I do not recommend the acceptance based on the limited contribution of this paper. The following is a detailed evaluation. 

1. The paper uses vague description such as “This approach can improve the robustness of discriminator and reduce adversarial noise contained in gradient” without convincing justification. Please give a formal description or notation of “adversarial noise contained in gradient”, and discuss how to remove the effect of “adversarial noise” in principle instead of extensively testing adversarial training of discriminator. 

2. The experiment is not convincing and the improvement is not significant. The author running adversarial training on CIFAR10 dataset with FGSM and the perturbation is tested from 0.2/255 ~ 4/255. The performance (FID score) is a bit sensitive to the amount of perturbation level. Moreover, this The improvement over DCGAN is quite limited given previous works such as WGAN-GP. Combined together, the result is not convincing (it seems to be a heavy tuning result rather than a principled solution).
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents an interesting idea based on introducing adversarial noise on real samples during GAN training. This novel approach may improve GAN training and have potentially large impact, but the paper in its current form is slightly below the standard of ICLR due to its lack of clarity.

While it is very interesting to apply adversarial noise in real data, this approach is not clearly motivated or explained. For example, at the beginning of page 2, why “As a consequence, training will become unstable when generated distribution approximates target distribution because the gradient given by non-robust discriminator around real samples contains more adversarial noise”? One may think that, on the contrary, the adversarial noise in generated samples would approximate that in real data when the generator distribution approximate the true data distribution. Similarly, after eq. 6, why “Consequently, gradient given by discriminator may vanish when discriminator becomes stronger than generator without capacity constrained.”? I would think the gradient would explode in this case given all the regularisation on gradients.

In addition, the term “non-robust discriminator” has been used several times in important places, but is not clearly defined. Properties used to justify the approach, such as “symmetric” and “balanced” need to be explained. For example, would it be possible to illustrate or measure the imbalance of discriminator?

The overhead of the algorithm needs to be stated more precisely. (after eq. 8) It is unclear to me that “backward propagation of Equation 5 with respect to x with negligible computation overhead”. Would this require back-prop through the entire discriminator, which can be very deep thus costly? It would be helpful to provide an estimation of runtime overhead supported by experiments.

In algorithm 1, why is it necessary to perform discriminator update twice? How about skipping step 4 (eq. 12)? I think this may better mirror the adversarial update for generated samples, which only involves updating generator parameters once.

In section 3.4, I am not sure it is similar to unrolled GAN when the perturbation is zero. Unrolled GAN required backprop through discriminator update, which I don’t think is the case here

Finally, the experimental results are confusing. In Table 1, why the reproduced results are already much better than those in the original papers? A solid baseline is necessary for any further assessment. Further more, since the DCGAN and ResNet baseline models did not use recent regularisation approaches such as spectral-norm, it is hard to assess whether the proposed method can work without those existing techniques. It would be helpful to use at least a SN-GAN baseline.

The analysis also need to be improved. It is hard to interpret the histograms and L1 norms in Figure 5 as related to “informative gradient” or “semantic information” as claimed in sectiion 4.4. Quantitative measurements such as correlation or mutual information may justify these claims.

Overall, I think the idea presented is certainly promising, but needs further development to be qualified for acceptance.

Nit:

The 1-line derivation of eq. 6 can be incorporated into the main text.

A transpose in eq.10 is missing.

Update:

I read the authors' rebuttal, which addressed many of my concerns and presented additional empirical results. The improvement over SN-GAN baseline is particularly convincing. I therefore believe the final camera-ready version can be a valuable contribution to the field, and would like to recommend accepting this paper.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes a new GAN training that additionally feeds adversarial examples as the real samples to the discriminator D. A key motivation here is to regularize the target real distribution simulated by D to be robust to adversarial perturbations. Experimental results show that the proposed GAN training generally improves the generation performance from the vanilla GAN training in CIFAR-10, CelebA and LSUN datasets. 

In overall, I liked its clear motivation and the simplicity of the method. Experimental results are also presented clearly, and shows a significant improvement. One of my main concerns, however, is that robustifying D in GAN training is not a new idea for some readers [1], so they need more clarification on the novelty of the proposed method, e.g. by discussing about it in related work or by comparing the performance. 

- In regarding robust optimization, I think [2] could cover a lot of practices considered in this paper. Some questions listed here are relevant to this point:
    (a) The paper only considers to use FGSM in adversarial training part, but FGSM training on large epsilon usually leads to overfitting [2] on CIFAR-10. I wonder if the authors have tried PGD counterpart in their method.
    (b) It seems that the method uses both natural and adversarial examples in adversarial training, as in [3]. Instead, there is another (and perhaps more common) type of adversarial training [2] that uses only adversarial examples for the training. What happens if this training is applied to the method?

- The method makes an additional parameter updates (Ep. 3 and 8) for adversarial training. Could this step make additional gain to the method by training more, i.e. perhaps it is a bit unfair to the vanilla training?

- It would strengthen the claim if the paper could present whether the robustness of D is indeed increased, e.g. by comparing adversarial accuracies?

- Table 1: I feel there should be more discussion about why the reproduced values are so different compared to that of previously reported values, as they might confuse the readers to convince the claimed results.

- Eq. 7: The first term in the right hand side have to be E[max (log D(x - d))] instead of E[(log D(x - d))]?

[1] Liu, X., & Hsieh, C. J. (2019). Rob-gan: Generator, discriminator, and adversarial attacker. CVPR 2019.
[2] Madry, A. et al. (2017). Towards deep learning models resistant to adversarial attacks. ICLR 2018.
[3] Goodfellow, I. et al. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
review_title: Official Blind Review #3

Meta-review: This paper proposes incorporating adversarial training on real images to improve the stability of GAN training. The key idea relies on the observation that GAN training already implicitly does a form of adversarial training on the generated images and so this work proposes adding adversarial training on real images as well. In practice, adversarial training on real images is performed using FGSM and experiments are conducted on CelebA, CiFAR10, and LSUN reporting using standard generative metrics like FID.

Initially all reviewers were in agreement that this work should not be accepted. However, in response to the discussion with the authors Reviewer 2 updated their score from weak reject to weak accept. The other reviewers recommendation remained unchanged. The core concerns of reviewers 3 and 1 is limited technical contribution and unconvincing experimental evidence. In particular, concerns were raised about the overlap with [1] from CVPR 2019. The authors argue that their work is different due to the focus on the unsupervised setting, however, this application distinction is minor and doesn’t result in any major algorithmic changes. With respect to experiments, the authors do provide performance across multiple datasets and architectures which is encouraging, however, to distinguish this work it would have been helpful to provide further study and analysis into the aspects unique to this work -- such as the settings and type of adversarial attack (as mentioned by R3) and stability across GAN variants. 

After considering all reviewer and author comments, the AC does not recommend this work for publication in its current form and recommends the authors consider both additional experiments and text description to clarify and solidify their contributions over prior work.

[1] Liu, X., & Hsieh, C. J. (2019). Rob-gan: Generator, discriminator, and adversarial attacker. CVPR 2019.

Decision: Reject