Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper applies two existing techniques derived from adversarial example literature and MSGAN to improve the quality and diversity of generated samples by GAN. The former idea from [Goodellow et al., 2014b] is used to shift a latent vector z originally sampled from the Gaussian, which is different from the original adversarial attack paper that transforms the input image. The direction to move the latent vector is calculated by using I-FGSM with a standard loss for the generator proposed in the vanilla GAN paper. The experiments show that this technique can improve the quality of generated samples. Furthermore, this paper also consider how to improve the diversity of generated samples by transforming the latent vectors before putting them into the generator network. The approach for that is based on MSGAN's idea that mines hard latent vectors for the discriminator. The authors combine these two techniques and achieved better results compared to DCGAN, WGAN, WGAN-GP, and SNGAN models on several relatively small scale datasets such as CIFAR-10, STL-10, etc.
main_review: This paper introduces their motivation convincingly with some easily comprehensible figures. However, the experiments to show the effectiveness of the proposed method are basically performed only with small scale datasets, so that it makes difficult to figure out how complex manifold in the target space can be handled by this approach, in particular, it is unclear how it can scale to the ImageNet dataset. The discussion to compare the proposed method with other work seems to be not enough because there are some more research that shed light on the importance of latent vector transformation. For example, the series of StyleGAN work is continuously improving the performance by precisely analyzing the relationship between the latent space and the pixel space. They also firstly adopted to transform the latent vector z to another latent space W using a mapping network. That should be discussed as one of the possible approach to transform the latent vector, i.e., using a trainable function instead of I-FGSM. 

Some figures to show the effectiveness of the proposed method seems difficult to interpret as the captions claim, e.g., Figure 5. It was difficult to find the clear difference between the results of "Anti-diverse" and "Original", and the "Diverse" results look worse than other two in terms of the quality (if it is intended because the figure tries to show the increase of diversity, but it still unclear that the "Diverse" row in the Figure 5 has higher diversity compared to other two.)
summary_of_the_review: The paper is well organized and easy to follow the authors' claims but the qualitative comparison results seems difficult to be interpret as the captions state. And from the perspective of transformation of the latent vector z, there can be more discussion in the comparison with other methods such as StyleGAN that transform z before putting it into the generator.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper looks at the problem of improving the generative quality of GANs. The paper makes improvement along two dimensions: a) The paper adjust the samples distribution of GANs using adversarial attacks (I-FGSM), and thus effectively samples from a potentially multi-modal distribution. b) The paper improves the diversity of generated samples using using adversarial attacks on the mode-seeking objective of Mao et al 2019.
main_review:  + The paper is generally well written and quite easy to understand. It does however mix preliminary work (I-FGSM) with the proposed contribution in 3.2.
 + The paper is able to improve vanilla GAN models using the presented objectives.
 - The paper does fully compare to prior work. Yes Table 8 highlights some of the similarities to prior work, but the main evaluation does not compare to alternative approaches that consider the interplay between GANs and adversarial attacks or latent space exploration. In order to see the efficacy of the presented method, the paper should experimentally compare to the majority of methods highlighted in Appendix C. The paper would be a lot stronger if it could show that the design choices made here are better than other attacks (i.e. perturbations on the generated or real images, instead of latent features, e.t.c.)
- The visual quality of the presented examples is somewhat underwhelming. Does the presented method work on larger GAN architectures such as StyleGAN2 or BigGAN? Does the presented method actually address the issues highlighted in Fig 3?
summary_of_the_review: The paper explores an interesting idea of adding adversarial robustness into GAN training to improve latent distribution sampling and diversification. Unfortunately, the paper falls a bit short in the experimental validation of the approach, and comparison to prior approaches.

---

Post rebuttal. The rebuttal makes a good case for their final algorithm using additional results. However, I still do not see what the paper adds on top of baselines, or how the problem setup in Figure 3 (interpolation artifacts) is actually addressed. The rebuttal mentions some experimental evidence that seems to indicate latent-space sampling can helps. However, I would need to see these results in an actual paper submission for review to feel comfortable about accepting it. As is the paper seems interesting, but not ready for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposed a sample shifting method in GAN, formulated as adding intermediate latent space to generated pixel space. Such method is based on observation of continuous mapping limit: image quality in pixel space is not as continuous as latent space; limited latent space will incur mode collapse thus poor image diversity. The main contributions are: a new optimization problem as sampling method to improve image generation by quality and diversity, propose to use I-FGSM optimization method to achieve this sampling optimization problem. The experiment showed improvement on public dataset of STL-10, CIFAR-10.
main_review: Pros:
1. The paper is well organized, with clear sub-titles and clear logic flow;
2. Ablation study on various baseline GAN architecture (DCGAN, WGAN, WGAN-GP, SNGAN) is conducted to show generalizability of such sampling method.

Cons:
1. Compare with baseline method MSGAN: (1) why it only compare the div+ with MSGAN; (2) improvement over baseline method MSGAN is very limited.
2. If generator trained with better quality regularization, the latent space after mapping should have better continuity? Comparison like Fig 3 after training would be needed to prove that.

Some minor issues:
1. The paper is a bit redundant on algorithms and figures. For example,  it seems lengthy to include both algorithm 1 and 2 in the main paper
2. Fig 1. It’s not obvious what the latent space did in this figure

summary_of_the_review: The paper is overall well written, and the idea is very clear and elegant. Meanwhile, I have some doubt on the sufficiency of experiment to show the improvement from quality. Also the improvement over baseline method MSGAN seems minor for me. Expect the rebuttal to clear my doubt.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors demonstrate that the generator in a GAN is a continuous function two latent codes that are close in the latent space are mapped to two images that are close in the pixel space. However, the quality of the generated images is not preserved as quality is not a continuous function in pixel space. To address this issue, the authors propose to transform the original latent codes and demonstrate that it results in better generation quality and diversity.
main_review: 1.    Related works have not been cited. For example, the following paper performs a similar optimization technique with a different objective.

a.       Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, Timothy Lillicrap. LOGAN: Latent Optimisation for Generative Adversarial Networks. (https://arxiv.org/pdf/1912.00953.pdf)

b. https://arxiv.org/abs/2005.02435

c. https://arxiv.org/abs/1809.03627

Please do a thorough literature review to incorporate any such missing work.

2.       The authors propose five methods viz (i) AdvLatGAN-z, (ii) AdvLatGAN-qua, (iii) AdvLatGAN-qua+, (iv) AdvLatGAN-div, and (v) AdvLatGAN-div+. 

Each method either focuses on improving quality or aims to enhance diversity. 

It would be interesting to see what happens when the proposed objectives in equation (5) and equation (8) are combined. Does the hybrid method outperform both AdvLatGAN-qua+ and AdvLatGAN-div+?

3.   Another limitation of the work is that they compute FID and JSD. FID, although a widely used metric, is not able to quantify the quality and diversity as it is a unidimensional score. Therefore, it would be nice to quantitatively verify the claims of enhancement in quality and diversity in AdvLatGAN-qua+ and AdvLatGAN-div+ respectively. Comparing other metrics such as precision/recall or density/coverage will be more meaningful towards such goals.

a.  Density/Coverage: Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, 2020. (https://arxiv.org/abs/2002.09797)


b.   Precision/Recall: Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, Sylvain Gelly. Assessing Generative Models via Precision and Recall. In NIPS, 2018. (https://arxiv.org/abs/1806.00035)

4. The number of baselines used in the experiment section is too less for the CIFAR-10 case?

5. It would be good have experiments on large-scale datasets such as FFHQ.

6. I wonder if FGSM is the only method this can be applied with or any other can be used as well? If so, how does the method depend on the adversarial training method employed?

7.  I am not sure why MSGAN was chosen as the baseline for regularization?

8. There is no theoretical justification on why should the proposed method work? There is some empirical evidence but there would be better to have some theoretical backing on why this method should aid in avoiding mode-collapse.

summary_of_the_review: Please refer to the above comments.
review_title: null

Meta-review: To improve the generative adversarial nets, the paper proposes to add an implicit transformation of the Gaussian latent variables before the top-down generator. To further obtain better generations with respect to quality and diversity, this paper introduces targeted latent transforms into a bi-level optimization of GAN.  Experiments are conducted to verify the effectiveness of the proposed method. The paper is highly motivated and well-written, but the experiment part still needs to be strengthened because the goal of the paper is to improve the GAN training, comprehensive and thorough evaluation of the proposed method is necessary. 

After the first round of review, in addition to the clarification issue and missing reference issue, two reviewers point out that the method is only tested in small-scale datasets, and suggest authors evaluate the performance of the proposed method in more complex datasets. Two reviewers point out that the experimental validation and comparison to prior approaches are insufficient. During the rebuttal, the authors provide extra experiment results to partially address some issues.  However, most of the major concerns from other reviewers, such as (i) how are the performance of the method in large scale datasets that have complex latent space manifolds, (ii) non-convincing performance gain, and unclear problem setup, still remain. After an internal discussion, AC agrees with all reviewers that the current paper is not ready for publication, thus recommending rejecting the paper. AC urges the authors to improve their paper by taking into account all the suggestions provided by the reviewers, and then resubmit it to the next venue.
Decision: Reject