Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper is about GAN training. The authors suggest that using a continuous latent distribution to model a discontinuous image distribution can lead to poor sample quality. The propose doing gradient updates directly on the latent representation to improve sample quality. They also do latent gradient updates to improve sample diversity. The authors show quantitative results on several natural image datasets, including ImageNet.
strengths_and_weaknesses: Originality:
The authors draw our attention to an interesting problem, where by a continuous the latent distribution is used to model a discontinuous data distribution and propose using gradient updates directly on the latent representations to remedy this. The also do gradient updates on the latent space to improve variation. In both cases the authors use existing losses and optimise w.r.t. the latent representation. 

Quality:
Figure 1 is helpful in motivating the problem that the authors are trying to solve.
Section 4.1: It is great that the authors consider many different variations of their model.
Impressive results in Figure 9 show how this approach can be used to "fix" poor samples. However, it would be better if we could see all the steps in the interpolation still and the cats ear is still partially missing.
Figure 4 is also great to show how the latent updates improve the image quality.
Results in Figure 8 further motivate the model. 
Table 6 shows results on a large number of datasets. Its is good to see that the results hold on ImageNet.

While the authors have some good results the experimental setup is not well described, it is not always clear which objectives are being optimised.

There is no explanation of how a model with "in-training latent sampling transform" (AdvLatGAN-qua) performs well without "post-training latent sampling transform"? Intuitively it should be worse?

Addressing sample diversity is an after-thought and does not appear to be directly related to the main motivation of the paper.

Clarity:
This paper was very poorly written and very hard to follow. I have suggestions for improvements in the box below.

Significance:
The proposed work can be used to improve the quality of samples and diversity at the cost of doing gradient updates on the latent representations during sampling (although the authors do show results only doing updates durning training). This may be a high computation cost. It is also not clear what the application of this approach is. Does it improve sample efficiency during training on another task? How does the approach affect the learned representations? 

It is also not clear to me why the problem of modeling a discontinuous data distribution is not solved by category conditional  generative models?

Strengths:
(1) The authors have very strong qualitative and quantitative results.

Weaknesses:
(1) The paper is very poorly written and very hard to follow.
(2) It is not clear what the application of this model is.

questions: Please could the authors clarify what they mean in lines 30-31 when they say that the generation quality is not a continuous function?

Line 51-52: “z* can be achieved by updating the latent z~p(z)” Can the authors please clarify that z* is a gradient update function on z? Or is it something else? Does z* have any parameters?

Should equation 2 be arg min?

Section 3.1 is confusing because it talks about both adversarial attacks and Generative Adversarial Networks which are different research areas that share the term “adversarial”. I would advise the authors to improve the clarify of this section.

How do equations (1) and (2) related to equations (3) and (4)?

Please clarify sentence on line 151-152: If all sub-manifolds M’s “are disconnected” how can the authors “require the splitting of Xr to the extent that each sub-manifold keeps connected.”

I suggest that the authors more clearly introduce the terms “latent space mining”, “adversarial sample mining” and how this is different to drawing samples from a generative model.

Proposition 3.2 is intuitive and makes sense.

Section 3.3: Are results on Table 1 computed across the whole test dataset? Results in Figure 6 do not suggest that there is an increase in sample diversity.

Results in Figure 5 are very nice and demonstrate the motivation (relating to sample quality) of the paper well. Please clarify in the caption which loss z* is being optimised for.

Line 243 “improved generation mapping G.” Does this method encourage the learning of better parameters in the generator, G? Or is it that you are able to draw better samples? Please clarify? If it is the former, how does your approach lead to improved learning of parameters for G?

Lin 254: Introduce acronyms IS and FID. For example, “Inception Score, SI [46], Fréchet Inception Distance, FID”.

Section 4.2. Which objective is being optimised for the results this section? Please reference the question in the text.

Results in Figure 9 are impressive. Are these cherry picked? Is it not possible to apply this to all steps in the interpolation from figure 3?

Line 301: Please clarify what is meant by “and does not work on unlabeled STL-10”. 

Table 6: Please double check that all values show in bold are statically significantly better that the baselines. Where this is not the case either underline the others or indicate in some other clear way.

What is the application of this method? How does it affect the learned representations? Can they be used for more data efficient learning (for example in a classification task)?
limitations: Authors have considered a single limitation.
It would also be helpful to note how their approach affects the representation quality and the computation cost of optimising z*.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the latent space in Generative Adversarial Nets (GANs) by introducing a implicit mapping on latent variable z before sending it to the generator G. Such a network z*(z) is trained by minimizing the generator loss, with G and D fixed. An additional level of optimization is introduced to optimize z* during the training of the proposed AdvLatGAN. Based on improving the quality and diversity of the model, two variants of the model are introduced, by modifying the objective of z*.
strengths_and_weaknesses: Strength:

1) The paper presents an interesting perspective of improving GANs. The idea seems to be novel.
2) Experiments show improvements on Cifar-10 and STL image generation, by addion AdvLat in different GAN framework.

Weakness:

1) The idea of using implicit distribution for generative models has been explored in a few works, including [1], it would be great to have some comparisons with these methods.
2) The idea of adding additional network to transform z seems to be something could have been done within the generator, if we think of the transform network and the generator network as a whole. I'm wondering if the improvement was actually achieved by the bi-level optimization. It would be great to see some ablation study on the effectiveness of each component of the proposed method.


[1] Fang, L., Li, C., Gao, J., Dong, W., & Chen, C. (2019). Implicit deep latent variable models for text generation. arXiv preprint arXiv:1908.11527.
questions: Please see the previous session
limitations: It would be great to see the performance of AdvLatGAN on some larger-scale image dataset.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: By introducing the adversarial techniques into GAN, this paper proposes new training and latent variable sampling methods to achieve better generation quality and diversity. The motivation is interesting that, the generator's limit as a continuous mapping: a) the contradiction between continuous mapping pixel-space and quality discontinuity; b) the mapping can incur mode collapse. Accordingly, the work adds a latent space sampling shift to amend the raw latent distribution and introduces targeted sampling shifts into GAN’s bi-level optimization for quality and diversity improvement. Extensive experiments on image data including large-scale dataset and backbone networks show that the proposed techniques lead to promising improvements.
strengths_and_weaknesses: ### Strengths:

1.This work is highly motivated and well presented. The idea is interesting.

2.The contributions are clear. This paper notes the effectiveness and importance of latent space sampling shift in GAN and may incur more follow-up studies devoted to making twofold efforts including sampling shift and mapping improvement.

3.The idea is supported by extensive evaluation with impressive results and theoretical arguments.

### Weaknesses:

1.The proposed latent sampling shift is iteratively conducted, which may incur more computational costs, especially in in-training algorithms. The trade-off between time consumption and performance gain needs to be discussed.

2.Some related works shall be discussed. The following papers mine latent space for different targets like conditional generation are worth for discussion.

[1] Nguyen A, Clune J, Bengio Y, et al. Plug & play generative networks: Conditional iterative generation of images in latent space, CVPR 2017. 

[2] Nguyen A, Dosovitskiy A, Yosinski J, et al. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks, NIPS 2016.



questions: 1.I can understand that the paper’s contribution is irrelevant to the specific choice of the backbone, and the experiments have already shown its effectiveness on various GAN backbones. Yet it is still informative to see its role on more SOTA designs e.g. StyleGAN.

2.I also understand that there exists a difference between qua (quality) and div (diversity) settings, but there is still room for synergizing the two parts. Why not try adopting qua and div at the same time? Or the authors shall better clarify this point.

3.Since many generative models other than GAN also use a single network to accomplish the generative task, do they also suffer from the same quality discontinuity issue? Can the proposed method be generalized to other generative models e.g. VAE?

limitations: Please see above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Author proposed a novel idea about training/sampling GAN from a discontinuous distribution by setting one more layer in-between known prior $P_z$ and real distribution $P_x$. Author provided mathematical proof that discontinuity of latent space (support of G) is required to cover the complex and discontinuous real distribution. It is also provided that minimizing Eq.5 is equivalent to minimizing Jenson Shanon Divergence between the latent distribution $P_z$ and a distribution of a union of disconnected subset in the latent space $P_z^{op}$. The effectiveness of the proposed idea is shown in Experiments.
strengths_and_weaknesses: $\textbf{Strengths}$
1. Proposed idea is interesting and may have a  possibility to be used to avoid from unrealistic generation from GAN.
2. Experiment result verifies the effectiveness of the proposed idea.

$\textbf{Weakness}$
1. Logical flow of the paper at Introduction part and connection between the motivation of the paper and the proposed method is not clear.
Specifically, the logical flow is like below:

  (1.1). Real distribution is a union of disconnected manifolds

  (1.2). Proof for necessity of disconnected latent space (at least in high level)

  (1.3). The gaussian space of ${G}$ is not disconnected.

  (1.4). Eq. 5 and Eq 8?

Here, (1.2) is not mentioned in Introduction, which I think necessary.
More importantly, (1.4) has a weak connection with the logical flow.

2. Mathematical proof: I wrote my thought in the Questions part.

3. Method part is written well, but Introduction part is NOT organized well. It also has some unnecessary or confusing expressions.

For example, 

at Line 50, “otherwise the generative expression will differ.” What does the generative expression stand for? Why is it necessary for justifying the perturbation-based method?

at Line 30-31, "One reason is that the generator is a continuous function (as neural nets) while the generation quality (e.g. natural images in pixel space) is not." Does it mean the generation quality is not a continuous function? How can the quality be a function?

at Line 38-40, Most generative models such as GAN, VAE, Diffusion Model and Flow-based model also inherently have a mapping between simple known distribution and complex real distribution. I believe disconnected modes of real distribution cannot be said to be "well-known difficulty in GANs training".


questions: 1. It is not understandable that if real distribution is a union of disconnected manifolds, the support of G needs to be disconnected as well.

Line 534-536: Let's say $U_1$ is Male and $U_2$ is Female. The proof goes like this; assume there exists a latent vector $z$ generating both Female and Male, i.e., $G(z) \in U_1$ and $G(z) \in U_2$. This implies $z \in U_1 \cap U_2$, which contradicts the condition 'Female and Male do not have intersection', i.e., $U_1 \cap U_2 = \emptyset$. To me, it looks like a simple repeat of the assumption and does not prove that $Z^{op}(G)$ is a union of disconnected subsets in the latent space. Please correct me If I am wrong.

And If my argument is not wrong, the entire logical flow of this paper must be revised as well.

2. The proof in Proposition B.3. includes an assumption that $E_{x \sim p_r}[log(D(x))]=E_{z \sim p_z^{op}}[log(D(G(z)))]$, which means the given generator (hence the discriminator as well) for the proof must be optimal. It needs to be mentioned. Furthermore, in practice in Alg. 1, optimizing Eq. 5 is conducted without optimal $G$ and $D$, which needs to be at least justified in the paper. Please correct me if I am wrong.

3. I am not sure, but isn't it bad in terms of diversity if $z$ is optimized by Eq.5? What is the number of real/fake samples for FID score in Table 4?
limitations: Limitation is not provided in the paper.
review_title: null

Meta-review: metareview: The paper under consideration proposes to improve sample quality in GANs by performing optimization-based adversarial mining on the latent space as a pre-processing step, using the fast gradient sign method of Goodfellow et al (2015). Optimization in the latent space is not a new area, and I was surprised to see e.g. neither Bojanowski et al, 2017's "Optimizing the Latent Space of Generative Networks" nor Azadi et al, 2018's "Discriminator rejection sampling" cited as related ideas.

Reviewers found the idea interesting and the experiments mostly convincing, the contributions clear. The motivation for this paper hinges upon a proof under certain assumptions that complex distributions require a latent prior with disjoint/disconnected support; some reviewers were unclear how this was connected to FGSM but this was cleared up in rebuttal. Several reviewers expressed concerns about the far-from-state-of-the-art GAN "backbone"/small scale of the experiments, however new experiments involving larger resolution datasets and a StyleGAN2 base model have quelled these concerns.

To the AC, the method is intriguing, appears well-motivated and well-validated, especially in light of new experiments on larger scale data. Leveraging an adversarial example discovery procedure for the purpose of improving the prior sampling distribution is a clever and non-obvious innovation, and as the authors note in their rebuttal, while other feed-forward generative networks may suffer from the same issues around being "too continuous", the training procedure used by GANs (wherein the discriminator defines a non-stationary objective function, and its gradients are used as the generator's learning signal) uniquely position them to exploit this trick of performing "surgery" on the latent prior. For all of their difficulties, GANs have a reputation for accomplishing quite a lot in terms of sample fidelity for a fixed model capacity, and ideas like the one presented herein may serve to further alleviate training challenges, as well as elucidate niches in which GANs remain useful (at a time when the preponderance of attention has shifted to diffusion models). I recommend acceptance.
confidence: Certain
Decision: Accept