Improving Generative Adversarial Networks via Adversarial Learning in Latent Space
Authors: Yang Li
Yichuan Mo
Liangliang Shi
Junchi Yan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generative Adversarial Networks
Adversarial Learning
Latent Space

Abstract: For Generative Adversarial Networks which map a latent distribution to the target distribution, in this paper, we study how the sampling in latent space can affect the generation performance, especially for images. We observe that, as the neural generator is a continuous function, two close samples in latent space would be mapped into two nearby images, while their quality can differ much as the quality generally does not exhibit a continuous nature in pixel space. From such a continuous mapping function perspective, it is also possible that two distant latent samples can be mapped into two close images (if not exactly the same). In particular, if the latent samples are mapped in aggregation into a single mode, mode collapse occurs. Accordingly, we propose adding an implicit latent transform before the mapping function to improve latent $z$ from its initial distribution, e.g., Gaussian. This is achieved using well-developed adversarial sample mining techniques, e.g. iterative fast gradient sign method (I-FGSM). We further propose new GAN training pipelines to obtain better generative mappings w.r.t quality and diversity by introducing targeted latent transforms into the bi-level optimization of GAN. Experimental results on visual data show that our method can effectively achieve improvement in both quality and diversity.

Improving Generative Adversarial Networks via Adversarial Learning in Latent Space
Yang Li', Yichuan Mo 12 Liangliang Shi Junchi Yan
DDpartment of Computer Science and Engineering,
MoE Key Lab of Artificial Intelligence. Shanghai Jiao Tong University Key Lab. of Machine Perception (MoE), School of Intelligence Science and Technology. Peking University (yanglily, shiliangliang
tu. edu. cn mo6666666stu, pku. edu. cn
Abstract
For Generative Adversarial Networks which map latent distribution to the target distribution, in this paper, we study how the sampling latent space can affect the generation performance, especially for images. We observe that, the neu- ral generator is a continuous function, two close samples in latent space would be mapped into two nearby images, while their quality can differ much the quality generally does not exhibit a continuous nature pixel space. From such continuous mapping function perspective, it is also possible that distant latent samples can be mapped into two close images (if not exactly the same). In particular, if the latent samples are mapped in aggregation into single mode, mode collapse occurs. Accordingly, we propose adding an implicit latent trans- form before the mapping function to improve latent from its initial distribution, e.g.. Gaussian. This is achieved using well well-developed adversarial sample mining techniques, c.g. iterative fast gradient sign method (I-FGSM). We further propose new GAN training pipelines to obtain better generative mappings w.r.t quality and diversity by introducing targeted latent transforms into the bi-level optimization of GAN. Experimental results on visual data show that our method can effectively achieve improvement in both quality and diversity. The implementation publicly available at hhttps://gite com/yangco- 1e/ AdAvvLaGGAN
1 Introduction
Generative Adversarial Networks (GANs) 1 have shown effectiveness for generating high-fidelity data, especially for images under various settings 2
5 6). Based on the zero-sum game, the model is trained by the adversarial process between the generator and the discriminator. Many efforts have been made to achieve a more realistic generation from different perspectives. For instance, [ aim to design better objective functions. ImprovedGAN [10], AC-GAN ITI propose practical training techniques. While more complex network structures are studied in BigGAN 12], StyleGAN [3]. SAGAN 14
WGAN (7), SNGAN ( LSGAN
Despite the above progress, relatively less studied question is how the latent space affects the generation quality and diversity, which can be orthogonal to the above frequently studied factors like model architectures and training techniques. In this paper, we argue that the latent samples from standard continuous distribution (e.g. Gaussian) can often be mapped to varying-qualing samples for the generation. One reason that the generator continuous function (as neural nets)
Junchi Yan is the correspondence author who is also with Shanghai Al Laboratory.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Piel
[e]
Latintt Space
Quality
(a) Quality Discontinuity
(b) Our Sampling Transform
Figure 1: Red dashed lines: real data manifolds and their preimages; yellow circles/squares: samples for sound gener- ation; black circles/squares: samples for bad generation; orange dashed lines: the generator mapping; blue dashed lines: & trans- form; blue shades and solid blue lines: latent distribution and generated distribution. Due to the continuity of the network- based generator, close samples in latent space will be mapped to close images in pixel space, while the quality can vary. Cat (a) and Cat (b) are close in pixel space, but Cat (a) loses its right car. Our method transforms bad latent towards the preimage of the nearest real manifold, thus avoiding terrible generation.
Figure 2: The generated distri- butions mapped from fixed la- tent distribution reflect the map- ping quality (same legends as in Fig. D Left: bad generation quality the generated distribu- tion does not match the real dis- tribution well. Right: bad gen- eration diversity as the mapping misses modes.
while the generation quality reflected by the matching degree to the natural image distribution in pixel space does not exhibit continuous nature as two nearby images' quality can differ much. has been shown in GAN literature [15 that image objects lie upon multiple disjoint manifolds. A continuous mapping from a continuous latent distribution, e.g.. Gaussian, may generate invalid samples the between of the manifolds when the generator seeks to cover all the meaningful modes. See Fig. [ (a) for illustration. This fact also implies a dilemma: for complex real-world data, there a trade-off between quality and diversity when the latent vectors are sampled from a continuous uni-mode distribution. On the other hand, to effectively capture the natural image distribution through a continuous generative mapping, is necessary to maintain a disconnected latent space support.
The above contradiction between the discontinuity in generation data quality and the continuity of latent distribution as well as mapping function, can be one of the significant factors leading to the well-known training difficulty/instabiiity of GANs, especially for generators, as one unreasonably hopes to enforce discontinuous mapping via continuous neural nets. On the other hand, the mapping is also critical to the performance of functions with continuous nature, as badly trained mapping can lead to poorly fitted generated distribution. Meanwhile, by realizing that the mapping function allows for many- many-to-one mapping, mode collapse [16] can naturally happen when different latent samples are mapped into few or even a single mode in the target space. See Fig. 2 for the schematic diagram.
To tackle these issues, we impose an extra (implicit) transform function =-(-) on the raw sampling z, and then the generation can be written by G(=*(z)) which requires us to make twofold efforts on both G(-) and 2" (.). From this perspective, most existing works employ 2 (.) as an identity function, while we try to find a more effective one. We model the z transform process as making perturbations to the original sampling since z" (z) shall not depart much from as we hope the main content of the generated image remains the same. The hope that small perturbations can achieve considerable positive quality variation leads us to the adversarial sample mining methods. Specifically, the implicit function based transform 2" 2-) can be achieved by updating the raw latent distribution samples z to minimize lossg i.c. log(1 1 (G()))) using adversarial sample mining techniques e.g. iterative fast gradient sign method (I-FGSM) [77. and we name the method as AdvLaiGAN-z. See Fig. T (b) for the schematic diagram. We will show that our transform (perturbations) can deliver latent distribution better fitting the real distribution with the fixed G.
On top of our first effort, we propose a training strategy to improve the generative mapping G respectively for quality and diversity. This achieved by introducing z optimization (i.e., targeted implicit transforms on z) into GAN training, specifically, updating raw latent distribution samples z using I-FGSM to find latent variables which benefit the optimization to calculate the loss during GAN training. Respectively for quality and diversity, we use two iterative updating strategies for

---

Figure 3: Generation examples of StyleGAN2-ada [8] Sampling equidistantly from p(z) in latent space, the samples from the generated distribution pixel space are shown here. The beginning and ending images of each row locate in different manifolds. The intermediate results are sampled in the between of disjoint manifolds. The discontinuity causes weird generation (see yellow boxes top: missed ear and crooked head; middle: white spots on the nose; bottom: two bodies share one head).
z under different objectives, and we name the two training algorithms as AdvLaiGAN-qua and AdvLaIGAN-div The highlights of this paper include:
1) We rethink the role of the generator as a continuous function, which may ineur quality discontinuity in target space when the latent samples are directly sampled from continuous distribution, together with poorly fitted generated distribution and mode aggregation (collapse) when the function suffers bad properties w.r.t quality or diversity. We propose to formulate the mapping from G(z) to G(z* (z)). provides new design guidance to GAN which requires efforts on both G(-) and = 2-).
2) By introducing adversarial techniques into GAN, we propose sampling transform pipeline for (+) and training pipelines for improving G(-). The generation results are obtained by first training generative mappings by our proposed training pipeline AdyLatGAN-quaadii Then the transform (-)) namely AdvLatGAN-z is enforced and computed implicitly to obtain improved distribution for a more realistic generation. The training algorithms AdyLatGAN qua and -div involve latent sample transform under different objective functions for quality and diversity.
3) Experimental results on synthetic data and natural images including large-scale datasets e.g. ImageNet show the notable improvements of our approach both generation quality and diversity, whereby the performance gain is basically orthogonal to those from existing GAN techniques.
2 Related Work
Our latent vector transform pipeline mines effective latent samples and the resulting perturbations relate to adversarial methods. The proposed GAN training pipelines include the aforementioned transform network training, which can be viewed as mining adversarial samples in training. namely adversarial training. Hence we briefly review existing works on latent space mining for GANs that modify latent space samples to achieve specific targets e.g. latent space sampling improvement for better generation and adversarial learning. Also, techniques for avoiding mode collapse are studied.
Latent space mining in GANs. Techniques have been devised for latent space mining for GANs. By utilizing the steerability of GANs in latent space, StyleGAN [33 introduces a latent variable shift to better decouple the various styles by using linear network (which is still continuous). 19
20 and (1] introduce the latent space mining to training to achieve clustering objectives or improve the dynamic of training. [22 and (23 achieve conditional generation by performing gradient ascent in latent space to maximize classification neuron activations. Some recent works focus on improving latent sampling quality obtain high-quality images. Interactive evolutionary computation is used to make the generation process more controllable in [24). which meanwhile
also brings additional manual efforts. To reduce manual intervention, [25] 26 use Koncept512
27 as a criterion to improve the generation quality. However, as discussed in 25], this strategy is somehow biased and can not work well for each dataset (e.g., Pokemons). The reason that quality estimator is not even designated to access the real-data distribution information. Instead, the generation is guided by a predefined score function. Exploring the real distribution's 's guidance from the discriminator, DOT 28 utilizes optimal transport in latent space to improve the initial sampled
3

---

distribution, DRS [29 proposes a rejection sampling scheme to to improve sampling quality, and
DDLS 30 develops discriminator driven Langevin dynamics to obtain high-quality samples.
Mitigating mode collapse for GANs. Efforts are made to mitigate mode collapse, which are based on the thinking that real-world samples lie in disjoint manifolds [15]. One way is introduce collection of generators such that each generator may cover a specific mode since single generator can hardly fulfill discontinuous function [15]. However, more works focus on improving mapping
properties, explicitly obtaining a mapping covering more modes [31 32 33 Note that the generated distribution obtained by this method will contain invalid parts due to the continuity of the latent distribution and the mapping. This calls for the selective sampling of the continuous latent distribution or directly amending the latent distribution, which is rarely noticed in these works.
Adversarial samples and adversarial training. Adversarial samples are manipulated samples crafted by adding indistinguishable perturbations cause significant deep nets' output variation e.g. wrong classification prediction. Adversarial sample mining methods investigate how to manipulate such samples, which are known adversarial attacks. Since (34), various adversarial attack methods have been devised [35) 17, 36). For defense, adversarial training is meanwhile developed [35] 37 38)
In particular, recent studies have shown that adversarial training can bring many benefits to GANs, not just improving robustness. RGAN (39] shows that adversarial training can enhance the generalization of the generator and discriminator. Based on theoretical analysis and empirical results,
40
show that updating the discriminator's 's parameters with the loss calculated by ad adversarial disturbed image samples can stabilize training and accelerate convergence.
In this paper, we use an off-the-shelf adversarial sample mining method iterative fast gradient sign method (I-FGSM) [17], perform latent transform and mine effective samples that benefit GAN training in its bi-level optimization process. More adversarial methods e.g. PGD [36] and MI-FGSM 43) are also tried and discussed in Appendix D. The pre-existing works mentioned above that interplay between GANs and adversarial learning make perturbations on raw pixel space (real samples or generated samples), while we instead work on mining the latent space and introducing perturbations on latent samples, which we show is novel and effective way to improve the generation.
3 Proposed Method
This section presents our twofold effort to lift the final generative performance: AdvLatGAN-z to achieve the targeted transform 2" (+): training algorithms AdvLaiGAN qua and AdvLaiGAN-div to train generation mapping G with better generation quality (i.e., -qua) and diversity (i.e., -div). Both methods are derived by introducing the iterative updating in latent space.
3.1 Preliminaries
This section introduces adversarial attack/defense techniques and the regularization technique devised in MSGAN (32) for mode secking. Adversarial techniques include: i) adversarial samples which we introduce in the latent space sampling to achieve 2 2²))) ii) adversarial training which we introduce in the bi-le optimization of GAN to develop new training algorithms for a more powerful G(-).
Adversarial samples and adversarial training. Adversarial training [6 44 aims to defend against adversarial attacks. It uses adversarial samples to calculate the loss. In classification, for each image-label pair (x,; yi) from the labeled training set, the adversarial samples x' (34 is defined as:
arg max Ixfo(x), y:)
(1)
||x Se
Here c is the radius of the closed ball to constrain perturbation. While fo and I denote the attack target (a neural net) and the corresponding loss (e.g. cross-entropy loss). p denotes the norm dimension. Given in total n adversarial samples, adversarial training is fulfilled by the optimization:
= arg min (()()(),
(2)
Mode coverage by regularizing distance of generated samples. MSGAN 32 mitigates mode collapse for conditional generation by adding regularizer over randomly sampled z and Zy:
Lys d, (G(e, 21), G(c,22))
(3)
de (Z1, 22)

---

Figure 4: Visualization results of AdvLatGAA-z on STL-100. The left column shows the generations of raw Gaussian samples, while samples from left to right present the process of AdvLat
where è is the condition vector for generation, and d, and dg are the distance metrics in the target (image) space and latent space, respectively. It encourages the distance to be maintained or enlarged the target space to avoid the generated samples being trapped into an aggregation.
By integrating the regularization term with the original objective of eGANs 45 the final minimiza- tion objective in MSGAN for training G and D is:
Cors + AmsLma, Lori = Be,, log Dcc x)] + B (log(1 [cc, G(c, z)))
(4)
where Cori denotes the original objective, x denotes real data, and Ams is the weight hyper-paramenn Since Eq. 3 is to be maximized hence one can interpret Xme as negative coefficient.
3.2 Latent Adversarial Mining for Generation Quality
3.2.1
AdvLatGAN-z: Latent Space Transform
Note that as the real-data space is union of disconnected manifolds [15] via continuous generative mapping G, the optimal subset in the latent space w.r.t the fixed G is in no way connected. We denote the real-data subset in the high-dimensional pixel space X as X,, C X, and each disjoint submanifold as M,. Then we obtain X, - Mj where M1 to Mn, are disconnected to each other. Here we require the splitting of X, to the extent that each submanifold is itself connected set.
Definition 3.1. Suppose GI-] P(2) - P(X) is the image-of-set function corresponding to the generative mapping function G(-) Z + X, where Z and denote subsets in latent space and pixel space, and (--)) denotes the power set. Then (G) is the optimal subset corresponding to G in latent space if GIZoP(G))
Proposition 3.2. For any given continuous GI-] whose codomain includes multiple disconnected manifolds, (G) is union of disconnected subsets the latent space.
The proof is given in Appendix B. Proposition 3.2
implies that with the fixed continuous G, fitting the real-data manifolds requires disconnected latent space support. Below we show how to implicitly transform the latent raw samples from Gaussian (or other forms e.g. uniform) to a new distribution closer to 20P (G' via the idea of mining adversarial samples introduced above.
We utilize the discriminator's output to quantify the generation quality, which trained to capture the difference between the generated and real distribution. Considering that small perturbations can cause considerable quality variation (technically D's output) as shown in Fig. and Fig. B we naturally resort to adversarial perturbation babbbbbbttioo-beeddd methods to achieve the transform. Guided by Eq. we first deliver the constrained optimization below, where both G and D are post-trained.
z (zo)= = arg min log - D(G(z)))
(5)
= (x)d(2o.=){c]
where zo denotes the original latent vector, z" (zo) denotes the newly computed z, d(...) denotes the distance which loo this paper, in line with the popular protocol in literature [35
The above target can be readily solved by the well-developed tools in adversarial sample mining.
e.g. the classic iterative fast gradient sign method (I-FGSM) (17) formulated below in Eq. 6 In Appendix D, we also present comparison of more adversarial techniques e.g. PGD 36. MI- FGSM [43), that can be seamlessly incorporated by framework, and we show the simple I-FGSM
outperforms. We conjecture the reason may be due to the simplicity of the latent space in our case.
Zi+1 < Z, - E sgn log(1 D(G(z)))
(6)
From the distribution perspective, the transform is given by: px= arg min Ezrp. [log(1 D(G(z)))
D(po
5

---

where pl denotes the original latent distribution, P2 denotes the newly searched one. D(p,, PP) S e means that there is a limit to the distribution variation. A newly mined z can be considered sample from pl if the sampling is within the perturbation circle. Fig. 4 shows the results by the pre-trained SNGAN [ (i.e. the given G and D) on STL-110 [46]
We then show that Eq. 7 guarantees the correct optimization direction that optimizes the source distribution Pa towards the optimal probability distribution in latent space according to the current fixed G. i.e. p (G) defined in Definition. 3.3 The proof for Proposition 3.4 is given in Appendix B.
Definition 3.3. Given G. (G) is the optimal probability distribution in latent space corresponding to (G) when (G) satisfies that if z ~ pTP(G) then G(z) ~ Pr- Here p, denotes the real-data
probability distribution in pixel space.
Proposition 3.4. Optimizing GAN's training criterion ie. ming max Pe [oo((1 - D((G())) Ex-p, [log(((x)) [(x)) is to minimize (C)
Prop. 3.4 implies that the GAN's s optimization is to minimize the difference between p2r and p!.
Note that for optimizing Pz, the criterion merely involves Ez-p, [log(1 - ((G()))) i.e. the objective in Eq. 7 Given optimized fixed and D (and p." is thereby fixed), leaving P= the only trainable parameter, optimizing the training criterion with P. in the neighborhood of p is driving P: closer to p.P. Note that P= in the neighborhood of p is necessary otherwise the optimized criterion will depart much from JSD, and we achieve this condition by enforcing D(p,, p²) S e in Eq. 7
3.2.2 AdvLatGAN-qua: Quality Targeted Training Pipeline
Considering the pairwise approach for adversarial sample mining, i.c. adversarial training fulfilled by 2 we hope to mine samples that benefit the optimization in GAN training. which is achieved by utilizing the aforementioned latent space transform. Conducting the transform in training to obtain the new (probably disconnected) latent distribution for each iteration can compensate for the difficulty for generators align the pace of generators and discriminators, as it widely recognized that is much more challenging to train the generator than the discriminator, for its inherent defect that it cannot map a continuous distribution to a disconnected one with multiple modes.
Specifically, the transform is conducted to the raw Gaussian sampling in D updating iterations while the rest of the training algorithm remains consistent with vanilla GAN (I] and other variants with the orthogonal efforts to latent space mining. We present the algorithm in Alg. I in the appendix which we call Adversarial Latent GAN with a post meaning quality (AdvLaiGAN-quaa) The latent transform is first computed by I-FGSM to mine the targeted samples in latent space, then 1 and G are updated. The iteration continues until enough steps or the convergence of G.
The above latent transform model does not consider how to avoid mode collapse, as each time only one latent sample is considered for optimization in isolation. similar constrained optimization formulation can be devised to improve generation diversity, as shown in the following subsection.
3.3 AdvLatGAN-div Latent Adversarial Mining for Diversity
In this section, we tend to modify the regularizer for mode seeking in MSGAN (32 by substituting the randomly selected regularized z pairs with hard sample pairs that are more inclined to collapse, lifting the efficiency for avoiding mode collapse. We first show how to search for such hard sample pairs. Given a sample Zu from the raw latent distribution, we aim to search its paired sample z to form a hard sample pair, in the sense that their generations are close in the target space given G. We consider two aspects. First, as hard samples, the paired samples shall still belong to the same category which can be modeled with conditional GAN. Second, their distance in the latent space shall also be not too close, otherwise they may be aggregated in the target space due to continuous mapping. In other words, hard samples causing mode collapse shall be those close in target space while apart in latent space, when G is given. Hence, guided by Eq. |
we design our search procedure as the form:
d, (G 20). G(c, z))
z (Zz) = arg min ze (x)
(8)
dz(2).z) z)

---

P2 STEP-30
P2 STOP
STOP
STEP3
STEP
STEP 10
STEP3
STEP
STEP
9
G
G
Peaks
Vallex
Figure 5: Latent distribution transform with WGAN-GP/DCGAN as backbones on MNIST. As the I-FGSM iteration guided by Eq. conttinuss the latent distribution deviates from the standard Gaussian, and contains more valleys and peaks which can be mapped into generated images as shown the bottom. Note that the generations by the valley points are of low quality, and our scheme can effectively avoid them as the sampling probability is low (valley) in our transformed distribution.
Esampio
Saxmple 2
Reseglle 1
Sxzoples
Esamples
AdclatiAN-qus
AdVLatGAN qua
x
AdMastiAN-2
600)
G-dtt
Ei nome
Addaatihh--e++
AMVESIGAN-- diy
X
Figure 7: The logic of our variant methods in Sec. 4.I. Blue and red is for quality and diversity.
Figure 6: Each of the six columns shows two compared pairs of generated examples (first pair: 1st + 2nd row; second pair: 3rd row 2nd row) by our diversity driven iterative transform scheme in latent space with the same number of iterations obtain 2" from initial z. Middle: generation by vanilla latent sampling 2; Top: by latent samples mined by Eq. g Bottom: by latent samples mined by Eq. B8s inverse form. It
Table 2: Five variants in Sec.
Latent Transform 2"
Improved mapping G
Variant
i)
ii)
iii)
shows that using Eq. [ generates more similar image pairs serving hard training samples.
iv)
The symbols and terms are similar to Eq. 3 Akin to AdvLaatGAN-qua, we use I-FGSM to obtain z and again use to to control deviation magnitude. For each update step, we randomly select the first latent vector Zo, and the second vector z" is searched by n iterations of the I-FGSM method under the loo restriction starting from the Gaussian neighborhood of zo with small standard deviation.
Table 1: Ratio between image and latent distance by Eq. g to derive new z" -inv: using inverse form.
Dataset Eq. 8
Eq. 8 inv
Cifar-10
0.241
4.023
FACADES
0.3785
1.3385
Table I shows the effectiveness of the above paired hard sample seeking method, where the results are calculated over 5k generated images with DCGAN 47 as backbone. The visual results on Facades as shown Fig. 6 suggest that the pair obtained by solving Eq, Leend to collapse (as they are hard samples), while the opposite leads to more different generation by using the inverse form of Fq. 8
The above experiments verify the efficacy of our mining technique. We further introduce it into the initial training process of MSGAN (32). In MSGAN, the regularization term by Eq. 3 regularizes the optimization process, where the selection of the pair entirely random, without using any heuristic information. We instead use the z pair that is more inclined to collapse by randomly taking one z and iterating by the aforementioned method get another z. The other settings remain unchanged.
By using sample pairs mined by Eq. g in the optimization as inspired by Eq. 2 we finally develop our new diversity enhancing GAN training approach AdyLatGAN-div as shown in Alg, 2 in the appendix, which can be further enhanced by post-training as discussed in experiments.
4 Experiments
Experiments on ImageNet and CelebA are performed on GPUs of Tesla V100. Other public bench- mark results are performed on single GPU of GeForce RTX 3090.
7

---

Table 3: Post- Post-training latent sampling improvement with three architectures on STL-10 (best in bold).
DCGAN
WGAN-GP
SNGAN
Method
1S(1)
HDD()
18(1)
FID(J)
1S(t)
HID(J)
hare sampling transform
7.199100:077
58.675=0.023
8.74940.093
35.706=0.395
8.48220.138
37.059=0.768
EvoIGAN [25]
7,174+000066
58.761 58.761=0.030
8.773+0.068
35.851=0.003
8.517=00069
36,343=0.512
DOT S3
7.195+00.022
58.670-0.076
9.708++0.013
30.745-0.113
8.798+0.024
36.769-0.133
DDLS 3D
7.30010.131
6..2561110422
9.3701.0:125
32.73811.483
8.59510.071
35.301
AdvLaiGAA,
7.933+-0.041
54.913-0.451
00.49990.048
27.083-0.245
9.426+0.058
31.891-0.027
4.1 Experimental Setup
We validate the effectiveness the proposed methods for two parts, i.e., effective latent sampling improvement =* and improved generation mapping G. Five variant methods can be derived for different targets, among which AdvLaIGAN-qua+ and AdvLatGAN-div- are our final full version.
i) AdvLatGAN-z: post-training latent sampling improvement fighting against quality discontinuity; ii) AdvLatGAN--uaa GAN training algorithm for better quality using in-training latent sampling transform; iii) AdyLatGAN-div. GAN training algorithm for more diverse generation by using in- training latent sampling transform; iv) AdyLatGAN-qua+ conducting -z over the trained networks of qua; v) AdvLatGAN-dive conducting -div over the trained networks of -div. We compare these methods in Fig. 7 and Table
We adopt the Inception Score, IS 48 Fréchet Inception Distance, FID 49 and density/coverage (50) for ovaluation. IS utilizes the classification results of the generated images by the Inception Network to evaluate the generation quality. FID additionally considers the target distribution and evaluates the quality by fitting the distribution distance. Density and coverage are more recently proposed evaluation metrics that analyze the target and generated manifolds and estimate the similarity. In the following experiments, IS and FID are calculated over 50,000 images, while density and coverage are calculated over 10,000 images. Overhead discussion is presented in Appendix C.
4.2 AdyLatGAN-z: Performance Boosting by Post-training Latent Sampling
We evaluate the boosting effect of post-training latent sampling improvement method AdvLaiGAN-z (also previously denoted as 2²), which is guided by Eq, 5 It aims to mitigate quality discontinuity.
Results Synthetic Data. We test AdvLatGAN- Grid and Ring datasets consisting of mixture of 25 and 8 2-D Gaussians as done in 51. The result is given in Fig. 9 By updating the latent variables, method achieves high-quality generation (samples closer to target Gaussian centers).
Results on MNIST. Low complexity of MNIST [22 makes possible to generate recognizable results by two-dimensional latent variables which can be better visualized. The iterative latent distribution change illustrated in Fig. 5 We respectively conduct AdvLarGAN-Z for the trained DCGAN and WGAN-GP models, with single-step iteration constraint using foo and iterated step size e is set as 0.03. We present the first 30 iterations that can reflect the most significant changes.
Results on STL-10. STL-10 46 maintains a higher resolution and richer image representation. We utilize AdvLaiGAN-z for pre-trained DCGAN, WGAN-GP and SNGAN, with for single-step constraint and the iteration step size c is set as 0.05. We conduct 20 steps each time. IS and FID are adopted for evaluation, which are calculated over 10,000 generated images. We compare other efforts enhancing GAN performance by latent space mining including DOT, DDLS and EvolGAN, Table present the results. AdvLaiGAN-z outperforms the baselines and delivers at most 24.3% advantage for FID on vanilla in the WGAN-GP setting.
Results on AFHQ and FFHQ To address the issues in Fig. we evaluate on AFHQ [54] and FFHQ (55) based on StyleGAN2-ada [18]. In Fig. setting, when the sampling encounters invalid examples. AdvLatGAN-z can transform the bad to generation-frient friendly (2) avoid bad gen- eration. Here we verify whether invalid samples can be improved with AdvLatGAN-z. We select six bad genera- tions with the first three corresponding to those in Fig. and conduct Ad to their latent vectors. Fig. showww that AdyLaIGAN-z can effectively fix avoid defects.
(a) IS
(b) FID
Figure 8: Influence of step size e and # #lteration: mean and std of differ- ent random seeds.
8

---

Table 4: Results of AdvLatGAN qua and AdvLaIGAN- qua+ for Cifar- 10 and STL-10 (best in bold).
bare
AdvLatGAN-qua
AdvLatGAN-qua+
Datasel
Method
15(t)
FID(4)
IS(*)
FID(L)
IS(t)
FID(L)
DCGAN
0.92
46.4 0.7
6:20
1.0
6.62 0.26
40.2
WGAN 7
6.63
3.5
1.20
2.7
7.93 1 0.02
26.9 1 0.6
Cifar-
WGAN-GP
56
7.47
24.7
7.60
22.6
8.84 0.08
16.6 0.4
SNGAN
7.29
2..5
0.08
0.5
8,33 = 0.03
1
LSGAN 9
5.87
49.3
6.13 =0227
42.8 1.3
0.10
42.3 +:
WGAN-div
7.43
23.8 0.3
7.81 +0003
20.6
+004
8.87
15.2
ACGAN 58
6.22
59.5 1.5
6.06 0.211
53.7
1.6
6.24 + 0.16
53.9 0.5
DCGAN
7.08
61.21:12
0.33
1.0
7.98
52.8
WGAN
7
6.00
73.0
7.62
5.6
8.57
2.9
STL-10
WGAN-GP
56
8.86
37.4
0.00
34.2
1..78
25.0
SNGAN
8.49
36.
8.08
34.5+0.21
9.64 0.05
29.9 0.1
LSGAN
9
7.08
62.9
2.2
58.5
7.85
55.3
1.4
WGAN-div
8.02
37.7 0.2
9.00 0.01
32.0 1.0.6
10.98 0.28
23.0
1.3
Table 5: Larger-scale evaluation (best in bold).
Densiry
Coverage 1
WGANGP
0,1192
0.247 0.0013
0.997 0.005
LSUN-66
WGANGP-qu
12:415 0.0228
0.755 +7.00-0.000
0.00 0.003
Original
Qualityehbanre
Dengiat
Diality-anhaooo
WGANGP quat
10.353 1 0.257
0,765 a 0.011
0.926 1 0.021
Figure 9: Post-training latent sampling improve- ment on Grid and Ring by AdvLatGAN-qua.
SNGAN
11.96 1.3331
L059 0.014
0.925 0.000
SNGAN-qua
00.104
0,033
0.39 0.002
SNGAN-qua-
1.063
WGANGP
14.180
0.756
0.007
0.35 0.007
Colsh-444 LSUN-123
WGANGP-qu
12.581
+.0.45
0881
0.0114
0.788 D.011
WGANGP--qu4
11,093+ 0.210
01869
0.004
0.914 0.004
SNGAN
16.078
1.1193
0,946
0.05
0.863
0.003
SNGAN-qua
0.035
0.052
0,80 0.004
SNGAN-qua-
WGANGP
19,83410.395
0,247.10017
10,011
WGANGP-qu
18:97::0.174
0.246::0005
0.602.50005
WGANGP
17.748
0.262-0.012
0L63 HUMIL
SNGAN
19. 19,49010.473
0.29510.002
0.61210.008
Figure 10: Results of AdvlatGAN-z on AFHQ and FFHQ. The top row are bad generations
SNGAN-qua-
SNGAN-qua
18.255-00.209
0.301-0.006
0.652-+0.002
WGANGP
25 25:208.10
0.20110.005
0.50510.015
(the first three correspond to those in Fig. 3
CaldhA-128 128
WGANGP-qua
22.376:10.181
0.247:00007
0.579:10.012
WGANGP-qUa
21.3744-0361
0,256 2.26-1
IL5V7+
with the defects as follows: 1st column:
cat
SNGAN 22,96010.092
0.66
0.6:910.004
misses ear; 2nd column: white spots the
SNGAN-qua
22.476-0.023
D.462-0.006
0.651-+1.005
SNGAN-qua-
20.235:0.4022
nose: 3rd column: two bodies share one head;
WGANGP
78.15010.298
0.200.0.002
10.005
4th column: green face; 5th and 6th columns: semi-existing glasses. The bottom row are re- sults under AdvLatGAN-z. AdvLatGAN-Z effectively mitigate defects.
InsagsNe
WGANGP-qu
73.677:0.576
0.242:-0.00
0.322-10.010
WGANGP
71.227-0.444
0,251
0,333-HU014
SNGAN
08.36610.758
0.188100003
0.213:10.077
SNGAN-qua
79.087-H0.544
0.207-0.000
0.266-+11007
SNGAN-qua-
74.643=:0844
Hypee--arrameter Study. Considering the influence of the step size e and the number of iterations, we evaluate WGAN-GP's performance STL-100 adopting IS and FID metrics. Fig. 8 shows the generative performance after the iteration number. We observe that larger e (e.g. 0.05) can deliver fast improvement but can be unstable for more iterations, while smaller can achieve stable improvement but is slightly less significant. If time permits for sufficient parameter tuning, we recommend adopting larger like 0.05 along with particularly selected iteration number, otherwise a smaller like 0.01 with sufficient iterations will enough for worthy performance gain.
4.3 AdvLatGAN-qua/diis Improving Generation Map
This section shows the effectiveness our training pipelines, which aim to improve the generation mapping G respectively for quality and diversity. The details presented in Alg. and Alg.2.
AdvLatGAN-qua for Quality Improvement. Table 4 shows the results on Cifar-10 10 and STL-10, using the mainstream architectures: DCGAN, WGAN, WGAN-GP, SNGAN, LSGAN, WGAN- div and ACGAN. We do not include ACGAN in the unlabeled STL-10 setting because it requires labels (58 IS and FID are adopted. We adversarially train GAN using AdvLatGAN-qua. The

---

Table 6: Results of AdyLaiGAN-div and AdyLatGAN-diig div+ for Cifar- 10 and STL-10 (best in bold).
Datiset
Metriss
Modic
ovcrall
sieplane
sutonobile
dog.
frog
Inese
ship
truck
MSGIA
78.258
57.778
56.811
69. 69:205
69.994
FID(I)
27,054
71,627
67.570
54,457
54,274
558356
65,083
69.662
63.127
AdvLaGGAN dive
26.169
67.100
67.499
73,449
67.470
55.817
$5.815
55.227
64.159
67.265
63.223
Ciar
MSGIAN
0.226
0.536
0.264
0.355
0.221
density(1)
ddi
0.534
0.250
0.254
0467
0.534
0.306
0.665
0.381
0.365
0.225
AdvLaiGANN-dvv
0.492
0.348
0.311
0.227
0.595
0.430
0.296
0.590
0.352
0.366
0.247
MSGAN
0.828
0.789
0.843
0.596
0.065
0.833
0.610
0
0.916
0.957
0.775
covengelt)
AddLasGANdii
0.84
0,50
0.908
0.900
0.793
0.854
0.607
0.919
0.15
0.962
0.788
AdvLuGANN dive
0.849
0.744
0.922
0,604
0,800
0.891
0.599
0.934
0.877
0.559
0.780
MSGAN
67,849
921021
125.723
108.334
118938
111.754
133.690
140.436
121,907
252.232
101 US9
FID(I)
AdvLaGAN div
65.053
92.007
225
109672
118.645
104.59
132,424
140.256
113.765
95.833
99.739
Adll -diiAN
60.205
R8871
118,866
104.206
111.238
99.255
125.287
151,170
106.568
91,895
95.715
STL-10
MSGAN
0.353
0.195
0.241
0
0.558
0.378
0.354
0.146
0.232
0.120
0.118
density(1)
Adul dei
0.420
0.220
0.292
0.114
0.686
0.484
0.384
0.193
0.336
0.164
0.123
AdvLaGAN diet
0.431
0.248
0.295
0.125
0,733
0.401
0.381
0.209
0.330
0.190
0.149
MSGAN
0.455
0.538
0,535
0.261
0.530
0.380
0.391
0.370
0.410
0.456
0.324
covenge(r)
AdtLaGAN div
0.514
0.526
0:655
0.350
0.624
0.683
0.009
0.513
0.439
0.313
0.265
AdvLaGAAN-de di++
0.540
0.554
0,519
0.364
0.699
0,514
0.433
0.550
0.484
0.564
0.383
number of latent iterations in training is set as one per discriminator step, and other details is given in Appendix J.1. As they are orthogonal to each other, integrate AdvLatGAN-z with AdvLatGAN- qua (i.e. AdvLaiGANN-qa+)) and include it in comparison. The iteration step size of AdyLatGAN-z is to 0.05 and conduct 20 steps each time. Our method achieves notable improvement compared to that without the latent space transform strategy. Advl LatGAN-uuu++ achieves the best gain on IS in WGAN STL--10 setting from 6.51 8.57 and FID in WGAN-div from 37.7 23.0.
Table. 5 presents results on larger datasets: LSUN Church [59]. CelebA (60] and ImageNet [61]. using WGAN-GP and SNGAN backbones, evaluated by FID, density and coverage. We include AdvLatGAN-qua and -qua+ in the comparison. The number of latent iterations during training is set as 1 per discriminator step, and other details are given in Appendix J.1. The iteration step size c of AdvLatGAN-z is set to 0.01 for SNGAN and 0.002 for WGANGP and we conduct 100 steps each time. AdvLatGAN-qua+ has achieved the best performance gain on FID in SNGAN LSUN-64 setting from 11.961 to 7.285.
AdvL AtV-Niiv for Diversity Improvement. We test on Cifar- and STL-10 for conditional generation and we adopt FID, density and coverage as the metrics, respectively for the entire set of generated images and for those in each label. We test AdvLaiGAN-div and AdvLaIGAN-div compared to the baseline MSGAN. The results are shown in Table 6 Our method outperforms MSGAN by all the three metrics. Note that the models are trained on the overall set and for each label and the optimization may not guarantee targeted enough guidance. AdvLatGAN-din achieves the best overall performance gain FID from 30.225 to 26. density from 0.353 to 0.431 and coverage from 0.455 to 0.54.
5 Conclusion and Broader Impact
This work analyzes GANs from the continuous mapping perspective and notes that lifting the overall generative performance requires a twofold effort including latent distribution transform and mapping improvement. Adversarial sample mining techniques are introduced to explore latent space and novel training pipelines are derived to improve the generative mapping. The proposed AdvLaIGAN has shown promising power in both quality and diversity. Our technique is basically orthogonal to existing mainstream methods. For its limitation, it remains open for how to effectively combine the proposed two techniques for diversity and quality. For potential negative social impact, fake content by the generation models including ours shall always be carefully treated to avoid abuse.
Acknowledgements
This work was partly supported by National Key Research and Development Program of China A2100A,00007000,, National Natural Science Foundation of China (61972250, 7206 1270033, and Shanghai Municipal Science and Technology (Major) Project 12I.LHHHZZ000022211110000
10

---

References
[1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Benggio, "Generative adversarial nets," in NIPS, 2014.
[2] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani,
J. Totz, Z. Wang et al., "Photo-realistic single image super-resolution using a generative adversarial network," in CVPR, 2017.
[3] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired image-to-ime to-iage translation using cycle-consistent adversarial networks," ICCV, 2017.
[4] X. Wang. K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, "Esrgan: Enhanced super-resolution generative adversarial networks," in ECCV Workshops, 2018.
[5] A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, and D. Mukhopadhyay, "Adversarial attacks and defences: A survey," arXiv preprint arXiv:1810.0000 00069, 2018.
[6] F. Zhan, H. Zhu, and S. Lu, "Spatial fusion gan for image synthesis," in CVPR, 2019.
[7] M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein generative adversarial networks," in ICML, 2017.
[8] T. Miyato, Kataoka, M. Koyama, and Y. Yoshida, "Spectral normalization for generative adversarial networks," in ICLR, 2018.
[9] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, "Least squares generative adversarial networks," in CVPR, 2017.
[10] T. Salimans, Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, "Improved techniques for training gans," in NIPS, 2016.
[11] A. Odena, C. Olah, and J. Shlens, "Conditional image synthesis with auxiliary classifier gans," in ICML, 2017.
[12] A. Brock, J. Donahue, and K. Simonyan, "Large scale gan training for high fidelity natural image synthesis," in ICLR, 2018.
[13] T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adversarial networks," in CVPR, 2019.
[14] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, "Self-attention generative adversarial networks," in ICML, 2019.
[15] M. Khayatkhoei, A. Elgammal, and M. Singh, "Disconnected manifold learning for generative adversarial networks," in NewrlPS, 2018.
[16] A. Srivastava, L. Valkov, C. Russell, M. Gutmann, and C. Sutton, "Veegan: Reducing mode collapse in gans using implicit variational learning." in NIPS, 2017.
[17] A. Kurakin, I. Goodfellow, S. Bengio al., "Adversarial examples in the physical world," 2016.
[18] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, "Training generative adversarial networks with limited data," in NeurIPS, 2020.
[19] D. Mishra, A. Jayendran, and A. Prathosh, "Effect of the latent structure on clustering with gans," IEEE Signal Processing Letters, 2020.
[20] S. Mukherjee, H. Asnani, Lin, and S. Kannan, "Clustergan: Latent space clustering in generative adversarial networks," in AAAI, 2019.
[21] Y. Wu, J. Donahue, D. Balduzzi, K. Simonyan, and T. Lillicrap, "Logan: Latent optimisation for generative adversarial networks," arXiv preprint 1912 20953, 2019.
[22] A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and Yosinski, "Plug & play generative networks: Conditional iterative generation of images in latent space," in CVPR, 2017.

---

[23] A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune, "Synthesizing the preferred imputs for neurons in neural networks via deep generator networks," NIPS, 2016.
[24] Bontrager, W. Lin, J. Togelius, and S. Risi, "Deep interactive evolution," " in EvoMUSART, 2018.
[25] B. Roziere, F. Teytaud, V. Hosu, H. Lin, J. Rapin, M. Zameshina, and O. Teytaud, "Evolgan: Evolutionary generative adversarial networks," in ACCV, 2020.
[26] B. Roziere, N. C. Rakotonirina, V. Hosu, A. Rasoanaivo, H. Lin, C. Couprie, and O. Teytaud, "Tarsier: Evolving noise injection in super-resolution gans," in ICPR, 2021.
[27] V. Hosu, H. Lin, T. Sziranyi, and D. Saupe, "Koniq-10k, An ecologically valid database for deep learning of blind image quality assessment," IEEE Transactions on Image Processing, 2020.
[28] H. Berard, G. Gidel, A. Almahairi, P. Vincent, and Lacoste-Julien, "A closer look at the optimization landscapes of generative adversaria networks," in ICLR, 2019.
[29] S. Azadi, C. Olsson, T. Darrell, I. Goodfellow, and A. Odena, reisrimiinatorrjjcccti sampling." in ICLR, 2018.
[30] T. Che, R. Zhang, J. Nohl--Dccsttei, H. Larochelle, L. Paull, Y. Cao, and Y. Bengio, "Your gan secretly an energy-basee model and you should use discriminator driven latent sampling," in NeurlPS, 2020.
[31] M. Elfeki, C. Couprie, M. Riviére, and M. Elhoseiny, "Gdpp: Learning diverse generations using determinantal point processes," in ICML,, 2019.
[32] Q. Mao, H.-Y. Lee, H.-Y. Tseng, S. Ma, and M.-H. Yang, "Mode seeking generative adversarial networks for diverse image synthesis," " in CVPR, 2019.
[33] H. Meulemeester, J. Schreurs, M. Fanuel, B. Moor, and J. Suykens, "The bures metric for taming mode collapse in generative adversarial networks," in CVPR, 2020.
[34] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, "Intriguing properties of neural networks," arXiv preprint arXiv:1312.6199: 2013.
[35] I. J. Goodfellow, Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," arXiv preprint 141.6657, 2014.
[36] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," in ICLR, 2018.
[37] A. Nøkland, "Improving back propaaaaannnn by adding an adversarial gradient," " arXiv preprint 1rXiv:151004188. 2015.
[38] Y. Wang. X. Ma, J. Bailey, J. Yi, B. Zhou, and Q. Gu, "On the convergence and robustness of adversarial training." in ICML, 2019.
[39] S. Zhang, Z. Qian, K. Huang, J. Xiao, and He, "Robust generative adversarial network," arXiv preprint arXiv:2004.13344 2020.
[40] X. Liu and C.-J. Hsieh, "Rob-gan: Generator, discriminator, and adversarial attacker," in CVPR, 2019.
[41] J. Zhong, X. Liu, and C.-J. Hsieh, "Improving the speed and quality of gan by adversarial training," arXiv preprint arXiv:2008.00 033644 2020.
[42] F. Liu, M. Xu, G. Li, J. Pei, L. Shi, and R. Zhao, "Adversarial symmetric gans: Bridging adversarial samples and adversarial networks," Neural Networks, 2021.
[43] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, Hu, and J. Li, "Boosting adversarial attacks with momentum," in CVPR, 2018
12

---

[44] Y. Wang, D. Zou, J. Yi Bailey, X. Ma, and Q. Gu, "Improving adversarial robustness requires revisiting misclassified examples," in ICLR, 2019.
[45] M. Mirza and S. Osindero, "Conditional generative adversarial nets," arXiv preprini arXiv:1411.1784 2014.
[46] A. Coates, A. Ng. and H. Lee, "An analysis single-layer networks in unsupervised feature learning," in AISTATS, 2011.
[47] A. Radford, L. Metz, and S. Chintala, "Unsupervised representation learning with deep convo- lutional generative adversarial networks," ICLR, 2016.
[48] Salimans, Goodfellow, W. Zaremba, Cheung, A. Radford, and X. Chen, "Improved techniques for training gans," in NIPS, 2017.
[49] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "Gans trained by a two time-scale update rule converge a local nash equilibrium," in NIPS, 2017.
[50] M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, and J. Yoo, "Reliable fidelity and diversity metrics for generative models," in ICML, 2020.
[51] L. Metz, B. Poole, D. Pfau, and Sohl-Dickstein, "Unrolled generative adversarial networks," in ICLR, 2017.
[52] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," " Proceedings of the IEEE, 1998.
[53] A. Tanaka, "Discriminator optimal transport," in NeurIPS, 2019.
[54] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, "Stargan v2: Diverse image synthesis for multiple domains," in CVPR, 2020.
[55] Karras, S. Laine, and Aila, "A style-based generator architecture for generative adversarial networks," in CVPR, 2019.
[56] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, "Improved training of wasserstein gans," in NIPS, 2017.
[57] Wu, Huang, Thoma, Acharya, and L. Van Gool, "Wasserstein divergence for gans," in ECCV, 2018.
[58] M. Kang, J. Shim, M. Cho, and J. Park, "Rebooting acgan: Auxiliary classifier gans with stable training." in NeurlPS, 2021.
[59] Yu, Y. Zhang, Song, A. Seff, and Xiao, "Lsun: Construction of large-scale image dataset using deep learning with humans in the loop," arXiv preprint arXiv: 1506,03365 2015.
[60] Z. Liu, Luo, X. Wang, and X. Tang, "Deep learning face attributes in the wild," in ICCV, 2015.
[61] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in CVPR, 2009.
[62] A. Madry, A. Makelov, L. Schmidt, Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," arXiv preprint 1oXiv:706600683, 2017.
[63] B. Zhou and P. Krähenbühl, "Don't let your discriminator be fooled," in ICLR, 2018.
[64] M. Binkowski, J. Sutherland, M. Arbel, and A. Gretton, "Demystifying mmd gans," in ICLR, 2018.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Abstract. (b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section + Appendix H, Appendix and Appendix J.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 4
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "kfJTGAWAotx", "reviewer": "Reviewer_Z7p2", "summary": "This paper studies the latent space in Generative Adversarial Nets (GANs) by introducing a implicit mapping on latent variable z before sending it to the generator G. Such a network z*(z) is trained by minimizing the generator loss, with G and D fixed. An additional level of optimization is introduced to optimize z* during the training of the proposed AdvLatGAN. Based on improving the quality and diversity of the model, two variants of the model are introduced, by modifying the objective of z*.", "questions": "Please see the previous session", "limitations": "It would be great to see the performance of AdvLatGAN on some larger-scale image dataset.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1) The paper presents an interesting perspective of improving GANs. The idea seems to be novel.\n2) Experiments show improvements on Cifar-10 and STL image generation, by addion AdvLat in different GAN framework.\n\nWeakness:\n\n1) The idea of using implicit distribution for generative models has been explored in a few works, including [1], it would be great to have some comparisons with these methods.\n2) The idea of adding additional network to transform z seems to be something could have been done within the generator, if we think of the transform network and the generator network as a whole. I'm wondering if the improvement was actually achieved by the bi-level optimization. It would be great to see some ablation study on the effectiveness of each component of the proposed method.\n\n\n[1] Fang, L., Li, C., Gao, J., Dong, W., & Chen, C. (2019). Implicit deep latent variable models for text generation. arXiv preprint arXiv:1908.11527.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jtMXyvzN9EI", "reviewer": "Reviewer_phhm", "summary": "Author proposed a novel idea about training/sampling GAN from a discontinuous distribution by setting one more layer in-between known prior $P_z$ and real distribution $P_x$. Author provided mathematical proof that discontinuity of latent space (support of G) is required to cover the complex and discontinuous real distribution. It is also provided that minimizing Eq.5 is equivalent to minimizing Jenson Shanon Divergence between the latent distribution $P_z$ and a distribution of a union of disconnected subset in the latent space $P_z^{op}$. The effectiveness of the proposed idea is shown in Experiments.", "questions": "1. It is not understandable that if real distribution is a union of disconnected manifolds, the support of G needs to be disconnected as well.\n\nLine 534-536: Let's say $U_1$ is Male and $U_2$ is Female. The proof goes like this; assume there exists a latent vector $z$ generating both Female and Male, i.e., $G(z) \\in U_1$ and $G(z) \\in U_2$. This implies $z \\in U_1 \\cap U_2$, which contradicts the condition 'Female and Male do not have intersection', i.e., $U_1 \\cap U_2 = \\emptyset$. To me, it looks like a simple repeat of the assumption and does not prove that $Z^{op}(G)$ is a union of disconnected subsets in the latent space. Please correct me If I am wrong.\n\nAnd If my argument is not wrong, the entire logical flow of this paper must be revised as well.\n\n2. The proof in Proposition B.3. includes an assumption that $E_{x \\sim p_r}[log(D(x))]=E_{z \\sim p_z^{op}}[log(D(G(z)))]$, which means the given generator (hence the discriminator as well) for the proof must be optimal. It needs to be mentioned. Furthermore, in practice in Alg. 1, optimizing Eq. 5 is conducted without optimal $G$ and $D$, which needs to be at least justified in the paper. Please correct me if I am wrong.\n\n3. I am not sure, but isn't it bad in terms of diversity if $z$ is optimized by Eq.5? What is the number of real/fake samples for FID score in Table 4?", "limitations": "Limitation is not provided in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n1. Proposed idea is interesting and may have a  possibility to be used to avoid from unrealistic generation from GAN.\n2. Experiment result verifies the effectiveness of the proposed idea.\n\n$\\textbf{Weakness}$\n1. Logical flow of the paper at Introduction part and connection between the motivation of the paper and the proposed method is not clear.\nSpecifically, the logical flow is like below:\n\n  (1.1). Real distribution is a union of disconnected manifolds\n\n  (1.2). Proof for necessity of disconnected latent space (at least in high level)\n\n  (1.3). The gaussian space of ${G}$ is not disconnected.\n\n  (1.4). Eq. 5 and Eq 8?\n\nHere, (1.2) is not mentioned in Introduction, which I think necessary.\nMore importantly, (1.4) has a weak connection with the logical flow.\n\n2. Mathematical proof: I wrote my thought in the Questions part.\n\n3. Method part is written well, but Introduction part is NOT organized well. It also has some unnecessary or confusing expressions.\n\nFor example, \n\nat Line 50, “otherwise the generative expression will differ.” What does the generative expression stand for? Why is it necessary for justifying the perturbation-based method?\n\nat Line 30-31, \"One reason is that the generator is a continuous function (as neural nets) while the generation quality (e.g. natural images in pixel space) is not.\" Does it mean the generation quality is not a continuous function? How can the quality be a function?\n\nat Line 38-40, Most generative models such as GAN, VAE, Diffusion Model and Flow-based model also inherently have a mapping between simple known distribution and complex real distribution. I believe disconnected modes of real distribution cannot be said to be \"well-known difficulty in GANs training\".\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K3c4Ek2S5Pv", "reviewer": "Reviewer_Y24j", "summary": "By introducing the adversarial techniques into GAN, this paper proposes new training and latent variable sampling methods to achieve better generation quality and diversity. The motivation is interesting that, the generator's limit as a continuous mapping: a) the contradiction between continuous mapping pixel-space and quality discontinuity; b) the mapping can incur mode collapse. Accordingly, the work adds a latent space sampling shift to amend the raw latent distribution and introduces targeted sampling shifts into GAN’s bi-level optimization for quality and diversity improvement. Extensive experiments on image data including large-scale dataset and backbone networks show that the proposed techniques lead to promising improvements.", "questions": "1.I can understand that the paper’s contribution is irrelevant to the specific choice of the backbone, and the experiments have already shown its effectiveness on various GAN backbones. Yet it is still informative to see its role on more SOTA designs e.g. StyleGAN.\n\n2.I also understand that there exists a difference between qua (quality) and div (diversity) settings, but there is still room for synergizing the two parts. Why not try adopting qua and div at the same time? Or the authors shall better clarify this point.\n\n3.Since many generative models other than GAN also use a single network to accomplish the generative task, do they also suffer from the same quality discontinuity issue? Can the proposed method be generalized to other generative models e.g. VAE?\n", "limitations": "Please see above.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n1.This work is highly motivated and well presented. The idea is interesting.\n\n2.The contributions are clear. This paper notes the effectiveness and importance of latent space sampling shift in GAN and may incur more follow-up studies devoted to making twofold efforts including sampling shift and mapping improvement.\n\n3.The idea is supported by extensive evaluation with impressive results and theoretical arguments.\n\n### Weaknesses:\n\n1.The proposed latent sampling shift is iteratively conducted, which may incur more computational costs, especially in in-training algorithms. The trade-off between time consumption and performance gain needs to be discussed.\n\n2.Some related works shall be discussed. The following papers mine latent space for different targets like conditional generation are worth for discussion.\n\n[1] Nguyen A, Clune J, Bengio Y, et al. Plug & play generative networks: Conditional iterative generation of images in latent space, CVPR 2017. \n\n[2] Nguyen A, Dosovitskiy A, Yosinski J, et al. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks, NIPS 2016.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5gNAlNspbXi", "reviewer": "Reviewer_qXs3", "summary": "This paper is about GAN training. The authors suggest that using a continuous latent distribution to model a discontinuous image distribution can lead to poor sample quality. The propose doing gradient updates directly on the latent representation to improve sample quality. They also do latent gradient updates to improve sample diversity. The authors show quantitative results on several natural image datasets, including ImageNet.", "questions": "Please could the authors clarify what they mean in lines 30-31 when they say that the generation quality is not a continuous function?\n\nLine 51-52: “z* can be achieved by updating the latent z~p(z)” Can the authors please clarify that z* is a gradient update function on z? Or is it something else? Does z* have any parameters?\n\nShould equation 2 be arg min?\n\nSection 3.1 is confusing because it talks about both adversarial attacks and Generative Adversarial Networks which are different research areas that share the term “adversarial”. I would advise the authors to improve the clarify of this section.\n\nHow do equations (1) and (2) related to equations (3) and (4)?\n\nPlease clarify sentence on line 151-152: If all sub-manifolds M’s “are disconnected” how can the authors “require the splitting of Xr to the extent that each sub-manifold keeps connected.”\n\nI suggest that the authors more clearly introduce the terms “latent space mining”, “adversarial sample mining” and how this is different to drawing samples from a generative model.\n\nProposition 3.2 is intuitive and makes sense.\n\nSection 3.3: Are results on Table 1 computed across the whole test dataset? Results in Figure 6 do not suggest that there is an increase in sample diversity.\n\nResults in Figure 5 are very nice and demonstrate the motivation (relating to sample quality) of the paper well. Please clarify in the caption which loss z* is being optimised for.\n\nLine 243 “improved generation mapping G.” Does this method encourage the learning of better parameters in the generator, G? Or is it that you are able to draw better samples? Please clarify? If it is the former, how does your approach lead to improved learning of parameters for G?\n\nLin 254: Introduce acronyms IS and FID. For example, “Inception Score, SI [46], Fréchet Inception Distance, FID”.\n\nSection 4.2. Which objective is being optimised for the results this section? Please reference the question in the text.\n\nResults in Figure 9 are impressive. Are these cherry picked? Is it not possible to apply this to all steps in the interpolation from figure 3?\n\nLine 301: Please clarify what is meant by “and does not work on unlabeled STL-10”. \n\nTable 6: Please double check that all values show in bold are statically significantly better that the baselines. Where this is not the case either underline the others or indicate in some other clear way.\n\nWhat is the application of this method? How does it affect the learned representations? Can they be used for more data efficient learning (for example in a classification task)?", "limitations": "Authors have considered a single limitation.\nIt would also be helpful to note how their approach affects the representation quality and the computation cost of optimising z*.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\nThe authors draw our attention to an interesting problem, where by a continuous the latent distribution is used to model a discontinuous data distribution and propose using gradient updates directly on the latent representations to remedy this. The also do gradient updates on the latent space to improve variation. In both cases the authors use existing losses and optimise w.r.t. the latent representation. \n\nQuality:\nFigure 1 is helpful in motivating the problem that the authors are trying to solve.\nSection 4.1: It is great that the authors consider many different variations of their model.\nImpressive results in Figure 9 show how this approach can be used to \"fix\" poor samples. However, it would be better if we could see all the steps in the interpolation still and the cats ear is still partially missing.\nFigure 4 is also great to show how the latent updates improve the image quality.\nResults in Figure 8 further motivate the model. \nTable 6 shows results on a large number of datasets. Its is good to see that the results hold on ImageNet.\n\nWhile the authors have some good results the experimental setup is not well described, it is not always clear which objectives are being optimised.\n\nThere is no explanation of how a model with \"in-training latent sampling transform\" (AdvLatGAN-qua) performs well without \"post-training latent sampling transform\"? Intuitively it should be worse?\n\nAddressing sample diversity is an after-thought and does not appear to be directly related to the main motivation of the paper.\n\nClarity:\nThis paper was very poorly written and very hard to follow. I have suggestions for improvements in the box below.\n\nSignificance:\nThe proposed work can be used to improve the quality of samples and diversity at the cost of doing gradient updates on the latent representations during sampling (although the authors do show results only doing updates durning training). This may be a high computation cost. It is also not clear what the application of this approach is. Does it improve sample efficiency during training on another task? How does the approach affect the learned representations? \n\nIt is also not clear to me why the problem of modeling a discontinuous data distribution is not solved by category conditional  generative models?\n\nStrengths:\n(1) The authors have very strong qualitative and quantitative results.\n\nWeaknesses:\n(1) The paper is very poorly written and very hard to follow.\n(2) It is not clear what the application of this model is.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept