Improving Generative Adversarial Networks via Adversarial Learning in Latent Space
Authors: Yang Li
Yichuan Mo
Liangliang Shi
Junchi Yan
Xiaolu Zhang
JUN ZHOU
Venue: ICLR 2022 Submitted (2022)
Keywords: Generative Adversarial Networks
Adversarial Traing
Latent Space

Abstract: Generative Adversarial Networks (GANs) have been widely studied as generative models, which map a latent distribution to the target distribution. Although many efforts have been made in terms of backbone architecture design, loss function, and training techniques, few results have been obtained on how the sampling in latent space can affect the final performance, and existing works on latent space mainly focus on controllability. We observe that, as the neural generator is a continuous function, two close samples in latent space would be mapped into two nearby images, while their quality can differ much as the quality is not a continuous function in pixel space. From the above continuous mapping function perspective, on the other hand, two distant latent samples are also possible to be mapped into two close images. If the latent samples are mapped in aggregation into limited modes or even a single mode, mode collapse occurs. Accordingly, we propose adding an implicit latent transform before the mapping function to improve latent $z$ from its initial distribution, e.g., Gaussian. This is achieved by using the iterative fast gradient sign method (I-FGSM). We further propose new GAN training strategies to obtain better generation mappings w.r.t quality and diversity by introducing targeted latent transforms into the bi-level optimization of GAN. Experimental results on visual data show that our method can effectively achieve improvement in both quality and diversity.

Under review as a conference paper at ICLR 2022
IMPROVING GENERATIVE ADVERSARIAL NETWORKS VIA ADVERSARIAL LEARNING IN LATENT SPACE
Anonymous authors
Paper under double-blind review
ABSTRACT
Generative Adversarial Networks (GANs) have been widely studied generative models, which map a latent distribution to the target distribution. Although many efforts have been made in terms of backbone architecture design, loss function, and training techniques, few results have been obtained on how the sampling in latent space affect the final performance, and existing works on latent space mainly focus on controllability. We observe that, as the neural generator is a continuous function, two close samples latent space would be mapped into two nearby images, while their quality can differ much as the quality is not continuous function in pixel space. From the above continuous mapping function perspective, on the other hand, two distant latent samples are also possible to be mapped into two close images. the latent samples are mapped in aggregation into limited modes or even a single mode, mode collapse occurs. Accordingly, we propose adding an implicit latent transform before the mapping function to improve latent z from its initial distribution, c.g., Gaussian. In this paper, this is specifically achieved by using the iterative fast gradient sign method (I-FGSM). We further propose new GAN training strategies to obtain better generation mappings w.r.t quality and diversity by introducing targeted latent transforms into the bi-level optimization of GAN. Experimental results on visual data show that our method can effectively achieve improvement in both quality and diversity.
INTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014a) have shown effectiveness for generating high-fidelit data, especially for images under various settings (Ledig et al., 2017; Wang et al, 2018; 8; Zhu et al., 2017; Zhan et al., 2019; Chakraborty et al., 2018). Based on the zero-sum game, the model is trained by the adversarial process between the generator and the discriminator. Many efforts have been made to achieve more realistic generation from different perspectives. WGAN (Arjovsky et al., 2017), SNGAN (Miyato et al., 2018), LSGAN (Mao et al., 2017) aim to design better objective functions. ImprovedGAN (Salimans et al., 2016), AC-GAN (Odena et al., 2017) propose practical training techniques. While more complex network structures are studied in BigGAN (Brock et al., 2018), StyleGAN (Karras et al., 2019a), SAGAN (Zhang et al., 2019).
Despite the above progress, an intriguing and relatively less studied question is how the latent space affects the generation in terms of quality and diversity, which can be orthogonal to the above frequently studied factors like model architectures and training techniques. Specifically, we argue that the latent samples from standard continuous distribution (e.g., Gaussian) can often be mapped to parying-quality samples for the generation. One reason is that the generator is a continuous function (by neural nets) while the generation quality (c.g., images in pixel space) is not. Moreover, has also been shown in GAN literature (Khayatkhoei et al., 2018) that many image objects lie upon multiple disjoint manifolds, and continuous mapping from continuous latent distribution, e.g., Gaussian must generate many invalid samples in the between of the manifolds when the generator seeks to cover all the meaningful modes. See Fig. 1 (a) for the schematic diagram. This fact also implies a dilemma: for complex real-world data, there is trade-off between quality and diversity when the latent vectors are sampled from a simple uni-mode distribution, e.g., Gaussian, and the generator is a continuous mapping based on neural networks. Moreover, by realizing that the mapping function in fact allows for many-to-on mapping, mode collapse (Srivastava et al., 2017) can naturally happen when different latent samples are mapped into limited or even single mode in target space.

---

Under review as a conference paper at ICLR 2022
G(2)
PHud Syare
(b)
Pixel Space
(b)
Latent Sianee
Latent Space
Space
(a) Quality Discontinuity
(b) Proposed Sampling Tranform
Figure Red dashed lines denote real data manifolds pixel space and their inverse images in latent space; yellow circles and squares denote samples for sound generation in latent space and pixel space; black circles and squares denote samples for bad generation in latent space and pixel space; the orange dashed lines denote the generator mapping; the blue dashed lines denote transform in latent space; blue shades and solid blue lines denote the latent distribution and the generated distribution. Due to the continuity of the neural network- based generator, close samples in latent space will be mapped to close images in pixel space, while the images' quality can vary. Cat (a) and Cat (b) are close in pixel space, but Cat (a) loses its right ear. Our method allows bad latent transform towards the inverse image of the nearest real manifold, thus avoiding terrible generation.
6(3).2-PP
6(2)2--) Pr
G
Bad Quality
quvLatGAN-qua
Bad Diversity
AdvlattGA--div
(a) Quality
(b) Diversity
Figure Red dashed lines denote real data manifolds; blue shades and blue lines denote the generated distribution. The generated distributions mapped from fixed latent distribution reflect the mapping quality. (a) shows the case of bad quality when the generated distribution does not match the real distribution well. (b) shows the case of bad diversity when the mapping misses modes. Our method tries to train mappings with better properties, as shown on the right side of each case.
The above contradiction between the discontinuity in generation data quality (e.g., images) and the continuity of latent distribution as well mapping function, can be one of the significant factors to the well-known difficulty in GANs training, especially for generators, as one unreasonably hopes to enforce discontinuous mapping via continuous neural nets. Meanwhile, the mapping property can largely affect the generative performance for functions with continuous nature, which is associated with most previous works devoted improving generative performance.
To tackle these issues, we impose an extra (implicit) transform function 2-) on the raw sampling z, and the generation can be written by Gz)" From this perspective, most existing works, including the vanilla GAN, employ =" =-(-) with an identical form. Specifically, the implicit function based transform (-) can be achieved by updating the latent ~ p(z) to minimize losse (i.e. log(1 - D(G(z))) using iterative fast gradient sign method (I-FGSM) (Kurakin et al., 2016) with the parameters of G and D fixed, and we name the method as AdvLatGAN-z. See Fig. 1 (b) for the schematic diagram. On top of the first effort, we propose novel training strategy improve the generative mapping respectively for quality and diversity. See Fig. for the schematic diagram. This is achieved by introducing iterations (i.c., targeted implicit transforms z) into GAN training, specifically, updating z ~ p(z) using I-FGSM to find more optimization -fienndyrr ones in the bi-level optimization of vanilla GAN. Respectively for quality and diversity, we use strategies for z under different objective functions training, and we name the two training algorithms respectively as AdvLatGAN qua and AdyLatGAN-dis The highlights of this paper include:
iterative updating
1) We rethink the role of the generator as a continuous function, which may incur quality discontinuity in target space and mode aggregation (collapse) when the latent samples are directly sampled from
2

---

Under review as a conference paper at ICLR 2022
continuous distribution e.g. Gaussian. Based above theoretical considerations, we propose composite generation perspective by formulating the mapping from G(z) to G(z (z)). This treatment provides a new methodological guidance to the progress of the generative model.
2) We propose a new pipeline for both Gi.) and = (-), by first training generative mappings for better generation quality and diversity, respectively, and then the transform 2-) is enforced and computed implicitly to obtain an improved distribution for more effective generation. Specifically, the training involves adversarial learning under different objective functions for quality and diversity, leading to new GAN models: AdvLatGAN-quall
and their further variants.
3) Experimental results on synthetic and visual data show the notable improvement of our approach on both generation quality and diversity. Our technique is orthogonal to existing GAN techniques.
2 RELATED WORK
Implicitly adapting the raw latent vectors z from Gaussian to improve the generation relates to latent space learning for GAN. While the specific training algorithm using latent vector adapting to mine cinniizattin-rriendii samples in training relates to adversarial training, which views the latent adapting in training as an attack to mine hard samples.
Latent space exploration in GANs. Techniques have been proposed for latent space exploration for GANs. By utilizing the steerability of GANs in latent space, StyleGAN (Karras et al., 201 introduce an latent space transform by using a mapping network to better decouple the various styles and Jahanian et al. (2019) shift the distribution to fit camera movement transformation. In- terFaceGAN (Shen et al., 2020) decouples entangled semantics with subspace projection to control facial attributes more precisely. Mishra et al. (2020), Mukherjee et al. (2019) and Wu et al. (2019) introduce the latent exploration into training to achieve clustering objectives or benefit GAN training. In addition the preceding works, some recent works focus improving latent sampling quality to generate high-quality images. For example, interactive evolutionary computation is used to make the generation process more controllable in Bontrager al. (2018), which meanwhile also brings addi- tional manual efforts. To reduce manual intervention, Roziere et al. (2020; 2021) use Koncept5 12 (Hosu et al., 2020) as a criterion to improve the quality of fake images. However, as discussed in Roziere et al. (2020), this strategy is somehow biased, can not work well in each dataset (e.g.. Pokemons). The reason is that the quality estimator not even designated to access the real-data distribution information. Instead, the generation is guided by predefined score function.
Compared to the above works, we take relevant idea to Roziere al. (2020) that also adapt the latent vectors to an improved condition before input to the generator G. Differently, our latent adaption approach incorporates the target distribution and utilize I-FGSM iterations to adapt latent vectors.
Mitigating mode collapse for GANs. There are also efforts in mitigating mode collapse, which are based on the thinking that many real-world samples lie in disjoint manifolds (Khayatkhoei et al., 2018). One way is to introduce collection of generators such that each generator may cover specific mode since a single generator can hardly fulfill a discontinuous function (Khayathhoei et al., 2018). However, more works focus on improving the properties of mapping, explicitly obtaining mapping covering more modes (Elfeki et al., 2019; Meulemeester et al., 2020; Mao et al., 2019). Note that the generated distribution obtained by this method will contain invalid parts due to the continuity of the latent distribution and the mapping. This calls for selective sampling of the continuous latent distribution or directly amending the latent distribution, which rarely noticed in these works.
Adversarial samples and adversarial training. Adversarial samples are malicious samples that can casily fool deep models, which are indistinguishable to human eyes. Since the seminal work (Szegedy et al., 2013), various adversarial attack methods have been devised (Goodfellow et al., 2014b; Akhtar & Mian, 2018; Kurakin et al., 2016; Madry et al., 2017). To defend the models from attacks, adversarial training is meanwhile developed (Goodfellow et al., 2014b; Wang et al., 2019; Nøkland, 2015). In particular, it is worth noting that several recent studies have shown that adversarial training can bring many benefits to GANs, not just improving robustness. RGAN (Zhang et al., 2020) shows that adversarial training can enhance the generalization of the generator and discriminator. Based on both theoretical analysis and empirical results, Zhong et al. (2020); Liu et al. (2021): Liu & Hsich (2019) show that updating discriminator's parameters with the loss calculated by adversarial samples generated by authentic images stabilize training and accelerate convergence.
3

---

Under review as a conference paper at ICLR 2022
Figure 3: Generation examples of StyleGAN2 ada (Karras al., 2020). Sampling equidistantly from p(z) in the latent space, the samples in the generated distribution p,, of the pixel space are shown here. The beginning and ending images of each row locate in different manifolds. The intermediate results are sampled by crossing between disconnected manifolds. The discontinuity causes weird generation in the transition, marked with yellow boxes (first row: the cat misses an ear, and its head is crooked; second row: white spots on the nose; third row: two bodies share one head).
In this paper, we utilize the adversarial sample search method, the iterative fast gradient sign method samples
(IFFGSM) (Kurakin et al., 2016) to conduct latent transform and mine optimization-frient in the GAN training process, which matches up with the procedure of adversarial training. While most existing works make perturbations on pixel space (real samples generated samples), instead we focus on mining the latent space and making perturbations on latent samples.
PROPOSED METHOD
This section presents AdvLa LatGAN- to achieve a transformation z" 2²()), and further AdvLatGAANqqua and AdwLaiGAN-div to obtain generation mapping with better generation quality (i.e., -qua) and diversity (i.e., -div). Both methods are derived from the iterative updating in latent space.
3.1 PRELIMINARIES
We introduce adversarial samples and adversarial training used to derive 2 (z) and devise training algorithms, and the regularization technique in MSGAN (Mao et al., 2019) for mode seeking.
Adversarial samples and adversarial training. Adversarial training aims to defend against adver- sarial attacks. It uses adversarial samples to calculate the optimized loss. In classification, for each image-label pair (x;, y:) E S from the labeled training set, the adversarial samples x' is defined as:
arg max 1(Jo(x)),) ||x Iily
Here e is the radius of the closed ball to constrain perturbation. While fo and I denote the attack target (a neural net) and the corresponding loss (e.g. cross-entropy loss). denotes the norm dimension. Given in total n adversarial samples, adversarial training is fulfilled by the optimization:
0 = min
(ffo(x).
(2)
0
Mode coverage by regularizing distance between generated samples. MSGAN (mode seeking GAN) (Mao et al., 2019) mitigates mode collapse in conditional generation setting by introducing regularization term for training G over two latent samples z and Z₂ as follows. The hope is that the two generated targets shall be far away from each other as much as possible:
Lms di (G(c, 21). G(c,zz))
(3)
d₂ (Zl, 22)
where e is the condition vector for generation, and d, and d, are the distance metrics in the target (image) space and latent space, respectively. encourages the distance to be maintained or even enlarged in the target space avoid the generated samples being trapped into aggregation.
4

---

Under review as a conference paper at ICLR 2022
Figure 4: Results of latent sample transform on the SNGAN model pre-trained on STL.-10. The original generated images are shown on the left, and the images generated with the newly found zs after 100 steps iteration of I-FGSM are shown on the right. We notice that the right figures contain richer colors and more realistic details, with the main content remaining consistent. We measure generation quality in a batch by Inception Score, which increases from 6.944 to 7.475.
By integrating the regularization term with the original objective function of eGANs (Mirza Osindero, 2014), the final objective for minimization in MSGAN for training G and D is given by:
L =Lors + Ams AmsLms
(4)
Cori E. (log (ee x)] + Re,, [log(1 - D(c, G(c, z)))]
where Lori denotes the original objective, x denotes real data, and Ams is the weight hyper-parametter Note since Eq. 3 is aimed to be maximized hence one can interpret Ams to be negative.
3.2 LATENT SPACE ADVERSARIAL. SAMPLE SEARCH FOR GENERATION QUALITY
We show how to implicitly transform the latent raw samples from Gaussian (or other popular forms e.g. uniform) to new distribution via the idea of searching adversarial samples as introduced above. This transform breaks the dilemma in diversity and quality where the bottleneck is due to the continuous latent distribution and continuous mapping function. Moreover, conducting the implicit transform in training to obtain the new (probably disconnected) latent distribution for each iteration can compensate for the difficulty for generators to align the pace of generators and discriminators! which cannot map continuous distribution to disconnected one with multiple modes.
Specifically, we design the following constrained optimization task that encourages the new z" to be close enough² to z. Its prediction score (as real data) by the discriminator D should be as high as possible to improve the generation quality. Recall otherwise the given zu may correspond to a poor generation as discussed in Sec. 1. Note here that both G' and D are fixed.
z (zo) = arg min log(1 - D(G()))
(5)
=E =E(
where zg denotes the original latent vectors, x (zn) denotes the newly computed z, d(-) denotes the distance of two vectors which is an foo distance in this paper, in line with the popular protocol in adversarial attack literature (Goodfellow et al., 2014b).
By borrowing the well-developed tools in adversarial attack, the solution of the above problem can be estimated by the so-called iterative fast gradient sign method (I-FGSM) (Kurakin et al., 2016):
Zi+1 + 2, esgn (V n, log(1 D(G()))))
(6)
From the distribution perspective, the transform can be characterized by:
p== = arg
min Ex-p, [log(1 - D(G())))
(7)
D(
where p denotes the original latent distribution, P denotes the newly searched one. D(p,, p2) S means that there is a limit to the distribution variation. A newly searched z can be considered as a sample from D. if the sampling is within the perturbation circle. Fig. 4 shows the results by the pre-trained SNGAN (Miyato et al., 2018) (i.e. the given and D) on STL-10 (Coates et al., 2011) 1).
We develop a novel GAN training algorithm by conducting the above latent distribution shift to the initial sampling of Gaussian in D updating iterations, while the rest of the training algorithm remains consistent with vanilla GAN (Goodfellow et al., 2014a). We present the algorithm in Alg. 1 which we call Adversarial Latent GAN (AdvLatGAN-qus. with a post meaning quality. The latent tranform is first computed using the I-FGSM to find the adversarial samples in latent space, then D and G are updated. The iteration continues until enough steps or convergence of G.
In literature, it widely recognized that it much more challenging to train the generator than discriminator. We do not want to departure too much from the basic GAN training protecol, and also the transform shall keep mode consistent maintain overall diversity.
5

---

Under review as a conference paper at ICLR 2022
Example 1
Example 2
Example
Example4
Example 5
Example 6
6(2
Hard Pair
Bqg 8
6(x)
Basy Pair
Eq, 8-inv
(()
Figure 5: Each of the six columns shows two compared pairs of generated example (first pair: 1st + 2nd row; second pair: 3rd row 2nd row) by our diversity driven iterative transform scheme in latent space with the same number of iterations to obtain 2* from initial z. Middle row: generation by vanilla latent sampling z: Top: by latent sample transformed by Eq. 8: Bottom: by latent sample transformed by Eq. 8's inverse form. It shows using Eq. 8 generates more similar image pairs serving as hard samples for training, rather than using its inverse form (see quantitative results Table 1).
The above latent transform model does not consider how to avoid mode collapse, as each time only one latent sample is considered for optimization in isolation. Similar constrained optimization formulation can be designed to improve the generation diversity, as shown in the following subsection.
3.3 LATENT SPACE ADVERSARIAL SAMPLE SEARCH FOR GENERATION DIVERSITY
We take a hard sample mining perspective to improve mode coverage. Given a sample Zg from the raw latent distribution, we aim to search its paired sample x² to form hard sample pair, in the sense that their generations are close in the target space given G. We consider two aspects. First, as hard samples, the paired samples shall still belong to the same category which can be modeled with conditional GAN. Second, their distance in the latent space shall also be not too close otherwise, they may be aggregated in target space due to continuous mapping. In other words, hard samples causing mode collapse shall be those close-in target space while apart in the latent space, when G is given. Based the above considerations, adopt the form of Eq. 3 to design our search procedure:
dr(G(c, 2o), G(c,))
z" (Zn) = arg min 22)
d₂ d=(20, z)
The symbols and terms are similar to Eq. 3. Akin to AdyLaiGAN-qua, we use I-FGSM to obtain z" and again use foe to control change magnitude. For each update step, we randomly select the first latent vector Zu. and the second vector x is searched by iterations of the I-FGSM method under the foo restriction starting from the neighborhood of zo: and the iterations are guided by Eq. 8.
Table 1: Ratio between image and latent distance by Eq. to derive new x" -nnv means using inverse form.
Datasel
Eq. 8
Eq. 8-inv
CIFAR-10 10
0.241
4.023
FACADES 0.3785
1.3385
Table shows the effectiveness of the above paired complex sample seeking approach, with ex- periments on CIFAR-10 (Krizhevsky al., 2009) and Facades (Tyleček & Šára, 2013) with DC- GAN (Radford et al., 2016) backbone. The visual results on Facades are shown in Fig. 5. We can see that the pair obtained by solving Eq. 8 tend to collapse (as they are hard samples), while the opposite leads to better diversity by using the inverse form of Eq. 8.
The above experiments show the effectiveness of our iteration method, and we further introduce it into the initial training process of MSGAN (Mao et al., 2019). In MSGAN, the regularization term 3 used to regularize the model optimization process, where the selection of the pair is entirely random, without using any heuristic information. We instead use the pair that more inclined to collapse by randomly taking one z and iterating by the method mentioned above to get another z. We keep the other settings of the algorithm unchanged.

---

Under review as a conference paper at ICLR 2022
Table 3: Post-training latent sampling improvement with two architectures on STL.-10.
SNGAN (two initials)
WGANGP (two initials)
Architecture
Method
IS(1)
FID(I)
IS(1)
FID(I)
1S(t)
FID(J)
IS(1)
FID(C)
Original
7.244
64.573
6.285
46,076
7.287
73.625
6.791
47.765
ResNet
AdyLaiGAN-z
8.116
62.184
6.833
43,099
7.743
73.443
7.369
45.121
Original
6.183
46.701
7.344
64.289
6.701
48.454
7.112
69.263
DCGAN
AdvLaGGANz
6.748
43.386
7.860
64.165
7.376
46,490
7.767
68.929
By using Eq. 8 to generate adversarial sample pairs to address mode collapse, we finally develop our new diversity enhancing GAN training approach AdvLaiGAN-di as shown in Alg, 2 in the appendix. which are further enhanced by post-training as discussed in experiments.
4 EXPERIMENTS
Experiments are performed on single GPU of GeForce RTX 3090. All datasets used in experiments are publicly available.
ADVLaGANS
4.1 EXPERIMENTAL. SETUP
We validate the effectiveness of the proposed methods for two parts, respectively for effective latent sampling improvement and better generation mapping G. In particular, five methods can be derived for different targets, among which AdvLatGAN- qua+ and AdvLatGAN-div are the final full version of our contributions. We present the methods below:
Figure 6: The logic of our 5 variant methods in Sec. 4.1. Blue and red for quality and diversity.
i) AdvLatGAN-z:
post-training latent sampling improvement fighting against quality discontinuity; ii) AdvLatGAN-quas GAN training algorithm for better generative quality using in- training latent sampling improvement; iii) AdvLatGAN-div:
Table 2: Components of our 5 vari- ant methods in Sec. 4.1.
Latent
Improved
Transform
mapping
GAN training algorithm for a more diverse generation using in-training latent sampling improvement: iv) AdvLatGAN- qua+: both in and post-training latent sampling improvement for generation quality: v) AdvLatGAN-dive N-div++ both in and post- training latent sampling improvement for generation diversity.
ii) iii)
iv)
For illustrative comparison, we show the schematic overview of these methods in Fig. 6 and Table
We adopt Inception Score (Salimans et al., 2017) and Fréchet Inception Distance (Heusel et al., 2017) to evaluate the performance, which utilizes the classification results of the generated images by the Inception Network to evaluate the performance of the model. Following (Richardson & Weiss, 2018). JSD is adopted to measure the similarity between generated images and real images, which first clusters real images and generated images into the same number of classes using the K-means algorithm, then calculate the similarity between real and fake clusters w.r.t. the clustering property. We also adopt recently proposed evaluation metrics density/coverage (Naeem et al., 2020), which separates the fidelity evaluation and the diversity evaluation.
4.2 ADVLATGAN-Z: POST-TRAINING LATENT SAMPLING IMPROVEMENT
This section shows the effectiveness of the post-training latent sampling improvement AdvLatGAN-z (denoted as 2n in previous sections, which is an implicit transform on the original latent sampling from Gaussian), which aims to mitigate the issue of quality discontinuity.
Results on Synthetic Data. In line with (Metz et al., 2017), we apply the proposed iterative strategy guided by Eq. 5 in both Grid and Ring datasets to evaluate its performance. By up- dating latent space variables, we achieve high- quality generated samples. The results in Fig. show the effectiveness of our method.
Origiinal
Quality enhnneed
Original
Quaiity enbanced
Figure 7: Post-training latent sampling improve-
ment Grid and Ring by AdvLatGAN-
Results on MNIST. MNIST contains 70,000 images of handwritten digits (LeCun et al., 1998), which can be generated from two-dimensional latent space in which the iterative latent distribution
7

---

Under review as a conference paper at ICLR 2022
pt STEP
P2 STEP 30
pf STEPO
STEPa
STIP
STEP 10
STEP3
STEP
STEP 10
STEP
1 4
Co
Proks
Valless
Figure 8: Latent distribution transform with WGAN-GP/DCGAA as backbone on MNIST. As the I-FGSM iteration continues, the latent distribution deviates from the standard Gaussian, and contains more valleys and peaks which can be mapped into generated images as shown in the bottom. Note the generations by the valley points are of low quality and they can be effectively avoided by scheme as the sampling probability is low (valley) in our transformed distribution.
Table 4: The results of AdLLaIGN--uua and AdvLa for CIFARIO.
Inception (core(t)
Fréchet Inception Distance(L)
Architecture
vanilla
AdvLatGAN-qui
AdvLaGGAN-qua+
vanilla
AdvLa
AdvLaGGAN-qua+
DOGAN
0.05
6.21 0000
6.58=0,34
46,4
41.7
40.1 1,5
WGAN
0.05
7.04
7.76 0.07
32.8=0.5
2.7
27.2 0.5
WGAN-GP
7.47 1- 0.09
7.60 0.06
8.59 0.10
24.7+0.2
22.6 0.4
18.3 1.1
SNGAN
7,29
7.58
0.96
25.5
2..
0.3
LSGAN
5.87 4-014
6.13 + 4-0.7
6.38 0.18
49.3 4.2
42.8 +13
41.0 + +-1
dGA
7.43
20.6
ACGAN
0.228
6.06 0.21
6.21 = 0.30
59.5 1.5
53.7
1.6
53.1 1.4
Table 5: The results of AdvLaiGAN-qua and AdyLaiGAN-qua+ for STL10
Inception Score(1 1)
Fréchet Inception Distance(L)
Architecture
vanilla
AdvLatGAN-qua
AdvLarGAN-qua+
vanilla
AdvLatGAN-quu
AdvLatGAN-qua+
DCGAN
7.18
1.3310007
0,06
612212
16.31100
$4.6
WGAN
6,51
0.64
8.16 0.27
73.0=0.2
51.0
49.1 1.2
WGAN-GP
8.86 0.00
0.90
10.32
3.4=004
0.
26.7 1.1
SNGAN
8.49 0.09
8.63++0.00
9.37 0.05
36.8 0.4
34.5++0.21
30.9 0.4
LSGAN
7,08 1.112
7.16 1.155
7.55 0,18
62.9
1.2
58.5 1.3
57.1 1:555
WGAN-div
8.82
9.00
1 0.01
10.68 0.17
37.7
0.2
32.0
#.66
24.3 1.0
change is illustrated in Fig. 8. We respectively conduct the experiments for the trained DCGAN model and WGAN- (Gulrajani et al., 2017) model, with single-step iteration constraint using foo and iterated step size+ e set to 0.03. To better reflect changes in the distribution, we only show the first 30 iterations that can reflect larger changes. For WGAN test, Fig. 8 shows the transformed latent distribution which contains more peaks and valleys and is different from the raw Gaussian.
Results on CIFAR-10 and STL-10. We run experiments and STL-10 (Coates et al., 2011). CIFAR-10 10 has been widely used with more diverse and closer to the quality of high-resolution images than MNIST. STL-10 10 has higher resolution and richer image representation. We use implicit transform in the latent space for pre-trained DCGAN and SNGAN respectively, with the single-step iteration constraint using foc and iteration step size is set 0.01. We conduct 100 steps for each iteration process. Inception Score and Fréchet Inception Distance are used to evaluate the generation quality, which are calculated over 1500 generated images for STL-10 and over 3000 images for CIFAR-IO, 10. For each of the four cases, we use two different initializations, each initialized randomly and not explicitly chosen. As shown in Table 3, our method has a significant impact on generation quality, with a maximum improvement of 12.03% by Inception Score.
CIFAR-10 (Krizhevsky et al., 2009)
Results on AFHQ and FFHQ. To show the effectiveness of AdvLaIGAN-z solve issues in Fig. 3. we evaluate on AFHQ (Choi et al., 2020) and FFHQ (Karras et al., 2019b) based on the StyleGAN2- ada (Karras al., 2020) model. Note that AdvLatGAN- works at the sample level (or distribution level as a distribution shift characterized by sample transforms) in the latent space. In the Fig. 3 setting, when the sampling encounters invalid examples, AdvLaiGAN- able to transform the bad to x (z) with better ability to avoid bad generation. Here we verify whether invalid samples can be improved with AdvLatGAN-z. We select six bad generations with the first three correspond to those in Fig. 3 and conducting AdyLatGAN-z to their latent vectors. The results show that AdyLatGAN-z can effectively fix or avoid image defects, as shown in Fig. 9.
4.3 ADYLATGAN-QUA/DIY IMPROVING THE GENERATION MAPPING
This section shows the effectiveness of the GAN training algorithms, which aims to improve the generation mapping G for better performance respectively on quality and diversity.
8

---

Under review as a conference paper at ICLR 2022
Figure 9: Results of AdvlatGAN-z AFHQ and FFHQ The first row had generations (the first three correspond to those in Fig. 3) with the defects as follows: first column: the cat misses an ear, second column: white spots on the nose; third column: two bodies share one head; fourth column: green face; fifth and sixth columns: semi-existing glasses. The second row are results under AdvLa The results show that AdvLatGAN- can effectively mitigate image defects.
Table 6: FID and JSD results on CIFAR-10.
10.
Metrics
Models
overall
sirplane
sutommoblle
bird
deer
dog
Frg
ship
truck
FID(A)
MSGAN
30.235
73.0K3
69.118
75.258
M4.335
63.287
69.
26.763
70,142
69.642
75.922
66.507
53.725
83912
54,993
64.943
69.715
62.173
JSD(J)
AdvLaGAN--dvv
MISIAN
0,00542
B.8075
0,02694
0.00044
0.02201
0302337
03036
0/02795
0.00623
0.02571
0.02835
IU0222
0.02305
0.03670
070381
0/02776
0.4140
0.83548
0.03635
01157%
0.03210
densiry(1)
MSGAN
0.350
0.516
0.338
0.333
0.216
0.255
0.366
0.267
0.635
0.532
0.511
0.272
0.318
0.665
0.674
0.296
0.331
0.353
0.224
0.231
coverage 1)
AdvLaGAN--ive
MSGAN
0.565
0.726
0.65
0.677
0.600
0.752
0.460
0.464
0.874
0.748
0.736
0.797
0350
0.466
0.776
0.843
0.743
0.779
0317
0.577
0.612
0.004
Table 7: FID and JSD results on STL-10.
Matries
Models
simplane
bird
dog.
monkey
trick
FIN()
Ad-LaGAN-Jiv+
MIGGAN
67.549
65.349
92.160
02.021
124.155
100 3590
118318
118,385
105.966
111.754
132.681
139.372
117.149
121507
101
95.298
99.590
JSD(1)
AdvLaaAAN-div
MSGAN
0.00661
0,00423
0.02965
0.02852
0.02515
0.02717
0.03177
0,88638
0.03849
0,03518
0.03822
0.03733
0.03107
0.03132
0.03996
0
0.0391
0.83196
0.02681
0.02239
0.03157
0.02751
donsity (13
AdyLaGAN
MSGAN
0.332
0.443
0.192
0.264
0.328
0.089
0.307
0.429
0.150
0.211
0.251
0.116
0.126
awwnng(1)
AdyLaGAN--iv-
MSGAN
0.361
0.396
0.383
0.405
0.329
0.364
0.236
0.169
0.118
0.508
0.424
0.306
0.309
0.289
0.390
0271
0.250
0.335
0.39
0.395
0226
0.395
AdvLatGAN-qua for Quality Improvement. We conduct experiments on CIFAR- and STL-10, using the mainstream architectures including DCGAN, WGAN, WGAN-GP. SNGAN, LSGAN, WGAN-div (Wu et al., 2018) and ACGAN (Kang et al., 2021). We do not include ACGAN in the STL-10 setting because it does not work on unlabeled dataset. We adversarially train GAN using a quality-enhancee strategy previously mentioned as Advl AdvlaatGANqqu. and the results show that the proposed method can greatly improve the quality of generated images. IS and FID are adopted to evaluate the performance of the model. We conduct only one latent iteration to save computation cost per generator step, and more details will be given in the appendix. We also try to use implicit transform after training improve the generation quality further. The result is shown in Table 4 and Table 5. It can be seen that the proposed method achieves a significant improvement in generative performance compared that without the latent space transformation strategy. In the WGAN STL-10 setting, IS is improved from 6.51 to 8.16, and FID from 73.0 to 49.1.
AdvLatGAN-div for Diversity Improvement. We conduct experiments on both CIFAR-1O 10 and STL-10 for conditional generation and adopt FID, JSD, density and coverage as the metrics, re- spectively for the entire set of generated images and each class. We test the training results on the basis of implicit latent transform to fight against quality discontinuity. We can find that the proposed method is prior over MSGAN reflected by JSD and FID metrics. The results are shown Table and Table 7. For the overall set which models are trained, AdvLatGAN-div has achieved at most 1 11.5% improvement FID, 36.0% JSD, 47.4% in density and 27.8% in coverage.
5 CONCLUSION
This work begins with two basic observations in GANs from the perspective of continuous mapping. Based on the observations, we introduce the adversarial attack techniques in GAN to explore latent space and derive novel training algorithms to improve the mapping property. With the efforts of both latent transform and mapping improvement, the proposed AdyLatGAN has shown promising ability for achieving generation with both quality and diversity verified by extensive experiments.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: survey. leee Access, 6, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICMI., 2017.
Philip Bontrager, Wending Lin, Julian Togelius, and Sebastian Risi. Deep interactive evolution. In EYoMUSART. Springer, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: /096, 2018.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey, arXiv preprint arXiv: 1810.0008 2018.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-basee -based model and you should use discriminator driven latent sampling. arXiv preprint vrXiv:2003.06060 2020.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS. JMLR Workshop and Conference Proceedings, 2011.
Mohamed Elfeki, Camille Couprie, Morgane Riviére, and Mohamed Elhoseiny. Gdpp: Learning diverse generations using determinantal point processes. ICML, 2019.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 14 1222.6522 2014b.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In NIPS, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NIPS, 2017.
Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing. 29, 2020.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks. arXiv preprint arXiv: 1r9.v0190..0717. 2019.
Minguk Kang, Woohyeon Joseph Shim, Minsu Cho, and Jaesik Park. Rebooting acgan: Auxiliary classifier gans with stable training. In NIPS, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019a.
Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019b.
Tero Karras. Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.0660 2020.
Mahyar Khayathhoei, Ahmed Elgammal, and Maneesh Singh. Disconnected manifold learning for generative adversarial networks. In NIPS, 2018
10

---

Under review as a conference paper at ICLR 2022
Alex Krizhevsky, Geoftrey Hinton, et al. Learning multiple layers of features from tiny images. Tech Report, 2009.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples in the physical world, 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, 2017.
Faqiang Liu, Mingkun Xu, Guoqi Li, Jing Pei, Luping Shi, and Rong Zhao. Adversarial symmetric gans: Bridging adversarial samples and adversarial networks. Neural Networks, 133, 2021.
Xuanqing Liu and Cho-Jui Hsieh. Rob-gan: Generator, discriminator, and adversarial attacker. In CVPR, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1eXXv:1706.0688 2017.
Qi Mao, Hsin-Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In CVPR, 2019.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks. In CVPR, 2017.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In ICLR, 2017.
Hannes Meulemeester, Joachim Schreurs, Michael Fanuel, Bart Moor, and Johan Suykens. The bures metric for taming mode collapse generative adversarial networks. In CVPR, 2020.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 2014.
Deepak Mishra, Aravind Jayendran, and AP Prathosh. Effect of the latent structure on clustering with gans. IEEE Signal Processing Letters, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan, Clustergan: Latent space clustering in generative adversarial networks. In AAAI, 2019.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In ICML. PMLR, 2020.
Arild Nøkland. Improving back-propag by adding an adversarial gradient. arXiv preprint arXiv: 1510.04189 2015.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In ICML. PMLR, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. ICLR, 2016.
Eitan Richardson and Yair Weiss. On gans and gmms. arXiv preprint arXiv: 1805. 0055/2242,, 2018.
Baptiste Roziere, Fabien Teytaud, Vlad Hosu, Hanhe Lin, Jeremy Rapin, Maria Zameshina, and Olivier Teytaud. Evolgan: Evolutionary generative adversarial networks. In ACCV, 2020.
11

---

Under review as a conference paper at ICLR 2022
Baptiste Roziere, Nathanaël Carraz Rakotonirina, Vlad Hosu, Andry Rasoanaivo, Hanhe Lin, Camille Couprie, and Olivier Teytaud. Tarsier: Evolving noise injection in super-resolution gans. In ICPR. IEEE, 2021.
Salimans, I. Goodfellow, W. Zaremba, Cheung, Radford, and X. Chen. Improved techniques for training gans. In NIPS, 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning, In NIPS, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv: 1312.6/99, 2013.
Radim Tyleček and Radim Sára. Spatial pattern templates for recognition of objects with regular structure. In Proc. GCPR, Saarbrucken, Germany, 2013.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In ECCV Workshops, 2018.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, volume 1, 2019.
Jiging Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Lue Van Gool. Wasserstein divergence for gans. ECCV, 2018.
Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv preprint arXiv:1912.00953, 2019.
Fangneng Zhan, Hongyuan Zhu, and Shijian Lu. Spatial fusion gan for image synthesis. In CVPR, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. ICML. PMLR, 2019.
Shufei Zhang, Zhuang Qian, Kaizhu Huang, Jimin Xiao, and Yuan He. Robust generative adversarial network. arXiv preprint arXiv:2004.13344, 2020.
Jiachen Zhong Xuanqing Liu, and Cho-Jui Hsieh. Improving the speed and quality of gan by adversarial training. arXiv preprint UrXiv:2008.03366 2020.
Brady Zhou and Philipp Krähenbühl. Don't let your discriminator be fooled. In ICLR, 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. ICCV, 2017.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A DETAILED ALGORITHM FOR ADVLATGAN-QUA AND ADVLATGAN-DIV
The specific algorithm for AdvLaiGAN-qua in Sec. 3.2 is presented in Alg.
Algorithm 1 AdvL.ttGAN-quee (2, Ip: the number of steps, other symbols are consistent with vanilla
GAN.
Input: Ps c.g. Gaussian, P, c.g. real images distribution; randomly initialized G and D; Output: trained generator G' and discriminator D;
1: while G has not converged do
2: // update D):
3: for i to to do Sample {z)) (2()))m 1m jk=1 ~ Pzi
4:
5:
// the proposed sampling shift (t-step I-FGSM): for i 1==10
6:
do
7:
Obtain {z, jk=1 by Eq.
(k)
Sample {x, 1m
8:
/k=1 ~ Pri
Calculate loss with {x, (1) ym
and {z) (a)
9:
Je=
10:
Update D with lossp:
11:
// update G:
12:
Sample (x(*) jk=1
Psi
13:
Calculate lossc with (m
14:
Update G with lossg:
The specific algorithm for AdvLarGAN-div in Sec. 3.3 is presented in Alg. 2.
Algorithm 2 AdyLatGAN-div tp: the number of steps, other symbols are consistent with vanilla GAN.
Input: p. e.g. Gaussian, pr e.g. real images distribution; Ams hyper-parameter, randomly initialized G and D:
Output: trained generator G and discriminator D:
1: while has not converged do
2: // update D:
3: for i= 1 to Ip do
4: Sample {z. 1
k=1 p=; Sample {(x)
(8)
5:
}1m ~ Pri
Compute lossp with {(x) (K) (k)) (k) m
6:
c z 1k=1
7:
Update D with lossD:
8:
I| update G:
9:
Randomly generate labels (m)
Sample (zo (k)mm // the proposed sampling shift (t-step I-FGSM);
kk=1 p2:
11:
12:
for i 1 to do
Obtain (2) M by Eq. 8; Calculate lossg and lossme (Eq. 3) with {m()) loss +- losse + Amsllsssss
13:
JR=1 1z pm and fz, (m)
14:
fk=13
Update G with loss;
B RELATIONSHIP WITH ADVERSARIAL ATTACKS AND DEFENSES
Our approach is closely related to adversarial attacks and defenses. As for works in adversarial attacks, the fundamental standpoint that deep neural networks' performance can vary significantly in the face of small perturbations. This scenario can be very similar to starting point described
13

---

Under review as a conference paper at ICLR 2022
in See I that small perturbation of the latent vectors can lead to massive quality variation. Thus it is quite natural to combine the adversarial attacks and defenses with generative adversarial networks, and specifically, utilize the latent transform using I-FGSM iterative method to overcome the quality discontinuity.
Our GAN training algorithms AdvLatGAN-qua and AdvLatGAN-diy can be analogous adversarial training techniques in the adversarial defense field. Adversarial training is a proactive defense approach that strengthens the model against attacks enhances its performance by modifying the inputs to use adversarial samples to train the model, making the model naturally defensive against attacks. This process involves conducting adversarial attacks during training and can be explained as mining targeted optimization friiindly samples for training. While in AdvLatGAN- and LddLattAN-div, we conduct z iterations the GAN training referencing attack method, which can be viewed as the same process as adversarial training. AdvLatGAN introduces the latent vector z iterations into the bi-level optimization between the generator G and the discriminator D. The training involves iterations of three components z, G and D, among which the introduced z iterations can mine the latent space during training, to some extent leading to the superiority of the method.
C DESCRIPTION OF CONSTRAINTS FOR ITERATION
There are two basic types constraint used in the field of adversarial attack: 1) & norm constraint; 2) &o norm constraint. In this paper, we follow the well-known attack method, iterative fast gradient sign method (I-FGSM) (Kurakin et al., 2016) to conduct the iterations of latent variables and we adopt the fo norm as our constraint.
D DIFFERENCES AND COMPARISONS TO OTHER ALGORIHTM
We present the differences between our approach and other previous works, involving two topics i.e. latent exploration and GAN model with adversarial training. An overview is presented in Table 8. Meanwhile, we conduct experiments to show our superiority over other representative works.
D.1 LATENT EXPLORATION
EvoIGAN (Roziere al., 2020), Tarsier (Roziere et al., 2021). These two models both use the well-knoww quality evaluator Koncepi512 (which is actually classifier) to guide the latent iterations. Their drawback is that they do not exploit the information of the real distribution, which leads that the optimization criterion of sampling quality is not for the proximity to the real distribution, but to match the data trained for the Koncept51 12 classifier. Our approach uses a discriminator to guide the latent variable updating, which makes full use of the information from the real distribution, while further achieving novel training algorithm to improve the generative mapping.
DDLS (Che et al., 2020). DDLS analyzes the sample-improving approach from the perspective of the energy-based model, which differs from ours terms of methodology. Besides, it didn't 't notice that when enhance the quality, we hope that the image is changed little possible. Combing the above two points, we explain the problem from the perspective attack, which quite different from the specialized sampling approach Langevin Dynamics used in (Che al., 2020).
D.2 GAN MODEL WITH ADVERSARIAL TRAINING
Our GAN training algorithm can be considered as kind of adversarial training from the perspective of adversarial attack and defense, and we differ from several previous works that combine adversarial training and GAN.
ASGAN (Liu et al., 2021), FastGAN (Zhong et al., 2020), Rob-GAN (Liu & Hsieh, 2019). These methods add perturbation to the real images (the latter is improvement in the loss function compared to the former), while we focus on the latent space, perturbing latent space vectors.
Robust GAN training (Zhou & Krähenbühl, 2018). This algorithm shows that robust discrimina- tor can benefit training and this robustness only need to enforced in expectation over the generated samples, again without focusing the latent space compared to us.
14

---

Under review as a conference paper at ICLR 2022
Table 8: Differences from Other Algorithms
Detailed Description
Viewpoint
Method
Different point
Theirs
Ours
Guide of the transform
Quality estimator
D.G
EvoIGAN
Koncept5
Achieving method
Evolutionary
I-FGMM iterations
Algorithm
Guide of the
Quality estimator
D.G
Latent Exploration
Tarsier
transform
Koncept5
Achieving method
Diagonal Covariance
IFFSMM iterations
Matrix Adaptation
Targeted Task
Super-resolution
General generation
generation tasks
tasks
Timing for the
Before training
After training G and
AB-OT-GAN
transform Achieving method
and D Firsttrain
T-GGMM fterations
Auto-Fncoder to
learn latent
distribution then use optimal trasport to achieve the transform
Calculating cost
Need to train an Auto- Muto-EEcoder for finding the latent
No extra networks are trained (guided by and D)
distribution
Basis theoreticul
Energy- based theory
General GAN theory
DDLS
analysis Achieving method
Markov Chain Monte
FFGSM lterations
Carlo
quiLuiGAN-qui
vanilla GAN
the training D
Samples from
Transformation
Gaussian
the original sampling
AdvLaiGAN-div
MSGAN
pair used in ms-regularization term
Individually sampled
Randomly choose 21, then transform get zg, forming a pair
from Gaussian
Adversarial Training
Common differences
Objects to which perturbations are added
Table 9: Experimental comparisons over GAN with adversarial training evaluated by Inception
Score.
Evolgan
Evolgan (a=0.5,b=80)
Evolgan
Evolgan
Framework
DDLS
AdvLaiGAN-Z
(a=0,b=80)
(a=0.5,b=20)
(a=1.b=20)
DCGAN
5,815
5.887
5.825
5.791
5,502
6.086
WGAN
6.528
6.496
6.521
6.546
2.607
7.334
WGAN-GP
7.349
7.316
7.441
7.236
7.074
8.279
SNGAN
7.195
7.274
7.298
7.221
6.879
7.832
D.3 EXPERIMENTAL COMPARISON OVER LATENT EXPLORATION
We compare our methods AdvaaGAANz with two representative works Evolgan and DDLS that combine latent exploration and GANs. For Evolgan implementation, we choose four sets of hyper- parameters for evaluation. Note that we did not find the official codes for Evolgan and DDLS for real-world experiment, so we reproduce the methods according to their papers. The experimental setting is aligned with Table 4 and we evaluate on CIFAR-10 10 with four framworks: DCGAN, WGAN, WGANGP and SNGAN. We conduct AdyLaIGAN-z on 10000 generated images batch and then evaluate the results. Inception Score and Fréchet Inception Distance are adopted as the evaluation metrics. The experimental results are presented in Table 9 and Table 10.
D.4 EXPERIMENTAL COMPARISON OVER GAN MODEL WITH ADVERSARIAL TRAINING
We compare our methods AdyLatGAN qua and AdyLaiGAN -qa++ with two representative works ASGAN and Robust GAN training that combine adversarial training and GANs. These two methods are related to perturbations on the generated samples and the real samples respectively. The experi- mental setting is aligned with Table and evaluate CIFAR-10 with four framworks: DCGAN,
15

---

Under review as a conference paper at ICLR 2022
Table 10: Experimental comparisons over GAN with adversarial training evaluated by Fréchet
Inception Distance.
Evolgan
Evolgan
Evolgan
Evolgan
Framework
DDLS
AdvLatGAN-z
(a=0,b=80)
(a=0.5.b=80)
(a=0.5,b=20)
(a=1.b=20)
DCGAN
49.237
48.428
49.080
48,945
58.133
45.763
WGAN
34.685
34.650
34.564
34.565
221.367
32.854
WGAN-GP
27.004
26.252
26.497
26.492
32.516
22.870
SNGAN
27.661
27.418
27.542
27.542
27.542
23.531
Table 1I: Experimental comparisons over GAN with adversarial training evaluated by Inception
Score.
Framework
ASGAN
RobDis
AdvLatGAN-qua
AdvLatGAN-qua+
DCGAN
6.21 0.07
6.03
6,28 0.04
6.58 0.34
WGAN
0.01
6.84
7.21 0.04
7.76 0.07
WGAN-GP
0.00
7.41
7.60 0.06
8.59 - 0.10
SNGAN
7.21 0.03
0.03 + 0.06
7.58 0.03
8.13 + 0.06
Table 12: Experimental comparisons over GAN with adversarial training evaluated by Fréchet Inception Distance.
Framework DCGAN
ASGAN
RobDis
AdyLatGAN-qua
AdvLatGAN-qua+
45.3
41.9 0.4
41.7
40.1 1.5
WGAN
1.3
1.4
0.7
27.2 0.5
WGAN-GP
0.3
2.1
22.6 0.4
18.3 1.1
SNGAN
26.7
22.3 0.5
21.9 + 0.3
WGAN, WGANGP and SNGAN. Inception Score and Fréchet Inception Distance adopted the evaluation metrics. The experimental results are presented Table 11 and Table 12.
E DETAILS OF LATENT SAMPLING IMPROVEMENT EXPERIMENT
E. PROTOCOLS FOR SYNTHETIC EXPERIMENTS.
We simulate two synthetic datasets. Ring dataset is mixture of 8 2-D Gaussians p(z) with mean {(2 cos (ir/4), 2 cos (iTT (r//4)) and standard deviation 0.001. 12.5K samples are simulated from each Gaussian distribution. 50K samples from p(z) used to generate for test. Grid dataset is mixture of 25 2-D isotropic Gaussians i.c. p(z) with mean {(2i, 2;) and standard deviation 0.0025. 4K samples are simulated from each Gaussian. 20K samples from p(z) are used to generate target samples {x} for test. In the synthetic experiments, we apply fully-connected networks for generation and the architectures are shown in Table 13 and Table 14.
Table 13: Architecture of generator G.
Table 14: Architecture of discriminator D.
Layer
Output size
Activation
Layer
Output size
Activation
Linear
100
ReLu
Linear 100
ReLu
Linear
200
ReLu
Linear
200
ReLu
Linear 100
ReLu
Linear
100
ReLu
Linear 2
Linear
E.2 DETAILS FOR RING SYNTHESIZING EXPERIMENTS.
To better demonstrate the practical effect and to show the characteristics of the network during the training process, we choose 2000 as the total number of iterations (1 epoch), and the result is shown in the left picture in Fig. 10. We apply the latent transform technique for the network in the left picture: frozen all parameters of the neural network and iterating 20,000 points sampled in 2D 2D-Gaussian distribution under the constraint of L-infinity norm. The step size is set as 0.003 and its optimizing target is upgrading discriminator judged score. After iterating 8,000 times, We get the high-quality
16

---

Under review as a conference paper at ICLR 2022
(a) Insufficient Train
(b) Quality-enhance
(c) Sufficient Train
Figure 10: Left: points generated by the generator after training for epoch. Middle: apply latent iterative sampling strategy for the generator shown in the left image. Right: generated points after training 100 epochs.
(a) vanilla points
(b) iterate 10 times
(c) iterate times
(d) iterate 8000 times
Figure 11: Results of using AdyLaiGAN-qua
samples in Fig. 10 (shown in the middle) which outperform the results after training 100 epochs. It is a powerful evidence to prove the problem of imbalance the input latent space and that discriminator tends converge faster.
E.3 DETAILS FOR GRID SYNTHESIZING EXPERIMENTS.
We try to simulate the case that the inappropriate selection of network architecture to prove the application prospects latent transform this situation. All hyperparameters are consistent with the previous ring experiment besides choosing training 50 epochs to make sure the network is fully trained. Because of the inappropriate choosing model architecture, the training result terrible. However, by applying with latent transform policy, almost all points converge towards target points. The result is shown in Fig. 10.
E.4 DETAILS FOR MNIST EXPERIMENT.
The DCGAN model and WGAN-GP model are used in this experiment. We DCGAN model for generating images of CIFAR-10 and ResNet model for STL-10 dataset. The hyperparameters and model structures are consistent with that used in https: /t//tttttt thub. com w8 6 pytorch-gan- cole-collcctiic ions with total of 30 iterations, each with step size of 0.03 and batch size of 8000. The configuration of the trained DCGAN model: batch size is 128, D-learning rate and G-learning rate are both 0.0002, the loss function is bee loss, the discriminator updates one step per generator iteration, the latent dimension 2, and the total training step 20,000. The configuration of training WGAN-GP model: the batch size is 128, the D-learning rate and G-learning rate are both 0.0002, the parameters of discriminator updates one step per generator iteration, the latent dimension is 2, and the total training step is 50,000.
Details for CIFAR-10 and STL-10 Experiment. used in this experiment, referencing https: / thub com/ ht0m//w676e0 w86763777/ pytorch gan- www.oollllciooooo with a total of 100 iterations, each with step size of 0.01 and the batch size is set 1500 for STL-110 and 3000 for CIFAR-10.
The SNGAN and WGAN-GP
17

---

Under review as a conference paper at ICLR 2022
Original
Quality Enhanced
Quality Weakened
(a) CIFAR-10
Original
Quality Enhanced
Quality Weakened
(b) STL-10
Figure 12: Results of SNGAN on CIFAR-10 10 and STL-10.
The configuration of SNGAN for CIFAR-I0: the batch size is 128, the D-learning rate and G-learning rate are both 0.0002, the loss function is hinge loss, the discriminator updates step per generator iteration, the latent dimension is 100, and the total training step 100,000. The training configuration of the trained WGAN-GP model for CIFAR-10: 10: the loss function is Wasserstein loss and other configurations are the same as SNGAN.
The training configuration of SNGAN model for STL-10: the batch size is 64, the D-learning rate and G-learning rate are both 0.0002, the loss function is hinge loss, the discriminator updates steps per generator iteration, the latent dimension 128, and the total training step is 100,000. The training configuration of the trained WGAN-GP model for CIFAR- 10: the loss function is Wasserstein loss and other configurations the same with SNGAN. The results are shown in Fig. 12, Fig. 13.
F EXPERIMENT FOR ADVERSARIAL SAMPLE SEARCH FOR DIVERSITY
We propose AdvLaIGAN-div to improve the mapping property for diversity, which introduces the latent z iterations by Eq. into the original bi-level optimization process. The validity of the algorithm requires the effectiveness of the z iterations searching for samples with varying diversity performance guided by Eq. 8, which we verify by experiments.
Details for CIFAR-10 Experiment. We first find random latent vector of batch size 64, respectively do multiple-step gradient sign descent and multiple-step gradient sign ascent on the ratio of the distance between the original image and the target image the distance of original input vector and target input vector (which denoted as Eq. 8). The results are presented in Fig. 14. The number of iterative steps set to 300 and the step length is 0.03. The model is a pre-trained CIFAR-10 10 DCGAN model.
Details for FACADES Experiment. Under the condition of the same image input, 20 latent space vectors are randomly generated to complete the generation, and each vector is updated in two directions to obtain 20 sets of comparison images.
18

---

Under review as a conference paper at ICLR 2022
Original
Quality Enhanced
Quality Weakened
(a) CIFAR-10
Original
Quality Enhanced
Quality Weakened
(b) STL-10
Figure 13: Results of WGAN-GP on CIFAR-10 and STL-10.
Original
Anti-diverse
Diverse
Figure Results of latent iterations for diversity by using DCGAN Radford et al. (2016) pre-trained
on CIFAR-10.
G DETAILS OF ADVERSARIAL LATENT GAN EXPERIMENT
G.1 EXPERIMENT FOR ITERATIVE QUALITY IMPROVEMENT
G.1.1 DETAILS FOR CIFAR- 10 EXPERIMENT
We regard latent space variables as iterable parameters and in case of divergence, we apply learning rate for latent vector updating process. And the hyperparameters in this experiment are shown in Table 17. We use basic convolutional neural networks for test. The architectures are presented in Table 15 and Table 16.
19

---

Under review as a conference paper at ICLR 2022
learning rate e DCGAN WGAN WGAN-GP SNGAN
AT+GD 0.006
0.0008
0.001
0.005
AT+LIN 0.01
0.003
0.005
0.02
Table 17: The hyperparameter setting in CIFAR-10 adversarial training experiment.
learning rate e DCGAN WGAN WGAN-GP SNGAN
AT+LIN 0.01
0.003
0.005
0.02
Table 18: Hyperparameter setting in adversarial training Experiment on STL-10.
Table 15: Architecture of generator G in CI-
Table 16: Architecture of discriminator in
FAR10 experiment.
CIFARIO experiment.
Layer
Output size
Layer
Output size
Conv Transpose2d
2 2 1024
Conv2d
16 x 16 X 64
BatchNorm2d
2 2 x 1024
LeakyRelu
16 x 16 x 64
Relu
2 2 x 1024
Conv2d
8 X 8 X 128
Conv Transpose2d
4 512
LeakyRelu
8 X 8 x 128
BatchNorm2d
4 4 512
BatchNorm2d
8 X 8 x 128
Relu
X 512
Conv Tranppooddd
8 8 256
Conv2d
X 4 X 256
BatchNorm2d
8 256
LeakyRelu
4 X 4 X 256
Relu
8 x 8 x 256
BatchNorm2d
4 4 X 256
Conv Transpose2d
16 16 128
Conv2d
2 2 512
BatchNorm2d
16 x 16 x 128
LeakyRelu
2 2 x 512
Relu
16 x 16 x 128
BatchNorm2d
2 x 2 x 512
Conv Conv Transposdd
32 x 32 x
faltten
2048
Tanh
32 x 32 x 3
linear
Table 19: Architecture of generator G' in STL- 10 experiment.
Table 20: Architecture of discriminator D in
STL- 10 experiment.
Layer
Output size
Layer
Output size
Conv Transposed
3 3 x 1024
Conv2d
24 x 24 x 64
BatchNorm2d
3 x 1024
LeakyRelu
24 24 64
Relu
3 x 1024
Conv2d
12 12 128
Conv Transpose2d
6 512
LeakyRelu
12 x 12 x 128
BatchNorm2d
6 x 6 x 512
BatchNorm2d
12 x 12 x 128
Relu
x 6 512
Conv2d
6 x 256
Conv' Cony'TTanppon Transposedd
12 x 12 x 256
LeakyRelu
6 6 256
BatchNorm2d
12 x 12 x 256
BatchNorm2d
6 6 256
Relu
12 x 12 x 256
Conv Transpose2d
24 x 24 x 128
Conv2d
3x 512
Barch
x x 128
LeakyRelu
3 3 x 512
Relu
24 x 24 x 128
BatchNorm2d
3 3 512
Conv Transpose2d
48 48
faltten
4608
Tanh
48 x 48 x 3
linear
G.2 DETAILS FOR STL-10 EXPERIMENT.
The learning rates for updating latent variables in STL-10 experiment presented in Table 18. We use basic convolutional neural networks for test. The architectures are presented in Table 19 and Table 20,
20

---

Under review as a conference paper at ICLR 2022
G.3 EXPERIMENT FOR ITERATIVE DIVERSITY IMPROVEMENT
G.3.1 DETAILS FOR CIFAR-I0 EXPRIMENT
The experiment is obtained based on the experimental extension of MSGAN (Mao et al., 2019), and the operation of latent variable search added to the original design. The latent vector is updated in the experiment guided by Eq. with a gradient update strategy constrained by too. In our experimental setting, we adopt updates per iteration and the step size is set as 0.01. Other training parameters referred to the original MSGAN experiment https: : / /gi thub com/ He leeMeMM/// /MSGAM without modification. The FID and JSD metrics for each class are calculated based on 5000 images generated for each class, and the total FID and JSD metrics calculated by the collection of generated images for each class mentioned above.
G.3.2 DETAILS FOR 10 EXPRIMENT
The basic setup this experiment follows the CIFAR-I0 experiment, and one difference is that: we use the supervised test part of STL10 as the real dataset for training and calculate the JSD and FID metrics for the generated images and the labeled training part STL10.
21


Reviews:
[{"review_id": "tvcKlkRNXU", "reviewer": "Reviewer_uSqk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well organized, with clear sub-titles and clear logic flow;\n2. Ablation study on various baseline GAN architecture (DCGAN, WGAN, WGAN-GP, SNGAN) is conducted to show generalizability of such sampling method.\n\nCons:\n1. Compare with baseline method MSGAN: (1) why it only compare the div+ with MSGAN; (2) improvement over baseline method MSGAN is very limited.\n2. If generator trained with better quality regularization, the latent space after mapping should have better continuity? Comparison like Fig 3 after training would be needed to prove that.\n\nSome minor issues:\n1. The paper is a bit redundant on algorithms and figures. For example,  it seems lengthy to include both algorithm 1 and 2 in the main paper\n2. Fig 1. It’s not obvious what the latent space did in this figure\n", "summary_of_the_paper": "This work proposed a sample shifting method in GAN, formulated as adding intermediate latent space to generated pixel space. Such method is based on observation of continuous mapping limit: image quality in pixel space is not as continuous as latent space; limited latent space will incur mode collapse thus poor image diversity. The main contributions are: a new optimization problem as sampling method to improve image generation by quality and diversity, propose to use I-FGSM optimization method to achieve this sampling optimization problem. The experiment showed improvement on public dataset of STL-10, CIFAR-10.", "summary_of_the_review": "The paper is overall well written, and the idea is very clear and elegant. Meanwhile, I have some doubt on the sufficiency of experiment to show the improvement from quality. Also the improvement over baseline method MSGAN seems minor for me. Expect the rebuttal to clear my doubt.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oWqAHrILaM", "reviewer": "Reviewer_ua7t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " + The paper is generally well written and quite easy to understand. It does however mix preliminary work (I-FGSM) with the proposed contribution in 3.2.\n + The paper is able to improve vanilla GAN models using the presented objectives.\n - The paper does fully compare to prior work. Yes Table 8 highlights some of the similarities to prior work, but the main evaluation does not compare to alternative approaches that consider the interplay between GANs and adversarial attacks or latent space exploration. In order to see the efficacy of the presented method, the paper should experimentally compare to the majority of methods highlighted in Appendix C. The paper would be a lot stronger if it could show that the design choices made here are better than other attacks (i.e. perturbations on the generated or real images, instead of latent features, e.t.c.)\n- The visual quality of the presented examples is somewhat underwhelming. Does the presented method work on larger GAN architectures such as StyleGAN2 or BigGAN? Does the presented method actually address the issues highlighted in Fig 3?", "summary_of_the_paper": "The paper looks at the problem of improving the generative quality of GANs. The paper makes improvement along two dimensions: a) The paper adjust the samples distribution of GANs using adversarial attacks (I-FGSM), and thus effectively samples from a potentially multi-modal distribution. b) The paper improves the diversity of generated samples using using adversarial attacks on the mode-seeking objective of Mao et al 2019.", "summary_of_the_review": "The paper explores an interesting idea of adding adversarial robustness into GAN training to improve latent distribution sampling and diversification. Unfortunately, the paper falls a bit short in the experimental validation of the approach, and comparison to prior approaches.\n\n---\n\nPost rebuttal. The rebuttal makes a good case for their final algorithm using additional results. However, I still do not see what the paper adds on top of baselines, or how the problem setup in Figure 3 (interpolation artifacts) is actually addressed. The rebuttal mentions some experimental evidence that seems to indicate latent-space sampling can helps. However, I would need to see these results in an actual paper submission for review to feel comfortable about accepting it. As is the paper seems interesting, but not ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z3ij0a4ZOZe", "reviewer": "Reviewer_JVu6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.    Related works have not been cited. For example, the following paper performs a similar optimization technique with a different objective.\n\na.       Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, Timothy Lillicrap. LOGAN: Latent Optimisation for Generative Adversarial Networks. (https://arxiv.org/pdf/1912.00953.pdf)\n\nb. https://arxiv.org/abs/2005.02435\n\nc. https://arxiv.org/abs/1809.03627\n\nPlease do a thorough literature review to incorporate any such missing work.\n\n2.       The authors propose five methods viz (i) AdvLatGAN-z, (ii) AdvLatGAN-qua, (iii) AdvLatGAN-qua+, (iv) AdvLatGAN-div, and (v) AdvLatGAN-div+. \n\nEach method either focuses on improving quality or aims to enhance diversity. \n\nIt would be interesting to see what happens when the proposed objectives in equation (5) and equation (8) are combined. Does the hybrid method outperform both AdvLatGAN-qua+ and AdvLatGAN-div+?\n\n3.   Another limitation of the work is that they compute FID and JSD. FID, although a widely used metric, is not able to quantify the quality and diversity as it is a unidimensional score. Therefore, it would be nice to quantitatively verify the claims of enhancement in quality and diversity in AdvLatGAN-qua+ and AdvLatGAN-div+ respectively. Comparing other metrics such as precision/recall or density/coverage will be more meaningful towards such goals.\n\na.  Density/Coverage: Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, 2020. (https://arxiv.org/abs/2002.09797)\n\n\nb.   Precision/Recall: Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, Sylvain Gelly. Assessing Generative Models via Precision and Recall. In NIPS, 2018. (https://arxiv.org/abs/1806.00035)\n\n4. The number of baselines used in the experiment section is too less for the CIFAR-10 case?\n\n5. It would be good have experiments on large-scale datasets such as FFHQ.\n\n6. I wonder if FGSM is the only method this can be applied with or any other can be used as well? If so, how does the method depend on the adversarial training method employed?\n\n7.  I am not sure why MSGAN was chosen as the baseline for regularization?\n\n8. There is no theoretical justification on why should the proposed method work? There is some empirical evidence but there would be better to have some theoretical backing on why this method should aid in avoiding mode-collapse.\n", "summary_of_the_paper": "The authors demonstrate that the generator in a GAN is a continuous function two latent codes that are close in the latent space are mapped to two images that are close in the pixel space. However, the quality of the generated images is not preserved as quality is not a continuous function in pixel space. To address this issue, the authors propose to transform the original latent codes and demonstrate that it results in better generation quality and diversity.", "summary_of_the_review": "Please refer to the above comments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8qiZ_P3WHPj", "reviewer": "Reviewer_NCez", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces their motivation convincingly with some easily comprehensible figures. However, the experiments to show the effectiveness of the proposed method are basically performed only with small scale datasets, so that it makes difficult to figure out how complex manifold in the target space can be handled by this approach, in particular, it is unclear how it can scale to the ImageNet dataset. The discussion to compare the proposed method with other work seems to be not enough because there are some more research that shed light on the importance of latent vector transformation. For example, the series of StyleGAN work is continuously improving the performance by precisely analyzing the relationship between the latent space and the pixel space. They also firstly adopted to transform the latent vector z to another latent space W using a mapping network. That should be discussed as one of the possible approach to transform the latent vector, i.e., using a trainable function instead of I-FGSM. \n\nSome figures to show the effectiveness of the proposed method seems difficult to interpret as the captions claim, e.g., Figure 5. It was difficult to find the clear difference between the results of \"Anti-diverse\" and \"Original\", and the \"Diverse\" results look worse than other two in terms of the quality (if it is intended because the figure tries to show the increase of diversity, but it still unclear that the \"Diverse\" row in the Figure 5 has higher diversity compared to other two.)", "summary_of_the_paper": "This paper applies two existing techniques derived from adversarial example literature and MSGAN to improve the quality and diversity of generated samples by GAN. The former idea from [Goodellow et al., 2014b] is used to shift a latent vector z originally sampled from the Gaussian, which is different from the original adversarial attack paper that transforms the input image. The direction to move the latent vector is calculated by using I-FGSM with a standard loss for the generator proposed in the vanilla GAN paper. The experiments show that this technique can improve the quality of generated samples. Furthermore, this paper also consider how to improve the diversity of generated samples by transforming the latent vectors before putting them into the generator network. The approach for that is based on MSGAN's idea that mines hard latent vectors for the discriminator. The authors combine these two techniques and achieved better results compared to DCGAN, WGAN, WGAN-GP, and SNGAN models on several relatively small scale datasets such as CIFAR-10, STL-10, etc.", "summary_of_the_review": "The paper is well organized and easy to follow the authors' claims but the qualitative comparison results seems difficult to be interpret as the captions state. And from the perspective of transformation of the latent vector z, there can be more discussion in the comparison with other methods such as StyleGAN that transform z before putting it into the generator.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: To improve the generative adversarial nets, the paper proposes to add an implicit transformation of the Gaussian latent variables before the top-down generator. To further obtain better generations with respect to quality and diversity, this paper introduces targeted latent transforms into a bi-level optimization of GAN.  Experiments are conducted to verify the effectiveness of the proposed method. The paper is highly motivated and well-written, but the experiment part still needs to be strengthened because the goal of the paper is to improve the GAN training, comprehensive and thorough evaluation of the proposed method is necessary. 

After the first round of review, in addition to the clarification issue and missing reference issue, two reviewers point out that the method is only tested in small-scale datasets, and suggest authors evaluate the performance of the proposed method in more complex datasets. Two reviewers point out that the experimental validation and comparison to prior approaches are insufficient. During the rebuttal, the authors provide extra experiment results to partially address some issues.  However, most of the major concerns from other reviewers, such as (i) how are the performance of the method in large scale datasets that have complex latent space manifolds, (ii) non-convincing performance gain, and unclear problem setup, still remain. After an internal discussion, AC agrees with all reviewers that the current paper is not ready for publication, thus recommending rejecting the paper. AC urges the authors to improve their paper by taking into account all the suggestions provided by the reviewers, and then resubmit it to the next venue.
Decision: Reject