Year: 2021

Abstract:
Generative Adversarial Networks (GANs) have been widely studied as generative models, which map a latent distribution to the target distribution. Although many efforts have been made in terms of backbone architecture design, loss function, and training techniques, few results have been obtained on how the sampling in latent space can affect the final performance, and existing works on latent space mainly focus on controllability. We observe that, as the neural generator is a continuous function, two close samples in latent space would be mapped into two nearby images, while their quality can differ much as the quality is not a continuous function in pixel space. From the above continuous mapping function perspective, on the other hand, two distant latent samples are also possible to be mapped into two close images. If the latent samples are mapped in aggregation into limited modes or even a single mode, mode collapse occurs. Accordingly, we propose adding an implicit latent transform before the mapping function to improve latent z from its initial distribution, e.g., Gaussian. In this paper, this is specifically achieved by using the iterative fast gradient sign method (I-FGSM). We further propose new GAN training strategies to obtain better generation mappings w.r.t quality and diversity by introducing targeted latent transforms into the bi-level optimization of GAN. Experimental results on visual data show that our method can effectively achieve improvement in both quality and diversity.

heading: 1 INTRODUCTION
text: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014a) have shown effectiveness for generating high-fidelity data, especially for images under various settings (Ledig et al., 2017; Wang et al., 2018; Zhu et al., 2017; Zhan et al., 2019; Chakraborty et al., 2018). Based on the zero-sum game, the model is trained by the adversarial process between the generator and the discriminator. Many efforts have been made to achieve a more realistic generation from different perspectives. WGAN (Arjovsky et al., 2017), SNGAN (Miyato et al., 2018), LSGAN (Mao et al., 2017) aim to design better objective functions. ImprovedGAN (Salimans et al., 2016), AC-GAN (Odena et al., 2017) propose practical training techniques. While more complex network structures are studied in BigGAN (Brock et al., 2018), StyleGAN (Karras et al., 2019a), SAGAN (Zhang et al., 2019).
Despite the above progress, an intriguing and relatively less studied question is how the latent space affects the generation in terms of quality and diversity, which can be orthogonal to the above frequently studied factors like model architectures and training techniques. Specifically, we argue that the latent samples from a standard continuous distribution (e.g., Gaussian) can often be mapped to varying-quality samples for the generation. One reason is that the generator is a continuous function (by neural nets) while the generation quality (e.g., images in pixel space) is not. Moreover, it has also been shown in GAN literature (Khayatkhoei et al., 2018) that many image objects lie upon multiple disjoint manifolds, and a continuous mapping from continuous latent distribution, e.g., Gaussian must generate many invalid samples in the between of the manifolds when the generator seeks to cover all the meaningful modes. See Fig. 1 (a) for the schematic diagram. This fact also implies a dilemma: for complex real-world data, there is a trade-off between quality and diversity when the latent vectors are sampled from a simple uni-mode distribution, e.g., Gaussian, and the generator is a continuous mapping based on neural networks. Moreover, by realizing that the mapping function in fact allows for many-to-one mapping, mode collapse (Srivastava et al., 2017) can naturally happen when different latent samples are mapped into limited or even a single mode in target space.
The above contradiction between the discontinuity in generation data quality (e.g., images) and the continuity of latent distribution as well as mapping function, can be one of the significant factors to the well-known difficulty in GANs training, especially for generators, as one unreasonably hopes to enforce a discontinuous mapping via continuous neural nets. Meanwhile, the mapping property can largely affect the generative performance for functions with continuous nature, which is associated with most previous works devoted to improving generative performance.
To tackle these issues, we impose an extra (implicit) transform function z∗(·) on the raw sampling z, and the generation can be written by G(z∗(z)). From this perspective, most existing works, including the vanilla GAN, employ z∗(·) with an identical form. Specifically, the implicit function based transform z∗(·) can be achieved by updating the latent z ∼ p(z) to minimize lossG (i.e. log(1−D(G(z)))) using iterative fast gradient sign method (I-FGSM) (Kurakin et al., 2016) with the parameters of G and D fixed, and we name the method as AdvLatGAN-z. See Fig. 1 (b) for the schematic diagram. On top of the first effort, we propose a novel training strategy to improve the generative mapping G respectively for quality and diversity. See Fig. 2 for the schematic diagram. This is achieved by introducing z iterations (i.e., targeted implicit transforms on z) into GAN training, specifically, updating z ∼ p(z) using I-FGSM to find more optimization-friendly ones in the bi-level optimization of vanilla GAN. Respectively for quality and diversity, we use two iterative updating strategies for z under different objective functions in training, and we name the two training algorithms respectively as AdvLatGAN-qua and AdvLatGAN-div. The highlights of this paper include:
1) We rethink the role of the generator as a continuous function, which may incur quality discontinuity in target space and mode aggregation (collapse) when the latent samples are directly sampled from
a continuous distribution e.g. Gaussian. Based on above theoretical considerations, we propose a composite generation perspective by formulating the mapping fromG(z) toG(z∗(z)). This treatment provides a new methodological guidance to the progress of the generative model.
2) We propose a new pipeline for both G(·) and z∗(·), by first training generative mappings for better generation quality and diversity, respectively, and then the transform z∗(·) is enforced and computed implicitly to obtain an improved distribution for more effective generation. Specifically, the training involves adversarial learning under different objective functions for quality and diversity, leading to new GAN models: AdvLatGAN-qua/div and their further variants.
3) Experimental results on synthetic and visual data show the notable improvement of our approach on both generation quality and diversity. Our technique is orthogonal to existing GAN techniques.

heading: 2 RELATED WORK
text: Implicitly adapting the raw latent vectors z from Gaussian to improve the generation relates to latent space learning for GAN. While the specific training algorithm using latent vector adapting to mine optimization-friendly samples in training relates to adversarial training, which views the latent adapting in training as an attack to mine hard samples.
Latent space exploration in GANs. Techniques have been proposed for latent space exploration for GANs. By utilizing the steerability of GANs in latent space, StyleGAN (Karras et al., 2019a) introduce an latent space transform by using a mapping network to better decouple the various styles and Jahanian et al. (2019) shift the distribution to fit camera movement transformation. InterFaceGAN (Shen et al., 2020) decouples entangled semantics with subspace projection to control facial attributes more precisely. Mishra et al. (2020), Mukherjee et al. (2019) and Wu et al. (2019) introduce the latent exploration into training to achieve clustering objectives or benefit GAN training. In addition to the preceding works, some recent works focus on improving latent sampling quality to generate high-quality images. For example, interactive evolutionary computation is used to make the generation process more controllable in Bontrager et al. (2018), which meanwhile also brings additional manual efforts. To reduce manual intervention, Roziere et al. (2020; 2021) use Koncept512 (Hosu et al., 2020) as a criterion to improve the quality of fake images. However, as discussed in Roziere et al. (2020), this strategy is somehow biased, can not work well in each dataset (e.g., Pokemons). The reason is that the quality estimator is not even designated to access the real-data distribution information. Instead, the generation is guided by a predefined score function.
Compared to the above works, we take a relevant idea to Roziere et al. (2020) that also adapt the latent vectors to an improved condition before input to the generator G. Differently, our latent adaption approach incorporates the target distribution and utilize I-FGSM iterations to adapt latent vectors.
Mitigating mode collapse for GANs. There are also efforts in mitigating mode collapse, which are based on the thinking that many real-world samples lie in disjoint manifolds (Khayatkhoei et al., 2018). One way is to introduce a collection of generators such that each generator may cover a specific mode since a single generator can hardly fulfill a discontinuous function (Khayatkhoei et al., 2018). However, more works focus on improving the properties of mapping, explicitly obtaining a mapping covering more modes (Elfeki et al., 2019; Meulemeester et al., 2020; Mao et al., 2019). Note that the generated distribution obtained by this method will contain invalid parts due to the continuity of the latent distribution and the mapping. This calls for selective sampling of the continuous latent distribution or directly amending the latent distribution, which is rarely noticed in these works.
Adversarial samples and adversarial training. Adversarial samples are malicious samples that can easily fool deep models, which are indistinguishable to human eyes. Since the seminal work (Szegedy et al., 2013), various adversarial attack methods have been devised (Goodfellow et al., 2014b; Akhtar & Mian, 2018; Kurakin et al., 2016; Madry et al., 2017). To defend the models from attacks, adversarial training is meanwhile developed (Goodfellow et al., 2014b; Wang et al., 2019; Nøkland, 2015). In particular, it is worth noting that several recent studies have shown that adversarial training can bring many benefits to GANs, not just improving robustness. RGAN (Zhang et al., 2020) shows that adversarial training can enhance the generalization of the generator and discriminator. Based on both theoretical analysis and empirical results, Zhong et al. (2020); Liu et al. (2021); Liu & Hsieh (2019) show that updating discriminator’s parameters with the loss calculated by adversarial samples generated by authentic images can stabilize training and accelerate convergence.
In this paper, we utilize the adversarial sample search method, the iterative fast gradient sign method (I-FGSM) (Kurakin et al., 2016) to conduct latent transform and mine optimization-friendly z samples in the GAN training process, which matches up with the procedure of adversarial training. While most existing works make perturbations on pixel space (real samples or generated samples), instead we focus on mining the latent space and making perturbations on latent samples.

heading: 3 PROPOSED METHOD
text: This section presents AdvLatGAN-z to achieve a transformation z∗(·), and further AdvLatGAN-qua and AdvLatGAN-div to obtain generation mapping G with better generation quality (i.e., -qua) and diversity (i.e., -div). Both methods are derived from the iterative updating in latent space.

heading: 3.1 PRELIMINARIES
text: We introduce adversarial samples and adversarial training used to derive z∗(z) and devise training algorithms, and the regularization technique in MSGAN (Mao et al., 2019) for mode seeking.
Adversarial samples and adversarial training. Adversarial training aims to defend against adversarial attacks. It uses adversarial samples to calculate the optimized loss. In classification, for each image-label pair (xi, yi) ∈ S from the labeled training set, the adversarial samples x′ is defined as:
x′ = argmax ||x′−xi||p≤ l(fθ(x ′), yi) (1)
Here is the radius of the closed ball to constrain perturbation. While fθ and l denote the attack target (a neural net) and the corresponding loss (e.g. cross-entropy loss). p denotes the norm dimension. Given in total n adversarial samples, adversarial training is fulfilled by the optimization:
θ = min θ
1
n n∑ i=1 l(fθ(x ′ i), yi) (2)
Mode coverage by regularizing distance between generated samples. MSGAN (mode seeking GAN) (Mao et al., 2019) mitigates mode collapse in conditional generation setting by introducing a regularization term for training G over two latent samples z1 and z2 as follows. The hope is that the two generated targets shall be far away from each other as much as possible:
Lms = dI (G(c, z1), G(c, z2))
dz (z1, z2) (3)
where c is the condition vector for generation, and dI and dz are the distance metrics in the target (image) space and latent space, respectively. It encourages the distance to be maintained or even enlarged in the target space to avoid the generated samples being trapped into an aggregation.
By integrating the regularization term with the original objective function of cGANs (Mirza & Osindero, 2014), the final objective for minimization in MSGAN for training G and D is given by:
L =Lori + λmsLms Lori =Ec,x[logD(c,x)] + Ec,z[log(1−D(c,G(c, z)))]
(4)
where Lori denotes the original objective, x denotes real data, and λms is the weight hyper-parameter. Note since Eq. 3 is aimed to be maximized hence one can interpret λms to be negative.

heading: 3.2 LATENT SPACE ADVERSARIAL SAMPLE SEARCH FOR GENERATION QUALITY
text: We show how to implicitly transform the latent raw samples from Gaussian (or other popular forms e.g. uniform) to a new distribution via the idea of searching adversarial samples as introduced above. This transform breaks the dilemma in diversity and quality where the bottleneck is due to the continuous latent distribution and a continuous mapping function. Moreover, conducting the implicit transform in training to obtain the new (probably disconnected) latent distribution for each iteration can compensate for the difficulty for generators to align the pace of generators and discriminators1, which cannot map a continuous distribution to a disconnected one with multiple modes.
Specifically, we design the following constrained optimization task that encourages the new z∗ to be close enough2 to z. Its prediction score (as real data) by the discriminator D should be as high as possible to improve the generation quality. Recall otherwise the given z0 may correspond to a poor generation as discussed in Sec. 1. Note here that both G and D are fixed.
z∗(z0) = argmin z∈{z|d(z0,z)≤ } log(1−D(G(z))) (5)
where z0 denotes the original latent vectors, z∗(z0) denotes the newly computed z, d(·) denotes the distance of two vectors which is an `∞ distance in this paper, in line with the popular protocol in adversarial attack literature (Goodfellow et al., 2014b).
By borrowing the well-developed tools in adversarial attack, the solution of the above problem can be estimated by the so-called iterative fast gradient sign method (I-FGSM) (Kurakin et al., 2016):
zi+1 ← zi − sgn{∇zi log(1−D(G(zi)))} (6)
From the distribution perspective, the transform can be characterized by:
p∗z = arg min D(pz,p0z)≤ Ez∼pz [log(1−D(G(z)))] (7)
where p0z denotes the original latent distribution, p ∗ z denotes the newly searched one. D(pz, p 0 z) ≤ means that there is a limit to the distribution variation. A newly searched z can be considered as a sample from p∗z if the sampling is within the perturbation circle. Fig. 4 shows the results by the pre-trained SNGAN (Miyato et al., 2018) (i.e. the given G and D) on STL-10 (Coates et al., 2011).
We develop a novel GAN training algorithm by conducting the above latent distribution shift to the initial sampling of Gaussian in D updating iterations, while the rest of the training algorithm remains consistent with vanilla GAN (Goodfellow et al., 2014a). We present the algorithm in Alg. 1 which we call Adversarial Latent GAN (AdvLatGAN-qua) with a post meaning quality. The latent tranform is first computed using the I-FGSM to find the adversarial samples in latent space, then D and G are updated. The iteration continues until enough steps or convergence of G.
1In literature, it is widely recognized that it is much more challenging to train the generator than discriminator. 2We do not want to departure too much from the basic GAN training protocol, and also the transform shall
keep mode consistent to maintain overall diversity.
The above latent transform model does not consider how to avoid mode collapse, as each time only one latent sample is considered for optimization in isolation. Similar constrained optimization formulation can be designed to improve the generation diversity, as shown in the following subsection.

heading: 3.3 LATENT SPACE ADVERSARIAL SAMPLE SEARCH FOR GENERATION DIVERSITY
text: We take a hard sample mining perspective to improve mode coverage. Given a sample z0 from the raw latent distribution, we aim to search its paired sample z∗ to form a hard sample pair, in the sense that their generations are close in the target space given G. We consider two aspects. First, as hard samples, the paired samples shall still belong to the same category c which can be modeled with conditional GAN. Second, their distance in the latent space shall also be not too close otherwise, they may be aggregated in target space due to continuous mapping. In other words, hard samples causing mode collapse shall be those close-in target space while apart in the latent space, when G is given. Based on the above considerations, we adopt the form of Eq. 3 to design our search procedure:
z∗(z0) = argmin z∈{z|d(z0,z)≤ }
dI(G(c, z0), G(c, z))
dz(z0, z) (8)
The symbols and terms are similar to Eq. 3. Akin to AdvLatGAN-qua, we use I-FGSM to obtain z∗ and again use `∞ to control change magnitude. For each update step, we randomly select the first latent vector z0, and the second vector z∗ is searched by n iterations of the I-FGSM method under the `∞ restriction starting from the neighborhood of z0, and the iterations are guided by Eq. 8.
Table 1 shows the effectiveness of the above paired complex sample seeking approach, with experiments on CIFAR-10 (Krizhevsky et al., 2009) and Facades (Tyleček & Šára, 2013) with DCGAN (Radford et al., 2016) as backbone. The visual results on Facades are shown in Fig. 5. We can see that the pair obtained by solving Eq. 8 tend to collapse (as they are hard samples), while the opposite leads to better diversity by using the inverse form of Eq. 8.
The above experiments show the effectiveness of our iteration method, and we further introduce it into the initial training process of MSGAN (Mao et al., 2019). In MSGAN, the regularization term 3 is used to regularize the model optimization process, where the selection of the pair is entirely random, without using any heuristic information. We instead use the z pair that are more inclined to collapse by randomly taking one z and iterating by the method mentioned above to get another z. We keep the other settings of the algorithm unchanged.
By using Eq. 8 to generate adversarial sample pairs to address mode collapse, we finally develop our new diversity enhancing GAN training approach AdvLatGAN-div as shown in Alg. 2 in the appendix, which are further enhanced by post-training as discussed in experiments.
4 EXPERIMENTS
Experiments are performed on a single GPU of GeForce RTX 3090. All datasets used in experiments are publicly available.

heading: 4.1 EXPERIMENTAL SETUP
text: We validate the effectiveness of the proposed methods for two parts, respectively for effective latent sampling improvement z∗ and better generation mapping G. In particular, five methods can be derived for different targets, among which AdvLatGANqua+ and AdvLatGAN-div+ are the final full version of our contributions. We present the methods below:
i) AdvLatGAN-z: post-training latent sampling improvement fighting against quality discontinuity; ii) AdvLatGAN-qua: GAN training algorithm for better generative quality using intraining latent sampling improvement; iii) AdvLatGAN-div: GAN training algorithm for a more diverse generation using in-training latent sampling improvement; iv) AdvLatGANqua+: both in and post-training latent sampling improvement for generation quality; v) AdvLatGAN-div+: both in and posttraining latent sampling improvement for generation diversity.
For illustrative comparison, we show the schematic overview of these methods in Fig. 6 and Table 2.
We adopt Inception Score (Salimans et al., 2017) and Fréchet Inception Distance (Heusel et al., 2017) to evaluate the performance, which utilizes the classification results of the generated images by the Inception Network to evaluate the performance of the model. Following (Richardson & Weiss, 2018), JSD is adopted to measure the similarity between generated images and real images, which first clusters real images and generated images into the same number of classes using the K-means algorithm, then calculate the similarity between real and fake clusters w.r.t. the clustering property. We also adopt recently proposed evaluation metrics density/coverage (Naeem et al., 2020), which separates the fidelity evaluation and the diversity evaluation.

heading: 4.2 ADVLATGAN-Z: POST-TRAINING LATENT SAMPLING IMPROVEMENT
text: This section shows the effectiveness of the post-training latent sampling improvement AdvLatGAN-z (denoted as z∗ in previous sections, which is an implicit transform on the original latent sampling from Gaussian), which aims to mitigate the issue of quality discontinuity.
Results on Synthetic Data. In line with (Metz et al., 2017), we apply the proposed iterative strategy guided by Eq. 5 in both Grid and Ring datasets to evaluate its performance. By updating latent space variables, we achieve highquality generated samples. The results in Fig. 7 show the effectiveness of our method.
Results on MNIST. MNIST contains 70,000 images of handwritten digits (LeCun et al., 1998), which can be generated from a two-dimensional latent space in which the iterative latent distribution
.
change is illustrated in Fig. 8. We respectively conduct the experiments for the trained DCGAN model and WGAN-GP (Gulrajani et al., 2017) model, with single-step iteration constraint using `∞ and iterated step size is set to 0.03. To better reflect changes in the distribution, we only show the first 30 iterations that can reflect larger changes. For WGAN test, Fig. 8 shows the transformed latent distribution which contains more peaks and valleys and is different from the raw Gaussian.
Results on CIFAR-10 and STL-10. We run experiments on CIFAR-10 (Krizhevsky et al., 2009) and STL-10 (Coates et al., 2011). CIFAR-10 has been widely used with more diverse and closer to the quality of high-resolution images than MNIST. STL-10 has a higher resolution and richer image representation. We use implicit transform in the latent space for pre-trained DCGAN and SNGAN respectively, with the single-step iteration constraint using `∞ and iteration step size is set to 0.01. We conduct 100 steps for each iteration process. Inception Score and Fréchet Inception Distance are used to evaluate the generation quality, which are calculated over 1500 generated images for STL-10 and over 3000 images for CIFAR-10. For each of the four cases, we use two different initializations, each initialized randomly and not explicitly chosen. As shown in Table 3, our method has a significant impact on generation quality, with a maximum improvement of 12.03% by Inception Score.
Results on AFHQ and FFHQ. To show the effectiveness of AdvLatGAN-z to solve issues in Fig. 3, we evaluate on AFHQ (Choi et al., 2020) and FFHQ (Karras et al., 2019b) based on the StyleGAN2ada (Karras et al., 2020) model. Note that AdvLatGAN-z works at the sample level (or distribution level as a distribution shift characterized by sample transforms) in the latent space. In the Fig. 3 setting, when the sampling encounters invalid examples, AdvLatGAN-z is able to transform the bad z to z∗(z) with better ability to avoid bad generation. Here we verify whether invalid samples can be improved with AdvLatGAN-z. We select six bad generations with the first three correspond to those in Fig. 3 and conducting AdvLatGAN-z to their latent vectors. The results show that AdvLatGAN-z can effectively fix or avoid image defects, as shown in Fig. 9.

heading: 4.3 ADVLATGAN-QUA/DIV: IMPROVING THE GENERATION MAPPING
text: This section shows the effectiveness of the GAN training algorithms, which aims to improve the generation mapping G for better performance respectively on quality and diversity.
AdvLatGAN-qua for Quality Improvement. We conduct experiments on CIFAR-10 and STL-10, using the mainstream architectures including DCGAN, WGAN, WGAN-GP, SNGAN, LSGAN, WGAN-div (Wu et al., 2018) and ACGAN (Kang et al., 2021). We do not include ACGAN in the STL-10 setting because it does not work on unlabeled dataset. We adversarially train GAN using a quality-enhanced strategy previously mentioned as AdvLatGAN-qua, and the results show that the proposed method can greatly improve the quality of generated images. IS and FID are adopted to evaluate the performance of the model. We conduct only one latent iteration to save computation cost per generator step, and more details will be given in the appendix. We also try to use implicit transform after training to improve the generation quality further. The result is shown in Table 4 and Table 5. It can be seen that the proposed method achieves a significant improvement in generative performance compared to that without the latent space transformation strategy. In the WGAN STL-10 setting, IS is improved from 6.51 to 8.16, and FID from 73.0 to 49.1.
AdvLatGAN-div for Diversity Improvement. We conduct experiments on both CIFAR-10 and STL-10 for conditional generation and adopt FID, JSD, density and coverage as the metrics, respectively for the entire set of generated images and each class. We test the training results on the basis of implicit latent transform to fight against quality discontinuity. We can find that the proposed method is prior over MSGAN reflected by JSD and FID metrics. The results are shown in Table 6 and Table 7. For the overall set which models are trained, AdvLatGAN-div+ has achieved at most 11.5% improvement in FID, 36.0% in JSD, 47.4% in density and 27.8% in coverage.

heading: 5 CONCLUSION
text: This work begins with two basic observations in GANs from the perspective of continuous mapping. Based on the observations, we introduce the adversarial attack techniques in GAN to explore latent space and derive novel training algorithms to improve the mapping property. With the efforts of both latent transform and mapping improvement, the proposed AdvLatGAN has shown promising ability for achieving generation with both quality and diversity verified by extensive experiments.

heading: APPENDIX

heading: A DETAILED ALGORITHM FOR ADVLATGAN-QUA AND ADVLATGAN-DIV
text: The specific algorithm for AdvLatGAN-qua in Sec. 3.2 is presented in Alg. 1.
Algorithm 1 AdvLatGAN-qua tz, tD: the number of steps, other symbols are consistent with vanilla GAN. Input: pz e.g. Gaussian, pr e.g. real images distribution; randomly initialized G and D; Output: trained generator G and discriminator D; 1: while G has not converged do 2: // update D: 3: for i = 1 to tD do 4: Sample {z(k)0 }mk=1 ∼ pz; 5: // the proposed sampling shift (t-step I-FGSM): 6: for i = 1 to tz do 7: Obtain {z(k)i }mk=1 by Eq. 5; 8: Sample {x(k)i }mk=1 ∼ pr; 9: Calculate lossD with {x(k)i }mk=1 and {z (k) t }mk=1; 10: Update D with lossD; 11: // update G: 12: Sample {z(k)}mk=1 ∼ pz; 13: Calculate lossG with {z(k)}mk=1; 14: Update G with lossG;
The specific algorithm for AdvLatGAN-div in Sec. 3.3 is presented in Alg. 2.
Algorithm 2 AdvLatGAN-div tz, tD: the number of steps, other symbols are consistent with vanilla GAN. Input: pz e.g. Gaussian, pr e.g. real images distribution; λms is a hyper-parameter; randomly initialized G and D; Output: trained generator G and discriminator D; 1: while G has not converged do 2: // update D: 3: for i = 1 to tD do 4: Sample {z(k)i }mk=1 ∼ pz; 5: Sample {(x(k)i , c (k) i )}mk=1 ∼ pr;
6: Compute lossD with {(x(k)i , c (k) i ), z (k) i }mk=1; 7: Update D with lossD; 8: // update G: 9: Randomly generate labels {c(k)}mk=1;
10: Sample {z(k)0 }mk=1 ∼ pz; 11: // the proposed sampling shift (t-step I-FGSM): 12: for i = 1 to tz do 13: Obtain {z(k)i }mk=1 by Eq. 8; 14: Calculate lossG and lossms (Eq. 3) with {c(k)}mk=1, {z (k) 0 }mk=1 and {z (k) t }mk=1; 15: loss← lossG + λmslossms; 16: Update G with loss;

heading: B RELATIONSHIP WITH ADVERSARIAL ATTACKS AND DEFENSES
text: Our approach is closely related to adversarial attacks and defenses. As for works in adversarial attacks, the fundamental standpoint is that deep neural networks’ performance can vary significantly in the face of small perturbations. This scenario can be very similar to our starting point as described
in Sec 1 that a small perturbation of the latent vectors can lead to massive quality variation. Thus it is quite natural to combine the adversarial attacks and defenses with generative adversarial networks, and specifically, we utilize the latent transform using I-FGSM iterative method to overcome the quality discontinuity.
Our GAN training algorithms AdvLatGAN-qua and AdvLatGAN-div can be analogous to adversarial training techniques in the adversarial defense field. Adversarial training is a proactive defense approach that strengthens the model against attacks or enhances its performance by modifying the inputs to use adversarial samples to train the model, making the model naturally defensive against attacks. This process involves conducting adversarial attacks during training and can be explained as mining targeted optimization-friendly samples for training. While in AdvLatGAN-qua and AdvLatGAN-div, we conduct z iterations in the GAN training referencing attack method, which can be viewed as the same process as adversarial training. AdvLatGAN introduces the latent vector z iterations into the bi-level optimization between the generator G and the discriminator D. The training involves iterations of three components z, G ,and D, among which the introduced z iterations can mine the latent space during training, to some extent leading to the superiority of the method.

heading: C DESCRIPTION OF CONSTRAINTS FOR ITERATION
text: There are two basic types of constraint used in the field of adversarial attack: 1) `2 norm constraint; 2) `∞ norm constraint. In this paper, we follow the well-known attack method, iterative fast gradient sign method (I-FGSM) (Kurakin et al., 2016) to conduct the iterations of latent variables and we adopt the `∞ norm as our constraint.

heading: D DIFFERENCES AND COMPARISONS TO OTHER ALGORITHMS
text: We present the differences between our approach and other previous works, involving two topics i.e. latent exploration and GAN model with adversarial training. An overview is presented in Table 8. Meanwhile, we conduct experiments to show our superiority over other representative works.

heading: D.1 LATENT EXPLORATION
text: EvolGAN (Roziere et al., 2020), Tarsier (Roziere et al., 2021). These two models both use the well-known quality evaluator Koncept512 (which is actually a classifier) to guide the latent iterations. Their drawback is that they do not exploit the information of the real distribution, which leads to that the optimization criterion of sampling quality is not for the proximity to the real distribution, but to match the data trained for the Koncept512 classifier. Our approach uses a discriminator to guide the latent variable updating, which makes full use of the information from the real distribution, while further achieving novel training algorithm to improve the generative mapping.
DDLS (Che et al., 2020). DDLS analyzes the sample-improving approach from the perspective of the energy-based model, which differs from ours in terms of methodology. Besides, it didn’t notice that when we enhance the quality, we hope that the image is changed as little as possible. Combing the above two points, we explain the problem from the perspective of attack, which is quite different from the specialized sampling approach Langevin Dynamics used in (Che et al., 2020).

heading: D.2 GAN MODEL WITH ADVERSARIAL TRAINING
text: Our GAN training algorithm can be considered as a kind of adversarial training from the perspective of adversarial attack and defense, and we differ from several previous works that combine adversarial training and GAN.
ASGAN (Liu et al., 2021), FastGAN (Zhong et al., 2020), Rob-GAN (Liu & Hsieh, 2019). These methods add perturbation to the real images (the latter is an improvement in the loss function compared to the former), while we focus on the latent space, perturbing latent space vectors.
Robust GAN training (Zhou & Krähenbühl, 2018). This algorithm shows that a robust discriminator can benefit training and this robustness only need to be enforced in expectation over the generated samples, again without focusing on the latent space compared to us.

heading: D.3 EXPERIMENTAL COMPARISON OVER LATENT EXPLORATION
text: We compare our methods AdvLatGAN-z with two representative works Evolgan and DDLS that combine latent exploration and GANs. For Evolgan implementation, we choose four sets of hyperparameters for evaluation. Note that we did not find the official codes for Evolgan and DDLS for real-world experiment, so we reproduce the methods according to their papers. The experimental setting is aligned with Table 4 and we evaluate on CIFAR-10 with four framworks: DCGAN, WGAN, WGANGP and SNGAN. We conduct AdvLatGAN-z on a 10000 generated images batch and then evaluate the results. Inception Score and Fréchet Inception Distance are adopted as the evaluation metrics. The experimental results are presented in Table 9 and Table 10.

heading: D.4 EXPERIMENTAL COMPARISON OVER GAN MODEL WITH ADVERSARIAL TRAINING
text: We compare our methods AdvLatGAN-qua and AdvLatGAN-qua+ with two representative works ASGAN and Robust GAN training that combine adversarial training and GANs. These two methods are related to perturbations on the generated samples and the real samples respectively. The experimental setting is aligned with Table 4 and we evaluate on CIFAR-10 with four framworks: DCGAN,
WGAN, WGANGP and SNGAN. Inception Score and Fréchet Inception Distance are adopted as the evaluation metrics. The experimental results are presented in Table 11 and Table 12.

heading: E DETAILS OF LATENT SAMPLING IMPROVEMENT EXPERIMENT

heading: E.1 PROTOCOLS FOR SYNTHETIC EXPERIMENTS.
text: We simulate two synthetic datasets. Ring dataset is a mixture of 8 2-D Gaussians p(z) with mean {(2 cos (iπ/4), 2 cos (iπ/4))}8i=1 and standard deviation 0.001. 12.5K samples are simulated from each Gaussian distribution. 50K samples from p(z) are used to generate x for test. Grid dataset is a mixture of 25 2-D isotropic Gaussians i.e. p(z) with mean {(2i, 2j)}2i,j=−2 and standard deviation 0.0025. 4K samples are simulated from each Gaussian. 20K samples from p(z) are used to generate target samples {x̃} for test. In the synthetic experiments, we apply fully-connected networks for generation and the architectures are shown in Table 13 and Table 14.
Table 13: Architecture of generator G.
Layer Output size Activation Linear 100 ReLu Linear 200 ReLu Linear 100 ReLu Linear 2 -
Table 14: Architecture of discriminator D.
Layer Output size Activation Linear 100 ReLu Linear 200 ReLu Linear 100 ReLu Linear 1 -

heading: E.2 DETAILS FOR RING SYNTHESIZING EXPERIMENTS.
text: To better demonstrate the practical effect and to show the characteristics of the network during the training process, we choose 2000 as the total number of iterations (1 epoch), and the result is shown in the left picture in Fig. 10. We apply the latent transform technique for the network in the left picture: frozen all parameters of the neural network and iterating 20,000 points sampled in a 2D-Gaussian distribution under the constraint of L-infinity norm. The step size is set as 0.003 and its optimizing target is upgrading discriminator judged score. After iterating 8,000 times, We get the high-quality
samples in Fig. 10 (shown in the middle) which outperform the results after training 100 epochs. It is a powerful evidence to prove the problem of imbalance in the input latent space and that discriminator tends to converge faster.

heading: E.3 DETAILS FOR GRID SYNTHESIZING EXPERIMENTS.
text: We try to simulate the case that the inappropriate selection of network architecture to prove the application prospects of latent transform in this situation. All hyperparameters are consistent with the previous ring experiment besides choosing training 50 epochs to make sure the network is fully trained. Because of the inappropriate choosing model architecture, the training result is terrible. However, by applying with latent transform policy, almost all points converge towards target points. The result is shown in Fig. 10.

heading: E.4 DETAILS FOR MNIST EXPERIMENT.
text: The DCGAN model and WGAN-GP model are used in this experiment. We use DCGAN model for generating images of CIFAR-10 and ResNet model for STL-10 dataset. The hyperparameters and model structures are consistent with that used in https://github.com/w86763777/ pytorch-gan-collections with a total of 30 iterations, each with a step size of 0.03 and a batch size of 8000. The configuration of the trained DCGAN model: batch size is 128, D-learning rate and G-learning rate are both 0.0002, the loss function is bce loss, the discriminator updates one step per generator iteration, the latent dimension is 2, and the total training step is 20,000. The configuration of training WGAN-GP model: the batch size is 128, the D-learning rate and G-learning rate are both 0.0002, the parameters of discriminator updates one step per generator iteration, the latent dimension is 2, and the total training step is 50,000.
Details for CIFAR-10 and STL-10 Experiment. The SNGAN and WGAN-GP are used in this experiment, referencing https://github.com/w86763777/ pytorch-gan-collections with a total of 100 iterations, each with step size of 0.01 and the batch size is set as 1500 for STL-10 and 3000 for CIFAR-10.
The configuration of SNGAN for CIFAR-10: the batch size is 128, the D-learning rate and G-learning rate are both 0.0002, the loss function is hinge loss, the discriminator updates one step per generator iteration, the latent dimension is 100, and the total training step is 100,000. The training configuration of the trained WGAN-GP model for CIFAR-10: the loss function is Wasserstein loss and other configurations are the same as SNGAN.
The training configuration of SNGAN model for STL-10: the batch size is 64, the D-learning rate and G-learning rate are both 0.0002, the loss function is hinge loss, the discriminator updates 5 steps per generator iteration, the latent dimension is 128, and the total training step is 100,000. The training configuration of the trained WGAN-GP model for CIFAR-10: the loss function is Wasserstein loss and other configurations are the same with SNGAN. The results are shown in Fig. 12, Fig. 13.

heading: F EXPERIMENT FOR ADVERSARIAL SAMPLE SEARCH FOR DIVERSITY
text: We propose AdvLatGAN-div to improve the mapping property for diversity, which introduces the latent z iterations by Eq. 8 into the original bi-level optimization process. The validity of the algorithm requires the effectiveness of the z iterations searching for samples with varying diversity performance guided by Eq. 8, which we verify by experiments.
Details for CIFAR-10 Experiment. We first find a random latent vector of batch size 64, respectively do multiple-step gradient sign descent and multiple-step gradient sign ascent on the ratio of the distance between the original image and the target image to the distance of original input vector and target input vector (which denoted as Eq. 8). The results are presented in Fig. 14. The number of iterative steps is set to 300 and the step length is 0.03. The model is a pre-trained CIFAR-10 DCGAN model.
Details for FACADES Experiment. Under the condition of the same image input, 20 latent space vectors are randomly generated to complete the generation, and each vector is updated in two directions to obtain 20 sets of comparison images.

heading: G DETAILS OF ADVERSARIAL LATENT GAN EXPERIMENT

heading: G.1 EXPERIMENT FOR ITERATIVE QUALITY IMPROVEMENT

heading: G.1.1 DETAILS FOR CIFAR-10 EXPERIMENT
text: We regard latent space variables as iterable parameters and in case of divergence, we apply learning rate for latent vector updating process. And the hyperparameters we use in this experiment are shown in Table 17. We use basic convolutional neural networks for test. The architectures are presented in Table 15 and Table 16.

heading: G.2 DETAILS FOR STL-10 EXPERIMENT.
text: The learning rates for updating latent variables in STL-10 experiment are presented in Table 18. We use basic convolutional neural networks for test. The architectures are presented in Table 19 and Table 20,

heading: G.3 EXPERIMENT FOR ITERATIVE DIVERSITY IMPROVEMENT

heading: G.3.1 DETAILS FOR CIFAR-10 EXPRIMENT
text: The experiment is obtained based on the experimental extension of MSGAN (Mao et al., 2019), and the operation of latent variable search is added to the original design. The latent vector is updated in the experiment guided by Eq. 8 with a gradient update strategy constrained by `∞. In our experimental setting, we adopt 3 updates per iteration and the step size is set as 0.01. Other training parameters are referred to the original MSGAN experiment https://github.com/HelenMao/MSGAN without modification. The FID and JSD metrics for each class are calculated based on 5000 images generated for each class, and the total FID and JSD metrics are calculated by the collection of generated images for each class mentioned above.

heading: G.3.2 DETAILS FOR STL-10 EXPRIMENT
text: The basic setup of this experiment follows the CIFAR-10 experiment, and one difference is that: we use the supervised test part of STL10 as the real dataset for training and calculate the JSD and FID metrics for the generated images and the labeled training part of STL10.

Review summaries:
This work proposed a sample shifting method in GAN, formulated as adding intermediate latent space to generated pixel space. Such method is based on observation of continuous mapping limit: image quality in pixel space is not as continuous as latent space; limited latent space will incur mode collapse thus poor image diversity. The main contributions are: a new optimization problem as sampling method to improve image generation by quality and diversity, propose to use I-FGSM optimization method to achieve this sampling optimization problem. The experiment showed improvement on public dataset of STL-10, CIFAR-10.

The authors demonstrate that the generator in a GAN is a continuous function two latent codes that are close in the latent space are mapped to two images that are close in the pixel space. However, the quality of the generated images is not preserved as quality is not a continuous function in pixel space. To address this issue, the authors propose to transform the original latent codes and demonstrate that it results in better generation quality and diversity.

The paper looks at the problem of improving the generative quality of GANs. The paper makes improvement along two dimensions: a) The paper adjust the samples distribution of GANs using adversarial attacks (I-FGSM), and thus effectively samples from a potentially multi-modal distribution. b) The paper improves the diversity of generated samples using using adversarial attacks on the mode-seeking objective of Mao et al 2019.

This paper applies two existing techniques derived from adversarial example literature and MSGAN to improve the quality and diversity of generated samples by GAN. The former idea from [Goodellow et al., 2014b] is used to shift a latent vector z originally sampled from the Gaussian, which is different from the original adversarial attack paper that transforms the input image. The direction to move the latent vector is calculated by using I-FGSM with a standard loss for the generator proposed in the vanilla GAN paper. The experiments show that this technique can improve the quality of generated samples. Furthermore, this paper also consider how to improve the diversity of generated samples by transforming the latent vectors before putting them into the generator network. The approach for that is based on MSGAN's idea that mines hard latent vectors for the discriminator. The authors combine these two techniques and achieved better results compared to DCGAN, WGAN, WGAN-GP, and SNGAN models on several relatively small scale datasets such as CIFAR-10, STL-10, etc.