Improving Generative Adversarial Networks via Adversarial Learning in Latent Space

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014a ) have shown effectiveness for generating high-fidelity data , especially for images under various settings ( Ledig et al. , 2017 ; Wang et al. , 2018 ; Zhu et al. , 2017 ; Zhan et al. , 2019 ; Chakraborty et al. , 2018 ) . Based on the zero-sum game , the model is trained by the adversarial process between the generator and the discriminator . Many efforts have been made to achieve a more realistic generation from different perspectives . WGAN ( Arjovsky et al. , 2017 ) , SNGAN ( Miyato et al. , 2018 ) , LSGAN ( Mao et al. , 2017 ) aim to design better objective functions . ImprovedGAN ( Salimans et al. , 2016 ) , AC-GAN ( Odena et al. , 2017 ) propose practical training techniques . While more complex network structures are studied in BigGAN ( Brock et al. , 2018 ) , StyleGAN ( Karras et al. , 2019a ) , SAGAN ( Zhang et al. , 2019 ) . Despite the above progress , an intriguing and relatively less studied question is how the latent space affects the generation in terms of quality and diversity , which can be orthogonal to the above frequently studied factors like model architectures and training techniques . Specifically , we argue that the latent samples from a standard continuous distribution ( e.g. , Gaussian ) can often be mapped to varying-quality samples for the generation . One reason is that the generator is a continuous function ( by neural nets ) while the generation quality ( e.g. , images in pixel space ) is not . Moreover , it has also been shown in GAN literature ( Khayatkhoei et al. , 2018 ) that many image objects lie upon multiple disjoint manifolds , and a continuous mapping from continuous latent distribution , e.g. , Gaussian must generate many invalid samples in the between of the manifolds when the generator seeks to cover all the meaningful modes . See Fig . 1 ( a ) for the schematic diagram . This fact also implies a dilemma : for complex real-world data , there is a trade-off between quality and diversity when the latent vectors are sampled from a simple uni-mode distribution , e.g. , Gaussian , and the generator is a continuous mapping based on neural networks . Moreover , by realizing that the mapping function in fact allows for many-to-one mapping , mode collapse ( Srivastava et al. , 2017 ) can naturally happen when different latent samples are mapped into limited or even a single mode in target space . The above contradiction between the discontinuity in generation data quality ( e.g. , images ) and the continuity of latent distribution as well as mapping function , can be one of the significant factors to the well-known difficulty in GANs training , especially for generators , as one unreasonably hopes to enforce a discontinuous mapping via continuous neural nets . Meanwhile , the mapping property can largely affect the generative performance for functions with continuous nature , which is associated with most previous works devoted to improving generative performance . To tackle these issues , we impose an extra ( implicit ) transform function z∗ ( · ) on the raw sampling z , and the generation can be written by G ( z∗ ( z ) ) . From this perspective , most existing works , including the vanilla GAN , employ z∗ ( · ) with an identical form . Specifically , the implicit function based transform z∗ ( · ) can be achieved by updating the latent z ∼ p ( z ) to minimize lossG ( i.e . log ( 1−D ( G ( z ) ) ) ) using iterative fast gradient sign method ( I-FGSM ) ( Kurakin et al. , 2016 ) with the parameters of G and D fixed , and we name the method as AdvLatGAN-z . See Fig . 1 ( b ) for the schematic diagram . On top of the first effort , we propose a novel training strategy to improve the generative mapping G respectively for quality and diversity . See Fig . 2 for the schematic diagram . This is achieved by introducing z iterations ( i.e. , targeted implicit transforms on z ) into GAN training , specifically , updating z ∼ p ( z ) using I-FGSM to find more optimization-friendly ones in the bi-level optimization of vanilla GAN . Respectively for quality and diversity , we use two iterative updating strategies for z under different objective functions in training , and we name the two training algorithms respectively as AdvLatGAN-qua and AdvLatGAN-div . The highlights of this paper include : 1 ) We rethink the role of the generator as a continuous function , which may incur quality discontinuity in target space and mode aggregation ( collapse ) when the latent samples are directly sampled from a continuous distribution e.g . Gaussian . Based on above theoretical considerations , we propose a composite generation perspective by formulating the mapping fromG ( z ) toG ( z∗ ( z ) ) . This treatment provides a new methodological guidance to the progress of the generative model . 2 ) We propose a new pipeline for both G ( · ) and z∗ ( · ) , by first training generative mappings for better generation quality and diversity , respectively , and then the transform z∗ ( · ) is enforced and computed implicitly to obtain an improved distribution for more effective generation . Specifically , the training involves adversarial learning under different objective functions for quality and diversity , leading to new GAN models : AdvLatGAN-qua/div and their further variants . 3 ) Experimental results on synthetic and visual data show the notable improvement of our approach on both generation quality and diversity . Our technique is orthogonal to existing GAN techniques . 2 RELATED WORK . Implicitly adapting the raw latent vectors z from Gaussian to improve the generation relates to latent space learning for GAN . While the specific training algorithm using latent vector adapting to mine optimization-friendly samples in training relates to adversarial training , which views the latent adapting in training as an attack to mine hard samples . Latent space exploration in GANs . Techniques have been proposed for latent space exploration for GANs . By utilizing the steerability of GANs in latent space , StyleGAN ( Karras et al. , 2019a ) introduce an latent space transform by using a mapping network to better decouple the various styles and Jahanian et al . ( 2019 ) shift the distribution to fit camera movement transformation . InterFaceGAN ( Shen et al. , 2020 ) decouples entangled semantics with subspace projection to control facial attributes more precisely . Mishra et al . ( 2020 ) , Mukherjee et al . ( 2019 ) and Wu et al . ( 2019 ) introduce the latent exploration into training to achieve clustering objectives or benefit GAN training . In addition to the preceding works , some recent works focus on improving latent sampling quality to generate high-quality images . For example , interactive evolutionary computation is used to make the generation process more controllable in Bontrager et al . ( 2018 ) , which meanwhile also brings additional manual efforts . To reduce manual intervention , Roziere et al . ( 2020 ; 2021 ) use Koncept512 ( Hosu et al. , 2020 ) as a criterion to improve the quality of fake images . However , as discussed in Roziere et al . ( 2020 ) , this strategy is somehow biased , can not work well in each dataset ( e.g. , Pokemons ) . The reason is that the quality estimator is not even designated to access the real-data distribution information . Instead , the generation is guided by a predefined score function . Compared to the above works , we take a relevant idea to Roziere et al . ( 2020 ) that also adapt the latent vectors to an improved condition before input to the generator G. Differently , our latent adaption approach incorporates the target distribution and utilize I-FGSM iterations to adapt latent vectors . Mitigating mode collapse for GANs . There are also efforts in mitigating mode collapse , which are based on the thinking that many real-world samples lie in disjoint manifolds ( Khayatkhoei et al. , 2018 ) . One way is to introduce a collection of generators such that each generator may cover a specific mode since a single generator can hardly fulfill a discontinuous function ( Khayatkhoei et al. , 2018 ) . However , more works focus on improving the properties of mapping , explicitly obtaining a mapping covering more modes ( Elfeki et al. , 2019 ; Meulemeester et al. , 2020 ; Mao et al. , 2019 ) . Note that the generated distribution obtained by this method will contain invalid parts due to the continuity of the latent distribution and the mapping . This calls for selective sampling of the continuous latent distribution or directly amending the latent distribution , which is rarely noticed in these works . Adversarial samples and adversarial training . Adversarial samples are malicious samples that can easily fool deep models , which are indistinguishable to human eyes . Since the seminal work ( Szegedy et al. , 2013 ) , various adversarial attack methods have been devised ( Goodfellow et al. , 2014b ; Akhtar & Mian , 2018 ; Kurakin et al. , 2016 ; Madry et al. , 2017 ) . To defend the models from attacks , adversarial training is meanwhile developed ( Goodfellow et al. , 2014b ; Wang et al. , 2019 ; Nøkland , 2015 ) . In particular , it is worth noting that several recent studies have shown that adversarial training can bring many benefits to GANs , not just improving robustness . RGAN ( Zhang et al. , 2020 ) shows that adversarial training can enhance the generalization of the generator and discriminator . Based on both theoretical analysis and empirical results , Zhong et al . ( 2020 ) ; Liu et al . ( 2021 ) ; Liu & Hsieh ( 2019 ) show that updating discriminator ’ s parameters with the loss calculated by adversarial samples generated by authentic images can stabilize training and accelerate convergence . In this paper , we utilize the adversarial sample search method , the iterative fast gradient sign method ( I-FGSM ) ( Kurakin et al. , 2016 ) to conduct latent transform and mine optimization-friendly z samples in the GAN training process , which matches up with the procedure of adversarial training . While most existing works make perturbations on pixel space ( real samples or generated samples ) , instead we focus on mining the latent space and making perturbations on latent samples . 3 PROPOSED METHOD . This section presents AdvLatGAN-z to achieve a transformation z∗ ( · ) , and further AdvLatGAN-qua and AdvLatGAN-div to obtain generation mapping G with better generation quality ( i.e. , -qua ) and diversity ( i.e. , -div ) . Both methods are derived from the iterative updating in latent space . 3.1 PRELIMINARIES . We introduce adversarial samples and adversarial training used to derive z∗ ( z ) and devise training algorithms , and the regularization technique in MSGAN ( Mao et al. , 2019 ) for mode seeking . Adversarial samples and adversarial training . Adversarial training aims to defend against adversarial attacks . It uses adversarial samples to calculate the optimized loss . In classification , for each image-label pair ( xi , yi ) ∈ S from the labeled training set , the adversarial samples x′ is defined as : x′ = argmax ||x′−xi||p≤ l ( fθ ( x ′ ) , yi ) ( 1 ) Here is the radius of the closed ball to constrain perturbation . While fθ and l denote the attack target ( a neural net ) and the corresponding loss ( e.g . cross-entropy loss ) . p denotes the norm dimension . Given in total n adversarial samples , adversarial training is fulfilled by the optimization : θ = min θ 1 n n∑ i=1 l ( fθ ( x ′ i ) , yi ) ( 2 ) Mode coverage by regularizing distance between generated samples . MSGAN ( mode seeking GAN ) ( Mao et al. , 2019 ) mitigates mode collapse in conditional generation setting by introducing a regularization term for training G over two latent samples z1 and z2 as follows . The hope is that the two generated targets shall be far away from each other as much as possible : Lms = dI ( G ( c , z1 ) , G ( c , z2 ) ) dz ( z1 , z2 ) ( 3 ) where c is the condition vector for generation , and dI and dz are the distance metrics in the target ( image ) space and latent space , respectively . It encourages the distance to be maintained or even enlarged in the target space to avoid the generated samples being trapped into an aggregation . By integrating the regularization term with the original objective function of cGANs ( Mirza & Osindero , 2014 ) , the final objective for minimization in MSGAN for training G and D is given by : L =Lori + λmsLms Lori =Ec , x [ logD ( c , x ) ] + Ec , z [ log ( 1−D ( c , G ( c , z ) ) ) ] ( 4 ) where Lori denotes the original objective , x denotes real data , and λms is the weight hyper-parameter . Note since Eq . 3 is aimed to be maximized hence one can interpret λms to be negative .

Summary: This work proposed a sample shifting method in GAN, formulated as adding intermediate latent space to generated pixel space. Such method is based on observation of continuous mapping limit: image quality in pixel space is not as continuous as latent space; limited latent space will incur mode collapse thus poor image diversity. The main contributions are: a new optimization problem as sampling method to improve image generation by quality and diversity, propose to use I-FGSM optimization method to achieve this sampling optimization problem. The experiment showed improvement on public dataset of STL-10, CIFAR-10.