Improving Generative Adversarial Networks via Adversarial Learning in Latent Space

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014a ) have shown effectiveness for generating high-fidelity data , especially for images under various settings ( Ledig et al. , 2017 ; Wang et al. , 2018 ; Zhu et al. , 2017 ; Zhan et al. , 2019 ; Chakraborty et al. , 2018 ) . Based on the zero-sum game , the model is trained by the adversarial process between the generator and the discriminator . Many efforts have been made to achieve a more realistic generation from different perspectives . WGAN ( Arjovsky et al. , 2017 ) , SNGAN ( Miyato et al. , 2018 ) , LSGAN ( Mao et al. , 2017 ) aim to design better objective functions . ImprovedGAN ( Salimans et al. , 2016 ) , AC-GAN ( Odena et al. , 2017 ) propose practical training techniques . While more complex network structures are studied in BigGAN ( Brock et al. , 2018 ) , StyleGAN ( Karras et al. , 2019a ) , SAGAN ( Zhang et al. , 2019 ) . Despite the above progress , an intriguing and relatively less studied question is how the latent space affects the generation in terms of quality and diversity , which can be orthogonal to the above frequently studied factors like model architectures and training techniques . Specifically , we argue that the latent samples from a standard continuous distribution ( e.g. , Gaussian ) can often be mapped to varying-quality samples for the generation . One reason is that the generator is a continuous function ( by neural nets ) while the generation quality ( e.g. , images in pixel space ) is not . Moreover , it has also been shown in GAN literature ( Khayatkhoei et al. , 2018 ) that many image objects lie upon multiple disjoint manifolds , and a continuous mapping from continuous latent distribution , e.g. , Gaussian must generate many invalid samples in the between of the manifolds when the generator seeks to cover all the meaningful modes . See Fig . 1 ( a ) for the schematic diagram . This fact also implies a dilemma : for complex real-world data , there is a trade-off between quality and diversity when the latent vectors are sampled from a simple uni-mode distribution , e.g. , Gaussian , and the generator is a continuous mapping based on neural networks . Moreover , by realizing that the mapping function in fact allows for many-to-one mapping , mode collapse ( Srivastava et al. , 2017 ) can naturally happen when different latent samples are mapped into limited or even a single mode in target space . The above contradiction between the discontinuity in generation data quality ( e.g. , images ) and the continuity of latent distribution as well as mapping function , can be one of the significant factors to the well-known difficulty in GANs training , especially for generators , as one unreasonably hopes to enforce a discontinuous mapping via continuous neural nets . Meanwhile , the mapping property can largely affect the generative performance for functions with continuous nature , which is associated with most previous works devoted to improving generative performance . To tackle these issues , we impose an extra ( implicit ) transform function z∗ ( · ) on the raw sampling z , and the generation can be written by G ( z∗ ( z ) ) . From this perspective , most existing works , including the vanilla GAN , employ z∗ ( · ) with an identical form . Specifically , the implicit function based transform z∗ ( · ) can be achieved by updating the latent z ∼ p ( z ) to minimize lossG ( i.e . log ( 1−D ( G ( z ) ) ) ) using iterative fast gradient sign method ( I-FGSM ) ( Kurakin et al. , 2016 ) with the parameters of G and D fixed , and we name the method as AdvLatGAN-z . See Fig . 1 ( b ) for the schematic diagram . On top of the first effort , we propose a novel training strategy to improve the generative mapping G respectively for quality and diversity . See Fig . 2 for the schematic diagram . This is achieved by introducing z iterations ( i.e. , targeted implicit transforms on z ) into GAN training , specifically , updating z ∼ p ( z ) using I-FGSM to find more optimization-friendly ones in the bi-level optimization of vanilla GAN . Respectively for quality and diversity , we use two iterative updating strategies for z under different objective functions in training , and we name the two training algorithms respectively as AdvLatGAN-qua and AdvLatGAN-div . The highlights of this paper include : 1 ) We rethink the role of the generator as a continuous function , which may incur quality discontinuity in target space and mode aggregation ( collapse ) when the latent samples are directly sampled from a continuous distribution e.g . Gaussian . Based on above theoretical considerations , we propose a composite generation perspective by formulating the mapping fromG ( z ) toG ( z∗ ( z ) ) . This treatment provides a new methodological guidance to the progress of the generative model . 2 ) We propose a new pipeline for both G ( · ) and z∗ ( · ) , by first training generative mappings for better generation quality and diversity , respectively , and then the transform z∗ ( · ) is enforced and computed implicitly to obtain an improved distribution for more effective generation . Specifically , the training involves adversarial learning under different objective functions for quality and diversity , leading to new GAN models : AdvLatGAN-qua/div and their further variants . 3 ) Experimental results on synthetic and visual data show the notable improvement of our approach on both generation quality and diversity . Our technique is orthogonal to existing GAN techniques . 2 RELATED WORK . Implicitly adapting the raw latent vectors z from Gaussian to improve the generation relates to latent space learning for GAN . While the specific training algorithm using latent vector adapting to mine optimization-friendly samples in training relates to adversarial training , which views the latent adapting in training as an attack to mine hard samples . Latent space exploration in GANs . Techniques have been proposed for latent space exploration for GANs . By utilizing the steerability of GANs in latent space , StyleGAN ( Karras et al. , 2019a ) introduce an latent space transform by using a mapping network to better decouple the various styles and Jahanian et al . ( 2019 ) shift the distribution to fit camera movement transformation . InterFaceGAN ( Shen et al. , 2020 ) decouples entangled semantics with subspace projection to control facial attributes more precisely . Mishra et al . ( 2020 ) , Mukherjee et al . ( 2019 ) and Wu et al . ( 2019 ) introduce the latent exploration into training to achieve clustering objectives or benefit GAN training . In addition to the preceding works , some recent works focus on improving latent sampling quality to generate high-quality images . For example , interactive evolutionary computation is used to make the generation process more controllable in Bontrager et al . ( 2018 ) , which meanwhile also brings additional manual efforts . To reduce manual intervention , Roziere et al . ( 2020 ; 2021 ) use Koncept512 ( Hosu et al. , 2020 ) as a criterion to improve the quality of fake images . However , as discussed in Roziere et al . ( 2020 ) , this strategy is somehow biased , can not work well in each dataset ( e.g. , Pokemons ) . The reason is that the quality estimator is not even designated to access the real-data distribution information . Instead , the generation is guided by a predefined score function . Compared to the above works , we take a relevant idea to Roziere et al . ( 2020 ) that also adapt the latent vectors to an improved condition before input to the generator G. Differently , our latent adaption approach incorporates the target distribution and utilize I-FGSM iterations to adapt latent vectors . Mitigating mode collapse for GANs . There are also efforts in mitigating mode collapse , which are based on the thinking that many real-world samples lie in disjoint manifolds ( Khayatkhoei et al. , 2018 ) . One way is to introduce a collection of generators such that each generator may cover a specific mode since a single generator can hardly fulfill a discontinuous function ( Khayatkhoei et al. , 2018 ) . However , more works focus on improving the properties of mapping , explicitly obtaining a mapping covering more modes ( Elfeki et al. , 2019 ; Meulemeester et al. , 2020 ; Mao et al. , 2019 ) . Note that the generated distribution obtained by this method will contain invalid parts due to the continuity of the latent distribution and the mapping . This calls for selective sampling of the continuous latent distribution or directly amending the latent distribution , which is rarely noticed in these works . Adversarial samples and adversarial training . Adversarial samples are malicious samples that can easily fool deep models , which are indistinguishable to human eyes . Since the seminal work ( Szegedy et al. , 2013 ) , various adversarial attack methods have been devised ( Goodfellow et al. , 2014b ; Akhtar & Mian , 2018 ; Kurakin et al. , 2016 ; Madry et al. , 2017 ) . To defend the models from attacks , adversarial training is meanwhile developed ( Goodfellow et al. , 2014b ; Wang et al. , 2019 ; Nøkland , 2015 ) . In particular , it is worth noting that several recent studies have shown that adversarial training can bring many benefits to GANs , not just improving robustness . RGAN ( Zhang et al. , 2020 ) shows that adversarial training can enhance the generalization of the generator and discriminator . Based on both theoretical analysis and empirical results , Zhong et al . ( 2020 ) ; Liu et al . ( 2021 ) ; Liu & Hsieh ( 2019 ) show that updating discriminator ’ s parameters with the loss calculated by adversarial samples generated by authentic images can stabilize training and accelerate convergence . In this paper , we utilize the adversarial sample search method , the iterative fast gradient sign method ( I-FGSM ) ( Kurakin et al. , 2016 ) to conduct latent transform and mine optimization-friendly z samples in the GAN training process , which matches up with the procedure of adversarial training . While most existing works make perturbations on pixel space ( real samples or generated samples ) , instead we focus on mining the latent space and making perturbations on latent samples . 3 PROPOSED METHOD . This section presents AdvLatGAN-z to achieve a transformation z∗ ( · ) , and further AdvLatGAN-qua and AdvLatGAN-div to obtain generation mapping G with better generation quality ( i.e. , -qua ) and diversity ( i.e. , -div ) . Both methods are derived from the iterative updating in latent space . 3.1 PRELIMINARIES . We introduce adversarial samples and adversarial training used to derive z∗ ( z ) and devise training algorithms , and the regularization technique in MSGAN ( Mao et al. , 2019 ) for mode seeking . Adversarial samples and adversarial training . Adversarial training aims to defend against adversarial attacks . It uses adversarial samples to calculate the optimized loss . In classification , for each image-label pair ( xi , yi ) ∈ S from the labeled training set , the adversarial samples x′ is defined as : x′ = argmax ||x′−xi||p≤ l ( fθ ( x ′ ) , yi ) ( 1 ) Here is the radius of the closed ball to constrain perturbation . While fθ and l denote the attack target ( a neural net ) and the corresponding loss ( e.g . cross-entropy loss ) . p denotes the norm dimension . Given in total n adversarial samples , adversarial training is fulfilled by the optimization : θ = min θ 1 n n∑ i=1 l ( fθ ( x ′ i ) , yi ) ( 2 ) Mode coverage by regularizing distance between generated samples . MSGAN ( mode seeking GAN ) ( Mao et al. , 2019 ) mitigates mode collapse in conditional generation setting by introducing a regularization term for training G over two latent samples z1 and z2 as follows . The hope is that the two generated targets shall be far away from each other as much as possible : Lms = dI ( G ( c , z1 ) , G ( c , z2 ) ) dz ( z1 , z2 ) ( 3 ) where c is the condition vector for generation , and dI and dz are the distance metrics in the target ( image ) space and latent space , respectively . It encourages the distance to be maintained or even enlarged in the target space to avoid the generated samples being trapped into an aggregation . By integrating the regularization term with the original objective function of cGANs ( Mirza & Osindero , 2014 ) , the final objective for minimization in MSGAN for training G and D is given by : L =Lori + λmsLms Lori =Ec , x [ logD ( c , x ) ] + Ec , z [ log ( 1−D ( c , G ( c , z ) ) ) ] ( 4 ) where Lori denotes the original objective , x denotes real data , and λms is the weight hyper-parameter . Note since Eq . 3 is aimed to be maximized hence one can interpret λms to be negative .

Summary: The authors demonstrate that the generator in a GAN is a continuous function two latent codes that are close in the latent space are mapped to two images that are close in the pixel space. However, the quality of the generated images is not preserved as quality is not a continuous function in pixel space. To address this issue, the authors propose to transform the original latent codes and demonstrate that it results in better generation quality and diversity.