Diversity-Sensitive Conditional Generative Adversarial Networks

1 INTRODUCTION . The objective of conditional generative models is learning a mapping function from input to output distributions . Since many conditional distributions are inherently ambiguous ( e.g . predicting the future of a video from past observations ) , the ideal generative model should be able to learn a multi-modal mapping from inputs to outputs . Recently , Conditional Generative Adversarial Networks ( cGAN ) have been successfully applied to a wide-range of conditional generation tasks , such as image-to-image translation ( Isola et al. , 2017 ; Wang et al. , 2018 ; Zhu et al. , 2017a ) , image inpainting ( Pathak et al. , 2016 ; Iizuka et al. , 2017 ) , text-to-image synthesis ( Huang et al. , 2017 ; Hong et al. , 2018 ) , video generation ( Villegas et al. , 2017 ) , etc .. In conditional GAN , the generator learns a deterministic mapping from input to output distributions , where the multi-modal nature of the mapping is handled by sampling random latent codes from a prior distribution . However , it has been widely observed that conditional GANs are often suffered from the mode collapse problem ( Salimans et al. , 2016 ; Arjovsky & Bottou , 2017 ) , where only small subsets of output distribution are represented by the generator . The problem is especially prevalent for highdimensional input and output , such as images and videos , since the model is likely to observe only one example of input and output pair during training . To resolve such issue , there has been recent attempts to learn multi-modal mapping in conditional generative models ( Zhu et al. , 2017b ; Huang et al. , 2018 ) . However , they are focused on specific conditional generation tasks ( e.g . image-toimage translation ) and require specific network architectures and objective functions that sometimes are not easy to incorporate into the existing conditional GANs . In this work , we introduce a simple method to regularize the generator in conditional GAN to resolve the mode-collapse problem . Our method is motivated from an observation that the mode-collapse happens when the generator maps a large portion of latent codes to similar outputs . To avoid this , we propose to encourage the generator to produce different outputs depending on the latent code , so as to learn a one-to-one mapping from the latent codes to outputs instead of many-to-one . Despite the simplicity , we show that the proposed method is widely applicable to various cGAN architectures and tasks , and outperforms more complicated methods proposed to achieve multi-modal conditional generation for specific tasks . Additionally , we show that we can control a balance between visual ∗Equal contribution quality and diversity of generator outputs with the proposed formulation . We demonstrate the effectiveness of the proposed method in three representative conditional generation tasks , where most existing cGAN approaches produces deterministic outputs : Image-to-image translation , image inpainting and video prediction . We show that simple addition of the proposed regularization to the existing cGAN models effectively induces stochasticity from the generator outputs . 2 RELATED WORK . Resolving the mode-collapse problem in GAN is an important research problem , and has been extensively studied in the standard GAN settings ( Metz et al. , 2017 ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Salimans et al. , 2016 ; Miyato et al. , 2018 ) . These approaches include unrolling the generator gradient update steps ( Metz et al. , 2017 ) , incorporating the minibatch statistics into the discriminator ( Salimans et al. , 2016 ) , employing the improved divergence measure to smooth the loss landscape of the discriminator ( Gulrajani et al. , 2017 ; Arjovsky et al. , 2017 ; Miyato et al. , 2018 ) , etc .. Although these approaches have been successful in modeling unconditional data distribution to some extent , recent studies have reported that it is still not sufficient to resolve a mode-collapse problem in many conditional generative tasks , especially for high-dimensional input and output . Recently , some approaches have been proposed to address the mode-collapse issue in conditional GAN . Zhu et al . ( 2017b ) proposed a hybrid model of conditional GAN and Variational Autoencoder ( VAE ) for multi-modal image-to-image translation task . The main idea is designing the generator to be invertible by employing an additional encoder network that predicts the latent code from the generated image . The similar idea has been applied to unsupervised image-to-image translation ( Huang et al. , 2018 ) and stochastic video generation ( Lee et al. , 2018 ) but with non-trivial task-specific modifications . However , these approaches are designed to achieve multi-modal generation in each specific task , and there has been no unified solution that addresses the mode-collapse problem for general conditional GANs . Recently , ( Odena et al. , 2018 ) proposed a method that regularizes the generator by clamping the generator Jacobian within a certain range . Our method shares the similar motivation with ( Odena et al. , 2018 ) but employs a different objective function that simply maximizes the norm of the generator gradient with an optional upper-bound , which we found that works much more stable over a wide range of tasks with less number of hyper-parameters . 3 METHOD . Consider a problem of learning a conditional mapping function G : X → Y , which generates an output y ∈ Y conditioned on the input x ∈ X . Our goal is to learn a multi-modal mapping G : X ×Z → Y , such that an input x can be mapped to multiple and diverse outputs in Y depending on the latent factors encoded in z ∈ Z . To learn such multi-modal mapping G , we consider a conditional Generative Adversarial Network ( cGAN ) , which learns both conditional generator G and discriminator D by optimizing the following adversarial objective : min G max D LcGAN ( G , D ) = Ex , y [ logD ( x , y ) ] + Ex , z [ log ( 1−D ( x , G ( x , z ) ) ) ] . ( 1 ) Although conditional GAN has been proved to work well for many conditional generation tasks , it has been also reported that optimization of Eq . ( 1 ) often suffers from the mode-collapse problem , which in extreme cases leads the generator to learn a deterministic mapping from x to y and ignore any stochasticity induced by z . To address such issue , previous approaches encouraged the generator to learn an invertible mapping from latent code to output by E ( G ( x , z ) ) = z ( Zhu et al. , 2017b ; Huang et al. , 2018 ) . However , incorporating an extra encoding network E into the existing conditional GANs requires non-trivial modification of network architecture and introduce the new training challenges , which limits its applicability to various models and tasks . We introduce a simple yet effective regularization on the generator that directly penalizes its modecollapsing behavior . Specifically , we add the following maximization objective to the generator : max G Lz ( G ) = Ez1 , z2 [ min ( ‖G ( x , z1 ) −G ( x , z2 ) ‖ ‖z1 − z2‖ , τ ) ] , ( 2 ) where ‖·‖ indicates a norm and τ is a bound for ensuring numerical stability . The intuition behind the proposed regularization is very simple : when the generator collapses into a single mode and produces deterministic outputs based only on the conditioning variable x , Eq . ( 2 ) approaches its minimum since G ( x , z1 ) ≈ G ( x , z2 ) for all z1 , z2 ∼ N ( 0,1 ) . By regularizing generator to maximize Eq . ( 2 ) , we force the generator to produce diverse outputs depending on latent code z . Our full objective function can be written as : min G max D LcGAN ( G , D ) − λLz ( G ) , ( 3 ) where λ controls an importance of the regularization , thus , the degree of stochasticity in G. If G has bounded outputs through a non-linear output function ( e.g . sigmoid ) , we remove the margin from Eq . ( 2 ) in practice and control its importance only with λ . In this case , adding our regularization introduces only one additional hyper-parameter . The proposed regularization is simple , general , and can be easily integrated into most existing conditional GAN objectives . In the experiment , we show that our method can be applied to various models under different objective functions , network architectures , and tasks . In addition , our regularization allows an explicit control over a degree of diversity via hyper-parameter λ . We show that different types of diversity emerge with different λ . Finally , the proposed regularization can be extended to incorporate different distance metrics to measure the diversity of samples . We show this extension using distance in feature space and for sequence data . 4 ANALYSIS OF THE PROPOSED REGULARIZATION . Connection to Generator Gradient . We show in Appendix A that the proposed regularization in Eq . ( 2 ) corresponds to a lower-bound of averaged gradient norm of G over [ z1 , z2 ] as : Ez1 , z2 [ ‖G ( x , z2 ) −G ( x , z1 ) ‖ ‖z2 − z1‖ ] ≤ Ez1 , z2 [ ∫ 1 0 ‖∇zG ( x , γ ( t ) ) ‖dt ] ( 4 ) where γ ( t ) = tz2 + ( 1− t ) z1 is a straight line connecting z1 and z2 . It implies that optimizing our regularization ( LHS of Eq . ( 4 ) ) will increase the gradient norm of the generator ‖∇zG‖ . It has been known that the GAN suffers from a gradient vanishing issue ( Arjovsky & Bottou , 2017 ) since the gradient of optimal discriminator vanishes almost everywhere ∇D ≈ 0 except near the true data points . To avoid this issue , many previous works had been dedicated to smoothing out the loss landscape of D so as to relax the vanishing gradient problem ( Arjovsky et al. , 2017 ; Miyato et al. , 2018 ; Gulrajani et al. , 2017 ; Kurach et al. , 2018 ) . Instead of smoothing ∇D by regularizing discriminator , we increase ‖∇zG‖ to encourage G ( x , z ) to be more spread over the output space from the fixed zj ∼ p ( z ) , so as to capture more meaningful gradient from D. Optimization Perspective . We provide another perspective to understand how the proposed method addresses the mode-collapse problem . For notational simplicity , here we omit the conditioning variable from the generator and focus on a mapping of latent code to output Gθ : Z → Y . Let a modeM denotes a set of data points in an output space Y , where all elements of the mode have very small differences that are perceptually indistinguishable . We consider that the mode-collapse happens if the generator maps a large portion of latent codes to the modeM . Under this definition , we are interested in a situation where the generator output Gθ ( z1 ) for a certain latent code z1 moves closer to a mode M by a distance of via a single gradient update . Then we show in Appendix B that such gradient update at z1 will also move the generator outputs of neighbors in a neighborhood Nr ( z1 ) to the same mode M. In addition , the size of neighborhood Nr ( z1 ) can be arbitrarily large but is bounded by an open ball of a radius r = · ( 4 infz { max ( ‖Gθt ( z1 ) −Gθt ( z ) ‖ ‖z1−z‖ , ‖Gθt+1 ( z1 ) −Gθt+1 ( z ) ‖ ‖z1−z‖ ) } ) −1 , where θt and θt+1 denote the generator parameters before and after the gradient update , respectively . Without any constraints on ‖G ( z1 ) −G ( z2 ) ‖‖z1−z2‖ , a single gradient update can cause the generator outputs for a large amount of latent codes to be collapsed into a modeM . We propose to shrink the size of such neighborhood by constraining ‖G ( z1 ) −G ( z2 ) ‖‖z1−z2‖ above some threshold τ > 0 , therefore prevent the generator placing a large probability mass around a modeM . Connection with BicycleGAN ( Zhu et al. , 2017b ) . We establish an interesting connection of our regularization with Zhu et al . ( 2017b ) . Recall that the objective of BicycleGAN is encouraging an invertibility of a generator by minimizing ‖z−E ( G ( z ) ) ‖1 . By taking derivative with respect to z , it implies that optimalE will satisfy I = ∇GE ( G ( z ) ) ∇zG ( z ) . Because an ideal encoderE should be robust against spurious perturbations from inputs , we can naturally assume that the gradient norm of E should not be very large . Therefore , to maintain invertibility , we expect the gradient of G should not be zero , i.e . ‖∇zG ( z ) ‖ > τ for some τ > 0 , which prevents a gradient of the generator being vanishing . It is related to our idea that penalizes a vanishing gradient of the generator . Contrary to BicycleGAN , however , our method explicitly optimizes a generator gradient to have a reasonably high norm . It also allows us to control a degree of diversity with a hyper-parameter λ .

Summary: The paper proposes a method for generating diverse outputs for various conditional GAN frameworks including image-to-image translation, image-inpainting, and video prediction. The idea is quite simple, simply adding a regularization term so that the output images are sensitive to the input variable that controls the variation of the images. (Note that the variable is not the conditional input to the network.) The paper also shows how the regularization term is related to the gradient penalty term. The most exciting feature about the work is that it can be applied to various conditional synthesis frameworks for various tasks. The paper includes several experiments with comparison to the state-of-the-art. The achieved performance is satisfactory.