Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation

1 INTRODUCTION . Recently , active research has led to a huge progress on conditional image generation , whose typical tasks include image-to-image translation ( Isola et al . ( 2017 ) ) , image inpainting ( Pathak et al . ( 2016 ) ) , super-resolution ( Ledig et al . ( 2017 ) ) and video prediction ( Mathieu et al . ( 2016 ) ) . At the core of such advances is the success of conditional GANs ( Mirza & Osindero ( 2014 ) ) , which improve GANs by allowing the generator to take an additional code or condition to control the modes of the data being generated . However , training GANs , including conditional GANs , is highly unstable and easy to collapse ( Goodfellow et al . ( 2014 ) ) . To mitigate such instability , almost all previous models in conditional image generation exploit the reconstruction loss such as ` 1/ ` 2 loss in addition to the GAN loss . Indeed , using these two types of losses is synergetic in that the GAN loss complements the weakness of the reconstruction loss that output samples are blurry and lack high-frequency structure , while the reconstruction loss offers the training stability required for convergence . In spite of its success , we argue that it causes one critical side effect ; the reconstruction loss aggravates the mode collapse , one of notorious problems of GANs . In conditional generation tasks , which are to intrinsically learn one-to-many mappings , the model is expected to generate diverse outputs from a single conditional input , depending on some stochastic variables ( e.g . many realistic street scene images for a single segmentation map ( Isola et al. , 2017 ) ) . Nevertheless , such stochastic input rarely generates any diversity in the output , and surprisingly many previous methods omit a random noise source in their models . Most papers rarely mention the necessity of random noise , and a few others report that the model completely ignores the noise even if it is fed into the model . For example , Isola et al . ( 2017 ) state that the generator simply learns to ignore the noise , and even dropout fails to incur meaningful output variation . The objective of this paper is to propose a new set of losses named moment reconstruction losses that can replace the reconstruction loss with losing neither the visual fidelity nor diversity in output samples . The core idea is to use maximum likelihood estimation ( MLE ) loss ( e.g . ` 1/ ` 2 loss ) to predict conditional statistics of the real data distribution instead of applying it directly to the generator as done in most existing algorithms . Then , we assist GAN training by enforcing the generated distribution to match its statistics to the statistics of the real distribution . In summary , our major contributions are three-fold . First , we show that there is a significant mismatch between the GAN loss and the reconstruction loss , thereby the model can not achieve the optimality w.r.t . both losses . Second , we propose two novel loss functions that enable the model to accomplish both training stability and multimodal output generation . Our methods simply replace the reconstruction loss , and thus are applicable to any conditional generation tasks . Finally , we show the effectiveness and generality of our methods through extensive experiments on three generation tasks , including image-to-image translation , super-resolution and image inpainting , where our methods outperform recent strong baselines in terms of realism and diversity . 2 RELATED WORKS . Conditional Generation Tasks . Since the advent of GANs ( Goodfellow et al . ( 2014 ) ) and conditional GANs ( Mirza & Osindero ( 2014 ) ) , there has been a large body of work in conditional generation tasks . A non-exhaustive list includes image translation ( Isola et al. , 2017 ; Wang et al. , 2017 ; 2018 ; Zhang et al. , 2017b ; Lyu et al. , 2017 ; Sangkloy et al. , 2017 ; Xian et al. , 2017 ; Zhang et al. , 2017a ; Liu et al. , 2017 ) , super-resolution ( Ledig et al. , 2017 ; Xie et al. , 2018 ; Mahapatra et al. , 2017 ; Xu et al. , 2017 ; Bulat et al. , 2018 ; Sajjadi et al. , 2017 ) , image inpainting ( Pathak et al. , 2016 ; Iizuka et al. , 2017 ; Ouyang et al. , 2018 ; Olszewski et al. , 2017 ; Yu et al. , 2018 ; Sabini & Rusak , 2018 ; Song et al. , 2018 ; Li et al. , 2017 ; Yang et al. , 2018 ; Yeh et al. , 2017 ) and video prediction ( Mathieu et al. , 2016 ; Jang et al. , 2018 ; Lu et al. , 2017 ; Villegas et al. , 2017 ; Zhou & Berg , 2016 ; Bhattacharjee & Das , 2017 ; Vondrick & Torralba , 2017 ) . However , existing models have one common limitation : lack of stochasticity for diverse output . Despite the fact that the tasks are to be one-to-many mapping , they ignore random noise input which is necessary to generate diverse samples from a single conditional input . A number of works such as ( Mathieu et al. , 2016 ; Isola et al. , 2017 ; Xie et al. , 2018 ) try injecting random noise into their models but discover that the models simply discard it and instead learn a deterministic mapping . Multimodality Enhancing Models . It is not fully understood yet why conditional GAN models fail to learn the full multimodality of data distribution . Recently , there has been a series of attempts to incorporate stochasticity in conditional generation as follows . ( 1 ) Conditional VAE-GAN . VAE-GAN ( Larsen et al. , 2016 ) is a hybrid model that combines the decoder in VAE ( Kingma & Welling , 2014 ) with the generator in GAN ( Goodfellow et al. , 2014 ) . Its conditional variants have been also proposed such as CVAE-GAN ( Bao et al. , 2017 ) , BicycleGAN ( Zhu et al. , 2017 ) and SAVP ( Lee et al. , 2018a ) . These models harness the strengths of the two models , output fidelity by GANs and diversity by VAEs , to produce a wide range of realistic images . Intuitively , the VAE structure drives the generator to exploit latent variables to represent the multimodality of the conditional distribution . ( 2 ) Disentangled representation . Huang et al . ( 2018 ) and Lee et al . ( 2018b ) propose to learn disentangled representation for multimodal unsupervised image-to-image translation . These models split the embedding space into a domain-invariant space for sharing information across domains and a domain-specific space for capturing styles and attributes . These models encode an input into the domain-invariant embedding and sample domain-specific embedding from some prior distribution . The two embeddings are fed into the decoder of the target domain , for which the model can generate diverse samples . Conditional VAE-GANs and disentanglement-based methods both leverage the latent variable to prevent the model from discarding the multimodality of output samples . On the other hand , we present a simpler and orthogonal direction to achieve multimodal conditional generation by introducing novel loss functions that can replace the reconstruction loss . 3 LOSS MISMATCH OF CONDITIONAL GANS . We briefly review the objective of conditional GANs in section 3.1 and discuss why the two loss terms cause the loss of modality in the sample distribution of the generator in section 3.2 . 3.1 PRELIMINARY : THE OBJECTIVE OF CONDITIONAL GANS . Conditional GANs aims at learning to generate samples that are indistinguishable from real data for a given input . The objective of conditional GANs usually consists of two terms , the GAN loss LGAN and the reconstruction loss LRec . L = LGAN + λRecLRec ( 1 ) Another popular loss term is the perceptual loss ( Johnson et al. , 2016 ; Bruna et al. , 2016 ; Ledig et al. , 2017 ) . While the reconstruction loss encodes the pixel-level distance , the perceptual loss is defined as the distance between the features encoded by neural networks . Since they share the same form ( e.g . ` 1/ ` 2 loss ) , we consider the perceptual loss as a branch of the reconstruction loss . The loss LGAN is defined to minimize some distance measure ( e.g . JS-divergence ) between the true and generated data distribution conditioned on input x . The training scheme is often formulated as the following minimax game between the discriminator D and the generator G. min G max D Ex , y [ logD ( x , y ) ] + Ex , z [ log ( 1−D ( x , G ( x , z ) ) ) ] ( 2 ) where each data point is a pair ( x , y ) , and G generates outputs given an input x and a random noise z . Note that D also observes x , which is crucial for the performance ( Isola et al. , 2017 ) . The most common reconstruction losses in conditional GAN literature are the ` 1 ( Isola et al. , 2017 ; Wang et al. , 2017 ) and ` 2 loss ( Pathak et al. , 2016 ; Mathieu et al. , 2016 ) . Both losses can be formulated as follows with p = 1 , 2 , respectively . LRec = Lp = Ex , y , z [ ‖y −G ( x , z ) ‖pp ] . ( 3 ) These two losses naturally stem from the maximum likelihood estimations ( MLEs ) of the parameters of Laplace and Gaussian distribution , respectively . The likelihood of dataset ( X , Y ) assuming each distribution is defined as follows . pL ( Y|X ; θ ) = N∏ i=1 1 2b exp ( −|yi − fθ ( xi ) | b ) , pG ( Y|X ; θ ) = N∏ i=1 1√ 2πσ2 exp ( − ( yi − fθ ( xi ) ) 2 2σ2 ) whereN is the size of the dataset , and f is the model parameterized by θ . The central and dispersion measure for Gaussian are the mean and variance σ2 , and the correspondences for Laplace are the median and mean absolute deviation ( MAD ) b . Therefore , using ` 2 loss leads the model output fθ ( x ) to become an estimate of the conditional average of y given x , while using ` 1 loss is equivalent to estimating the conditional median of y given x ( Bishop , 2006 ) . Note that the model fθ is trained to predict the mean ( or median ) of the data distribution , not to generate samples from the distribution . 3.2 LOSS OF MODALITY BY THE RECONSTRUCTION LOSS . We argue that the joint use of the reconstruction loss with the GAN loss can be problematic , because it may worsen the mode collapse . Below , we discuss this argument both mathematically and empirically . One problem of the ` 2 loss is that it forces the model to predict only the mean of p ( y|x ) , while pushing the conditional variance to zero . According to James ( 2003 ) , for any symmetric loss function Ls and an estimator ŷ for y , the loss is decomposed into one irreducible term Var ( y ) and two reducible terms , SE ( ŷ , y ) and VE ( ŷ , y ) , where SE refers to systematic effect , the change in error caused by the bias of the output , while VE refers to variance effect , the change in error caused by the variance of the output . Ey , ŷ [ Ls ( y , ŷ ) ] =Ey [ Ls ( y , Sy ) ] ︸ ︷︷ ︸ Var ( y ) +Ey [ Ls ( y , Sŷ ) − Ls ( y , Sy ) ] ︸ ︷︷ ︸ SE ( ŷ , y ) + Ey , ŷ [ Ls ( y , ŷ ) − Ls ( y , Sŷ ) ] ︸ ︷︷ ︸ VE ( ŷ , y ) , ( 4 ) where S is an operator that is defined to be Sy = argmin µ Ey [ Ls ( y , µ ) ] . Notice that the total loss is minimized when ŷ = Sŷ = Sy , reducing both SE and VE to 0 . For ` 2 loss , Sy and Sŷ are the expectations of y and ŷ . Furthermore , SE and VE correspond to the squared bias and the variance , respectively . SE ( ŷ , y ) = Ey [ ( y − Sŷ ) 2 − ( y − Sy ) 2 ] = ( Sŷ − Sy ) 2 , ( 5 ) VE ( ŷ , y ) = Ey , ŷ [ ( y − ŷ ) 2 − ( y − Sŷ ) 2 ] = Eŷ [ ( ŷ − Sŷ ) 2 ] . ( 6 ) The decomposition above demonstrates that the minimization of ` 2 loss is equivalent to the minimization of the bias and the variance of prediction . In the context of conditional generation tasks , this implies that ` 2 loss minimizes the conditional variance of output . Figure 1 shows some examples of the Pix2Pix model ( Isola et al . ( 2017 ) ) that is applied to translate segmentation labels to realistic photos in Cityscapes dataset ( Cordts et al. , 2016 ) . We slightly modify the model so that it takes additional noise input . We use ` 1 loss as the reconstruction loss as done in the original paper . We train four models that use different combinations of loss terms , and generate four samples with different noise input . As shown in Figure 1 ( c ) , the model trained with only the GAN loss fails to generate realistic images , since the signal from the discriminator is too unstable to learn the translation task . In Figure 1 ( d ) , the model with only ` 1 loss is more stable but produces results far from being realistic . The combination of the two losses in Figure 1 ( e ) helps not only to reliably train the model but also to generate visually appealing images ; yet , its results lack variation . This phenomenon is also reported in ( Mathieu et al. , 2016 ; Isola et al. , 2017 ) , although the cause is unknown . Pathak et al . ( 2016 ) and Iizuka et al . ( 2017 ) even state that better results are obtained without noise in the models . Finally , Figure 1 ( f ) shows that our new objective enables the model to generate not only visually appealing but also diverse output samples .

Summary: The paper proposes a modification to the traditional conditional GAN objective (which minimizes GAN loss as well as either L1 or L2 pixel-wise reconstruction losses) in order to promote diverse, multimodal generation of images. The modification involves replacing the L1/L2 reconstruction loss -- which predicts the first moment of a pixel-wise gaussian/laplace (respectively) likelihood model assuming a constant spherical covariance matrix -- with a new objective that matches the first and second moments of a pixel-wise gaussian/laplace likelihood model with diagonal covariance matrix. Two models are proposed for matching the first and second moments - the first one involves using a separate network to predict the moments from data which are then used to match the generator’s empirical estimates of the moments (using K samples of generated images). The second involves directly matching the empirical moment estimates using monte carlo.