Title: HARMONIZING MAXIMUM LIKELIHOOD WITH GANS FOR MULTIMODAL CONDITIONAL GENERATION

Abstract: Recent advances in conditional image generation tasks, such as image-to-image translation and image inpainting, are largely accounted to the success of conditional GAN models, which are often optimized by the joint use of the GAN loss with the reconstruction loss. However, we reveal that this training recipe shared by almost all existing methods causes one critical side effect: lack of diversity in output samples. In order to accomplish both training stability and multimodal output generation, we propose novel training schemes with a new set of losses named moment reconstruction losses that simply replace the reconstruction loss. We show that our approach is applicable to any conditional generation tasks by performing thorough experiments on image-to-image translation, super-resolution and image inpainting using Cityscapes and CelebA dataset. Quantitative evaluations also confirm that our methods achieve a great diversity in outputs while retaining or even improving the visual fidelity of generated samples.

1. INTRODUCTION:

Recently, active research has led to a huge progress on conditional image generation, whose typical tasks include image-to-image translation (Isola et al. (2017) ), image inpainting (Pathak et al. (2016) ), super-resolution (Ledig et al. (2017) ) and video prediction (Mathieu et al. (2016) ). At the core of such advances is the success of conditional GANs (Mirza & Osindero (2014) ), which improve GANs by allowing the generator to take an additional code or condition to control the modes of the data being generated. However, training GANs, including conditional GANs, is highly unstable and easy to collapse (Goodfellow et al. (2014) ). To mitigate such instability, almost all previous models in conditional image generation exploit the reconstruction loss such as 1 / 2 loss in addition to the GAN loss. Indeed, using these two types of losses is synergetic in that the GAN loss complements the weakness of the reconstruction loss that output samples are blurry and lack high-frequency structure, while the reconstruction loss offers the training stability required for convergence.

In spite of its success, we argue that it causes one critical side effect; the reconstruction loss aggravates the mode collapse, one of notorious problems of GANs. In conditional generation tasks, which are to intrinsically learn one-to-many mappings, the model is expected to generate diverse outputs from a single conditional input, depending on some stochastic variables (e.g. many realistic street scene images for a single segmentation map (Isola et al., 2017) ). Nevertheless, such stochastic input rarely generates any diversity in the output, and surprisingly many previous methods omit a random noise source in their models. Most papers rarely mention the necessity of random noise, and a few others report that the model completely ignores the noise even if it is fed into the model. For example, Isola et al. (2017) state that the generator simply learns to ignore the noise, and even dropout fails to incur meaningful output variation.

The objective of this paper is to propose a new set of losses named moment reconstruction losses that can replace the reconstruction loss with losing neither the visual fidelity nor diversity in output samples. The core idea is to use maximum likelihood estimation (MLE) loss (e.g. 1 / 2 loss) to predict conditional statistics of the real data distribution instead of applying it directly to the generator as done in most existing algorithms. Then, we assist GAN training by enforcing the generated distribution to match its statistics to the statistics of the real distribution. In summary, our major contributions are three-fold. First, we show that there is a significant mismatch between the GAN loss and the reconstruction loss, thereby the model cannot achieve the optimality w.r.t. both losses. Second, we propose two novel loss functions that enable the model to accomplish both training stability and multimodal output generation. Our methods simply replace the reconstruction loss, and thus are applicable to any conditional generation tasks. Finally, we show the effectiveness and generality of our methods through extensive experiments on three generation tasks, including image-to-image translation, super-resolution and image inpainting, where our methods outperform recent strong baselines in terms of realism and diversity.

2. RELATED WORKS:

Conditional Generation Tasks. Since the advent of GANs (Goodfellow et al. (2014) ) and conditional GANs (Mirza & Osindero (2014) ), there has been a large body of work in conditional generation tasks. A non-exhaustive list includes image translation (Isola et al., 2017; Wang et al., 2017; 2018; Zhang et al., 2017b; Lyu et al., 2017; Sangkloy et al., 2017; Xian et al., 2017; Zhang et al., 2017a; Liu et al., 2017) , super-resolution (Ledig et al., 2017; Xie et al., 2018; Mahapatra et al., 2017; Xu et al., 2017; Bulat et al., 2018; Sajjadi et al., 2017) , image inpainting (Pathak et al., 2016; Iizuka et al., 2017; Ouyang et al., 2018; Olszewski et al., 2017; Yu et al., 2018; Sabini & Rusak, 2018; Song et al., 2018; Li et al., 2017; Yang et al., 2018; Yeh et al., 2017) and video prediction (Mathieu et al., 2016; Jang et al., 2018; Lu et al., 2017; Villegas et al., 2017; Zhou & Berg, 2016; Bhattacharjee & Das, 2017; Vondrick & Torralba, 2017) .

However, existing models have one common limitation: lack of stochasticity for diverse output. Despite the fact that the tasks are to be one-to-many mapping, they ignore random noise input which is necessary to generate diverse samples from a single conditional input. A number of works such as (Mathieu et al., 2016; Isola et al., 2017; Xie et al., 2018) try injecting random noise into their models but discover that the models simply discard it and instead learn a deterministic mapping.

Multimodality Enhancing Models. It is not fully understood yet why conditional GAN models fail to learn the full multimodality of data distribution. Recently, there has been a series of attempts to incorporate stochasticity in conditional generation as follows.

(1) Conditional VAE-GAN. VAE- GAN (Larsen et al., 2016 ) is a hybrid model that combines the decoder in VAE (Kingma & Welling, 2014) with the generator in GAN (Goodfellow et al., 2014) . Its conditional variants have been also proposed such as CVAE-GAN (Bao et al., 2017) , Bicycle-GAN (Zhu et al., 2017) and SAVP (Lee et al., 2018a) . These models harness the strengths of the two models, output fidelity by GANs and diversity by VAEs, to produce a wide range of realistic images. Intuitively, the VAE structure drives the generator to exploit latent variables to represent the multimodality of the conditional distribution.

(2) Disentangled representation. Huang et al. (2018) and Lee et al. (2018b) propose to learn disentangled representation for multimodal unsupervised image-to-image translation. These models split the embedding space into a domain-invariant space for sharing information across domains and a domain-specific space for capturing styles and attributes. These models encode an input into the domain-invariant embedding and sample domain-specific embedding from some prior distribution. The two embeddings are fed into the decoder of the target domain, for which the model can generate diverse samples.

Conditional VAE-GANs and disentanglement-based methods both leverage the latent variable to prevent the model from discarding the multimodality of output samples. On the other hand, we present a simpler and orthogonal direction to achieve multimodal conditional generation by introducing novel loss functions that can replace the reconstruction loss.

3. LOSS MISMATCH OF CONDITIONAL GANS:

We briefly review the objective of conditional GANs in section 3.1 and discuss why the two loss terms cause the loss of modality in the sample distribution of the generator in section 3.2.

3.1. PRELIMINARY: THE OBJECTIVE OF CONDITIONAL GANS:

Conditional GANs aims at learning to generate samples that are indistinguishable from real data for a given input. The objective of conditional GANs usually consists of two terms, the GAN loss L GAN and the reconstruction loss L Rec .

L = L GAN + λ Rec L Rec (1)

Another popular loss term is the perceptual loss (Johnson et al., 2016; Bruna et al., 2016; Ledig et al., 2017) . While the reconstruction loss encodes the pixel-level distance, the perceptual loss is defined as the distance between the features encoded by neural networks. Since they share the same form (e.g. 1 / 2 loss), we consider the perceptual loss as a branch of the reconstruction loss.

The loss L GAN is defined to minimize some distance measure (e.g. JS-divergence) between the true and generated data distribution conditioned on input x. The training scheme is often formulated as the following minimax game between the discriminator D and the generator G.

EQUATION

where each data point is a pair (x, y), and G generates outputs given an input x and a random noise z. Note that D also observes x, which is crucial for the performance (Isola et al., 2017) .

The most common reconstruction losses in conditional GAN literature are the 1 (Isola et al., 2017; Wang et al., 2017) and 2 loss (Pathak et al., 2016; Mathieu et al., 2016) . Both losses can be formulated as follows with p = 1, 2, respectively.

EQUATION

These two losses naturally stem from the maximum likelihood estimations (MLEs) of the parameters of Laplace and Gaussian distribution, respectively. The likelihood of dataset (X, Y) assuming each distribution is defined as follows.

p L (Y|X; θ) = N i=1 1 2b exp(- |y i -f θ (x i )| b ), p G (Y|X; θ) = N i=1 1 √ 2πσ 2 exp(- (y i -f θ (x i )) 2 2σ 2 )

where N is the size of the dataset, and f is the model parameterized by θ. The central and dispersion measure for Gaussian are the mean and variance σ 2 , and the correspondences for Laplace are the median and mean absolute deviation (MAD) b. Therefore, using 2 loss leads the model output f θ (x) to become an estimate of the conditional average of y given x, while using 1 loss is equivalent to estimating the conditional median of y given x (Bishop, 2006) . Note that the model f θ is trained to predict the mean (or median) of the data distribution, not to generate samples from the distribution.

3.2. LOSS OF MODALITY BY THE RECONSTRUCTION LOSS:

We argue that the joint use of the reconstruction loss with the GAN loss can be problematic, because it may worsen the mode collapse. Below, we discuss this argument both mathematically and empirically.

One problem of the 2 loss is that it forces the model to predict only the mean of p(y|x), while pushing the conditional variance to zero. According to James (2003) , for any symmetric loss function L s and an estimator ŷ for y, the loss is decomposed into one irreducible term Var(y) and two reducible terms, SE(ŷ, y) and VE(ŷ, y), where SE refers to systematic effect, the change in error caused by the bias of the output, while VE refers to variance effect, the change in error caused by the variance of the output.

EQUATION

)

where S is an operator that is defined to be Notice that the total loss is minimized when ŷ = S ŷ = Sy, reducing both SE and VE to 0. For 2 loss, Sy and S ŷ are the expectations of y and ŷ. Furthermore, SE and VE correspond to the squared bias and the variance, respectively.

Sy = arg min µ E y [L s (y, µ)].

EQUATION

EQUATION

)

The decomposition above demonstrates that the minimization of 2 loss is equivalent to the minimization of the bias and the variance of prediction. In the context of conditional generation tasks, this implies that 2 loss minimizes the conditional variance of output.

Figure 1 shows some examples of the Pix2Pix model (Isola et al. (2017) ) that is applied to translate segmentation labels to realistic photos in Cityscapes dataset (Cordts et al., 2016) . We slightly modify the model so that it takes additional noise input. We use 1 loss as the reconstruction loss as done in the original paper. We train four models that use different combinations of loss terms, and generate four samples with different noise input. As shown in Figure 1 (c), the model trained with only the GAN loss fails to generate realistic images, since the signal from the discriminator is too unstable to learn the translation task. In Figure 1 (d), the model with only 1 loss is more stable but produces results far from being realistic. The combination of the two losses in Figure 1 (e) helps not only to reliably train the model but also to generate visually appealing images; yet, its results lack variation. This phenomenon is also reported in (Mathieu et al., 2016; Isola et al., 2017) , although the cause is unknown. Pathak et al. (2016) and Iizuka et al. (2017) even state that better results are obtained without noise in the models. Finally, Figure 1 (f) shows that our new objective enables the model to generate not only visually appealing but also diverse output samples.

4. APPROACH:

We propose novel alternatives for the reconstruction loss that are applicable to virtually any conditional generation tasks. Trained with our new loss terms, the model can accomplish both the stability of training and multimodal generation as already seen in Figure 1 (f). Figure 2 illustrates architectural comparison between conventional conditional GANs and our models with the new loss terms. In conventional conditional GANs, MLE losses are applied to the generator's objective to make sure that it generates output samples well matched to their ground-truth. On the other hand, The second model proxy MR-GAN is a more stable variant of MR-GAN. The predictor P , a twin network of G, is trained with an MLE loss to estimate the conditional mean and variances of y given x. Then, we match the sample mean μ to the predicted μ and the sample variance σ2 to the predicted σ2 .

𝐺 𝐷 ℒ GAN ℒ Rec 𝑥 𝑦 𝑦 𝑧 (a) Conditional GAN 𝐺 𝐷 ℒ GAN 𝜎 2 𝜇 𝑥 𝑦 1:𝐾 ℒ MR 𝑦 𝑧 1:𝐾 (b) MR-GAN 𝐺 𝐷 𝑃 ℒ GAN 𝜎 2 𝜇 ො 𝜇, ො 𝜎 2 ℒ pMR ℒ MLE 𝑥 𝑦 𝑦 1:𝐾 𝑧 1:𝐾 (c) proxy MR-GAN

our key idea is to apply MLE losses to make sure that the statistics, such as mean or variance, of the conditional distribution p(y|x) are similar between the generator's sample distribution and the actual data distribution.

In section 4.1, we extend the 1 and 2 loss to estimate all the parameters of Laplace and Gaussian distribution, respectively. In section 4.2-4.3, we propose two novel losses for conditional GANs. Finally, in section 4.4, we show that the reconstruction loss conflicts with the GAN loss while our losses do not.

4.1. THE MLE FOR MEAN AND VARIANCE:

The 2 loss encourages the model to perform the MLE of the conditional mean of y given x while the variance σ 2 , the other parameter of Gaussian, is assumed to be fixed. If we allow the model to estimate the conditional variance as well, the MLE loss corresponds to

EQUATION

where the model f θ now estimates both μ and σ2 for x. Estimating the conditional variance along with the mean can be interpreted as estimating the heteroscedastic aleatoric uncertainty in Kendall & Gal (2017) where the variance is the measure of aleatoric uncertainty.

For the Laplace distribution, we can derive the similar MLE loss as

EQUATION

)

where m is the predicted median and b is the predicted MAD (Bloesch et al., 2018) . In practice, it is more numerically stable to predict the logarithm of variance or MAD (Kendall & Gal, 2017) .

In the following, we will describe our methods mainly with the 2 loss under Gaussian assumption.

It is straightforward to obtain the Laplace version of our methods for the 1 loss by simply replacing the mean and variance with the median and MAD.

4.2. THE MOMENT RECONSTRUCTION LOSS:

Our first loss function is named Moment Reconstruction (MR) loss, and we call a conditional GAN with the loss as MR-GAN. As depicted in Figure 2 (b), the overall architecture of MR-GAN follows that of a conditional GAN, but there are two important updates. First, the generator produces K different samples ỹ1:K for each input x by varying noise input z. Second, the MLE loss is applied to the sample moments (i.e. mean and variance) in contrast to the reconstruction loss which is used directly on the samples. Since the generator is supposed to approximate the real distribution, we can regard that the moments of the generated distribution estimate the moments of the real distribution.

We approximate the moments of the generated distribution with the sample moments μ and σ2 , which are defined as

EQUATION

The MR loss is defined by plugging μ and σ2 in Eq.( 9) into μ and σ2 in Eq.( 7). The final loss of MR-GAN is the weighted sum of the GAN loss and the MR loss:

EQUATION

)

As a simpler variant, we can use only μ and ignore σ2 . We denote this simpler one as MR 1 (i.e. using mean only) and the original one as MR 2 (i.e. using both mean and variance) according to the number of moments used.

EQUATION

In addition, we can easily derive the Laplace versions of the MR loss that use median and MAD.

4.3. THE PROXY MOMENT RECONSTRUCTION LOSS:

One possible drawback of the MR loss is that the training can be unstable at the early phase, since the irreducible noise in y directly contributes to the total loss. For more stable training, we propose a variant called Proxy Moment Reconstruction (proxy MR) loss and a conditional GAN with the loss named proxy MR-GAN whose overall training scheme is depicted in Figure 2 (c). The key difference is the presence of predictor P , which is a clone of the generator with some minor differences: (i) no noise source as input and (ii) prediction of both mean and variance as output. The predictor is trained prior to the generator by the MLE loss in Eq.( 7) with ground-truth y to predict conditional mean and variance, i.e. μ and σ2 . When training the generator, we utilize the predicted statistics of real distribution to guide the outputs of the generator. Specifically, we match the predicted mean/variance and the mean/variance of the generator's distribution, which is computed by the sample mean μ and variance σ2 from ỹ1:K . Then, we define the proxy MR loss L pMR as the sum of squared errors between predicted statistics and sample statistics:

EQUATION

One possible variant is to match only the first moment µ. As with the MR loss, we denote the original method proxy MR 2 and the variant proxy MR 1 where the number indicates the number of matched moments. Deriving the Laplace version of the proxy MR loss is also straightforward. The detailed algorithms of all eight variants are presented in appendix A.

Compared to proxy MR-GAN, MR-GAN allows the generator to access real data y directly; thus there is no bias caused by the predictor. On the other hand, the use of predictor in proxy MR-GAN provides less variance in target values and leads more stable training especially when the batch or sample size is small. Another important aspect worth comparison is overfitting, which should be carefully considered when using MLE with finite training data. In proxy MR-GAN, we can choose the predictor with the smallest validation loss to avoid overfitting, and freeze it while training the generator. Therefore, the generator trained with the proxy MR loss suffers less from overfitting, compared to that trained with the MR loss that directly observes training data. To sum up, the two methods have their own pros and cons. We will empirically compare the behaviors of these two approaches in section 5.2.

4.4. ANALYSES:

We here show the mismatch between the GAN loss and the 2 loss in terms of the set of optimal generators. We then discuss why our approach does not suffer from the loss of diversity in the output samples. Refer to appendix E for the mismatch of 1 loss.

The sets of optimal generators for the GAN loss and the 2 loss, denoted by G and R respectively, can be formulated as follows:

EQUATION

)

Recall that 2 loss is minimized when both bias and variance are zero, which implies that R is a subset of V where G has no conditional variance:

EQUATION

In conditional generation tasks, however, the conditional variance Var(y|x) is assumed to be nonzero (i.e. diverse output y for a given input x). Thereby we conclude G∩V = ∅, which reduces to G∩ R = ∅. It means that any generator G cannot be optimal for both GAN and 2 loss simultaneously. Therefore, it is hard to anticipate what solution is attained by training and how the model behaves when the two losses are combined. One thing for sure is that the reconstruction loss alone is designed to provide a sufficient condition for mode collapse as discussed in section 3.2. Now we discuss why our approach does not suffer from loss of diversity in the output samples unlike the reconstruction loss in terms of the set of optimal generators. The sets of optimal mappings for our loss terms are formulated as follows:

EQUATION

EQUATION

where M 1 corresponds to MR 1 and proxy MR 1 while M 2 corresponds to MR 2 and proxy MR

2 . It is straightforward to show that G ⊂ M 2 ⊂ M 1 ; if p data (y|x) = p G (y|x

) is satisfied at optimum, the conditional expectations and variations of both sides should be the same too:

EQUATION

To summarize, there is no generator that is both optimal for the GAN loss and the 2 loss since G ∩ R = ∅. Moreover, as the 2 loss pushes the conditional variance to zero, the final solution is likely to lose multimodality. On the other hand, the optimal generator w.r.t. the GAN loss is also optimal w.r.t. our loss terms since G ⊂ M 2 ⊂ M 1 .

This proof may not fully demonstrate why our approach does not give up multimodality, which could be an interesting future work in line with the mode collapsing issue of GANs. Nonetheless, we can at least assert that our loss functions do not suffer from the same side effect as the reconstruction loss does. Another remark is that this proof is confined to conditional GAN models that have no use of latent variables. It may not be applicable to the models that explicitly encode latent variables such as Bao et al. ( 2017 

5. EXPERIMENTS:

In order to show the generality of our methods, we apply them to three conditional generation tasks: image-to-image translation, super-resolution and image inpainting, for each of which we select Pix2Pix, SRGAN (Ledig et al., 2017) and GLCIC (Iizuka et al., 2017) as base models, respectively. We use Maps (Isola et al., 2017) and Cityscapes dataset (Cordts et al., 2016) for image translation and CelebA dataset (Liu et al., 2015) for the other tasks. We minimally modify the base models to include random noise input and train them with MR and proxy MR objectives. We do not use any other loss terms such as perceptual loss, and train the models from scratch. We present the details of training and implementation and more thorough results in the appendix.

5.1. QUALITATIVE EVALUATION:

In every task, our methods successfully generate diverse images as presented in Figure 3 . From qualitative aspects, one of the most noticeable differences between MR loss and proxy MR loss lies in the training stability. We find that proxy MR loss works as stably as the reconstruction loss in all three tasks. On the other hand, we cannot find any working configuration of MR loss in the image inpainting task. Also, the MR loss is more sensitive to the number of samples K generated for each input. In SRGAN experiments, for example, both methods converge reliably with K = 24, while the MR loss often diverges at K = 12. Although the MR loss is simpler and can be trained in an end-to-end manner, its applicability is rather limited compared to the proxy MR loss. We provide generated samples for each configuration in the appendix.

5.2. QUANTITATIVE EVALUATION:

Following Zhu et al. (2017) , we quantitatively measure diversity and realism of generated images. We evaluate our methods on Pix2Pix-Cityscapes, SRGAN-CelebA and GLCIC-CelebA tasks. For each (method, task) pair, we generate 20 images from each of 300 different inputs using the trained model. As a result, the test sample set size is 6,000 in total. For diversity, we measure the average LPIPS score (Zhang et al., 2018) . Among 20 generated images per input, we randomly choose 10 pairs of images and compute conditional LPIPS values. We then average the scores over the test set. For realism, we conduct a human evaluation experiment from 33 participants. We present a real or fake image one at a time and ask participants to tag whether it is real or fake. The images are presented for 1 second for SRGAN/GLCIC and 0.5 second for Pix2Pix, as done in Zhu et al. (2017) . We calculate the accuracy of identifying fake images with averaged F-measure F and use 2(1 -F ) as the realism score. The score is assigned to 1.0 when all samples are completely indistinguishable from real images and 0 when the evaluators make no misclassification.

There are eight configurations of our methods in total, depending on the MLE (Gaussian and Laplace), the number of statistics (one and two) and the loss type (MR and proxy MR). We test all variants for the Pix2Pix task and only Gaussian proxy MR 2 loss for the others. We compare with base models in every task and additionally BicycleGAN (Zhu et al., 2017) for the Pix2Pix task. We use the official implementation by the authors with minimal modification. BicycleGAN has not been applied to image inpainting and super-resolution tasks, for which we do not report its result.

Table 1 summarizes the results. In terms of both realism and diversity, our methods achieve competitive performance compared to the base models, sometimes even better. For instance, in Pix2Pix-Cityscapes, three of our methods, Gaussian MR 1 , Gaussian proxy MR 1 , and Gaussian proxy MR 2 , significantly outperform BicycleGAN and Pix2Pix+noise in both measures. Without exception, the diversity scores of our methods are far greater than those of the baselines while maintaining competitive realism scores. These results confirm that our methods generate a broad spectrum of high-quality images from a single input. Interestingly, MR 1 and proxy MR 1 loss generate comparable or even better outputs compared to MR 2 and proxy MR 2 loss. That is, matching means could be enough to guide GAN training in many tasks. It implies that adding more statistics (i.e. adding more guidance signal to the model) may be helpful in some cases, but generally may not improve the performance. Moreover, additional errors may arise from predicting more statistics, which could degrade the GAN training. 

(c) GLCIC-CelebA

Table 1 : Quantitative evaluation on three (method, dataset) pairs. Realism is measured by 2(1 -F ) where F is the averaged F-measure of identifying fake by human evaluators. Diversity is scored by the average of conditional LPIPS values. In both metrics, the higher the better. In all three tasks, our methods generate highly diverse images with competitive or even better realism.

6. CONCLUSION:

In this work, we pointed out that there is a mismatch between the GAN loss and the conventional reconstruction losses. As alternatives, we proposed a set of novel loss functions named MR loss and proxy MR loss that enable conditional GAN models to accomplish both stability of training and multimodal generation. Empirically, we showed that our loss functions were successfully integrated with multiple state-of-the-art models for image translation, super-resolution and image inpainting tasks, for which our method generated realistic image samples of high visual fidelity and variability on Cityscapes and CelebA dataset.

There are numerous possible directions beyond this work. First, there are other conditional generation tasks that we did not cover, such as text-to-image synthesis, text-to-speech synthesis and video prediction, for which our methods can be directly applied to generate diverse, high-quality samples. Second, in terms of statistics matching, our methods can be extended to explore other higher order statistics or covariance. Third, using the statistics of high-level features may capture additional correlations that cannot be represented with pixel-level statistics.

A ALGORITHMS:

We elaborate on the algorithms of all eight variants of our methods in detail from Algorithm 5 to Algorithm 4. The presented algorithms assume a single input per update, although we use mini-batch training in practice. Also, we use non-saturating GAN loss, -log D(x, ỹ) (Goodfellow, 2016) .

For Laplace MLEs, the statistics that we compute are median and MAD. Unlike mean, however, the gradient of median is defined only in terms of the single median sample. Therefore, a naive implementation would only calculate the gradient for the median sample, which is not effective for training. Therefore, we use a special trick to distribute the gradients to every sample. In proxy MR loss, we first calculate the difference between the predicted median and the sample median, and then add it to samples ỹ1:K to set the target values t 1:K . We consider t 1:K as constants so that the gradient is not calculated for the target values (this is equivalent to Tensor.detach() in PyTorch and tf.stop gradient() in TensorFlow). Finally, we calculate the loss between the target values and samples, not the medians. We use the similar trick for the MR loss.

Algorithm 1 Generator update in Gaussian MR 1 -GAN Require: Generator G, discriminator D, MR loss coefficient λ MR Require: input x, ground truth y, the number of samples K 1: for i = 1 to K do 2:

z i ← GenerateNoise() 3: ỹi ← G(x, z i ) {Generate K samples} 4: end for 5: μ ← 1 K K i=1 ỹi {Sample mean} 6: L MR ← (y -μ) 2 7: L GAN ← 1 K K i=1 -log D(x, y i ) 8: θ G ← Optimize(θ G , ∇ θ G L GAN + λ MR L MR )

Algorithm 2 Generator update in Gaussian MR 2 -GAN Require: Generator G, discriminator D, MR loss coefficient λ MR Require: input x, ground truth y, the number of samples K 1: for i = 1 to K do 2:

z i ← GenerateNoise() 3: ỹi ← G(x, z i ) {Generate K samples} 4: end for 5: μ ← 1 K K i=1 ỹi {Sample mean} 6: σ2 ← 1 K-1 K i=1 (ỹ i -μ) 2 {Sample variance} 7: L MR ← (y-μ) 2 2σ 2 + 1 2 log σ2 8: L GAN ← 1 K K i=1 -log D(x, y i ) 9: θ G ← Optimize(θ G , ∇ θ G L GAN + λ MR L MR ) Algorithm 3 Generator update in Laplace MR 1 -GAN Require: Generator G, discriminator D, MR loss coefficient λ MR Require: input x, ground truth y, the number of samples K 1: for i = 1 to K do 2: z i ← GenerateNoise() 3: ỹi ← G(x, z i ) {Generate K samples} 4: end for 5: m ← med(ỹ 1:K ) {Sample median} 6: t 1:K = detach(ỹ 1:K + (y -m)) 7: L MR ← 1 K K i=1 |t i -ỹi | 8: L GAN ← 1 K K i=1 -log D(x, y i ) 9: θ G ← Optimize(θ G , ∇ θ G L GAN + λ MR L MR )

Algorithm 4 Generator update in Laplace MR 2 -GAN Require: Generator G, discriminator D, MR loss coefficient λ MR Require: input x, ground truth y, the number of samples K 1: for i = 1 to K do 2:

z i ← GenerateNoise() 3: ỹi ← G(x, z i ) {Generate K samples} 4: end for 5: m ← med(ỹ 1:K ) {Sample median} 6: b ← 1 K K i=1 |ỹ i -m| {Sample MAD} 7: t 1:K = detach(ỹ 1:K + (y -m)) 8: L MR ← 1 K K i=1 |ti-ỹi| b + log b 9: L GAN ← 1 K K i=1 -log D(x, y i ) 10: θ G ← Optimize(θ G , ∇ θ G L GAN + λ MR L MR )

Algorithm 5 Generator update in Gaussian proxy MR 1 -GAN Require: Generator G, discriminator D, pre-trained predictor P , proxy MR loss coefficient λ pMR Require: input x, ground truth y, the number of samples K 1: for i = 1 to K do 2:

z i ← GenerateNoise() 3: ỹi ← G(x, z i ) {Generate K samples} 4: end for 5: μ ← P (x) {Predicted mean} 6: μ ← 1 K K i=1 ỹi {Sample mean} 7: L pMR ← (μ -μ) 2 8: L GAN ← 1 K K i=1 -log D(x, y i ) 9: θ G ← Optimize(θ G , ∇ θ G L GAN + λ pMR L pMR )

Algorithm 6 Generator update in Gaussian proxy MR 2 -GAN Require: Generator G, discriminator D, pre-trained predictor P , proxy MR loss coefficient λ pMR Require: input x, ground truth y, the number of samples K 1: for i = 1 to K do 2:

z i ← GenerateNoise() 3:

ỹi ← G(x, z i ) {Generate K samples} 4: end for 5: μ, σ2 ← P (x) {Predicted mean and variance}

6: μ ← 1 K K i=1 ỹi {Sample mean} 7: σ2 ← 1 K-1 K i=1 (ỹ i -μ) 2 {Sample variance} 8: L pMR ← (μ -μ) 2 + (σ 2 -σ2 ) 2 9: L GAN ← 1 K K i=1 -log D(x, y i ) 10: θ G ← Optimize(θ G , ∇ θ G L GAN + λ pMR L pMR )

Algorithm 7 Generator update in Laplace proxy MR 1 -GAN Require: Generator G, discriminator D, pre-trained predictor P , proxy MR loss coefficient λ pMR Require: input x, ground truth y, the number of samples K 1: for i = 1 to K do 2:

z i ← GenerateNoise() 3:

ỹi ← G(x, z i ) {Generate K samples} 4: end for 5: m ← P (x) {Predicted median} 6: m ← med(ỹ 1:K ) {Sample median} 7:

t 1:K = detach(ỹ 1:K + ( m -m)) 8: L pMR ← 1 K K i=1 (t i -ỹi ) 2 9: L GAN ← 1 K K i=1 -log D(x, y i ) 10: θ G ← Optimize(θ G , ∇ θ G L GAN + λ pMR L pMR )

Algorithm 8 Generator update in Laplace proxy MR 2 -GAN Require: Generator G, discriminator D, pre-trained predictor P , proxy MR loss coefficient λ pMR Require: input x, ground truth y, the number of samples K 1: for i = 1 to K do 2:

z i ← GenerateNoise() 3: ỹi ← G(x, z i ) {Generate K samples} 4: end for 5: m, b ← P (x) {Predicted median and MAD} 6: m ← med(ỹ 1:K ) {Sample median} 7: b ← 1 K K i=1 |ỹ i -m| {Sample MAD} 8: t 1:K = detach(ỹ 1:K + ( m -m)) 9: L pMR ← 1 K K i=1 (t i -ỹi ) 2 + ( b -b) 2 10: L GAN ← 1 K K i=1 -log D(x, y i ) 11: θ G ← Optimize(θ G , ∇ θ G L GAN + λ pMR L pMR )

• Noise input: We concatenate Gaussian noise tensor of size H × W × 32 at each input of the first and second dilated convolution layers. We also inject the noise to the convolution layer before the first dilated convolution layer. Each element in the noise tensors are independently sampled from N (0, 1). • Input resizing and masking: We use square-cropped CelebA images and resize them to 128 × 128. For masking, we randomly generate a hole of size between 48 and 64 and fill it with the average pixel value of the entire training dataset. • Input normalization: We make 128 × 128 × 3 input images' pixel values lie between 0 and 1. We do not further normalize them with their mean and standard deviation. • Batch sizes: We use 16 for the discriminator and the predictor and 8 for the generator.

When training the generator, we generate 12 samples for each input, therefore its total batch size is 96. • Loss weights: For GLCIC, we tested Gaussian proxy MR 2 loss and MR lossess. We successfully trained GLCIC with the Gaussian proxy MR 2 loss using λ pMR = 1000. However, we could not find any working setting for the MR loss. For the baseline model, we use 2 loss for the reconstruction loss and set λ 2 = 100. • Update ratio: We update generator three times per every discriminator update.

C PREVENTIVE EFFECTS ON THE MODE COLLAPSE:

Our MR and proxy MR losses have preventive effect on the mode collapse. Figure 4 shows toy experiments of unconditional generation on synthetic 2D data which is hard to learn with GANs due to the mode collapse. We train a simple 3-layer MLP with different objectives. When trained only with GAN loss, the model captures only one mode as shown in figure 4b . Adding 2 loss cannot fix this issue either as in figure 4c . In contrast, all four of our methods (figure 4g-4f ) prevent the mode collapse and successfully capture all eight modes. Notice that even the simpler variants MR 1 and proxy MR 1 losses effectively keep the model from the mode collapse. Intuitively, if the generated samples are biased toward a single mode, their statistics, e.g. mean or variance, deviate from real statistics. Our methods penalize such deviations, thereby reducing the mode collapse significantly. Although we restrict the scope of this paper to conditional generation tasks, these toy experiments show that our methods have a potential to mitigate the mode collapse and stabilize training even for unconditional generation tasks. 

E MISMATCH BETWEEN 1 LOSS AND GAN LOSS:

In section 4.4, we showed that if there exists some x such that Var(y|x) > 0, there is no generator optimal for both 2 loss and GAN loss. On the other hand, 1 loss has some exceptional cases where the generator can minimize GAN loss and 1 loss at the same time. We identify what the cases are and explain why such cases are rare.

To begin with, 1 loss is decomposed as follows:

L 1 = E x,y,z |y -G(x, z)| = E x,z E y |y -G(x, z)| = E x,z p(y|x)|y -G(x, z)|dy = E x,z G(x,z) -∞ p(y|x)(G(x, z) -y)dy + ∞ G(x,z) p(y|x)(y -G(x, z))dy

To minimize 1 loss, we need the gradient w.r.t. G(x, z) to be zero for all x and z that p(x) > 0 and p(z) > 0. Note that this is a sufficient condition for minimum since the 1 loss is convex.

EQUATION

Therefore, G(x, z) should be the conditional median to minimize the loss. Unlike 2 loss, there can be an interval of G(x, z) that satisfies Eq.( 19). Specifically, any value between interval [a, b] is a conditional median if a -∞ p(y|x)dy = b -∞ p(y|x)dy = 1 2 . If every real data belongs to the interval of conditional median, then the generator can be optimal for both GAN loss and 1 loss.

For instance, assume that there are only two discrete values of y possible for any given x, say -1 and 1, with probability 0.5 for each. Then the interval of median becomes [-1, 1], thus any G(x, z) in the interval [-1, 1] minimizes the 1 loss to 1. If the generated distribution is identical to the real distribution, i.e. generating -1 and 1 with the probability of 0.5, the generator is optimal w.r.t. both GAN loss and 1 loss. However, we note that such cases hardly occur. In order for such cases to happen, for any x, every y with p(y|x) > 0 should be the conditional median, which is unlikely to happen in natural data such as images. Therefore, the set of optimal generators for 1 loss is highly likely to be disjoint with the optimal set for GAN loss.

F EXPERIMENTS ON MORE COMBINATIONS OF LOSS FUNCTIONS:

We present some more experiments on different combinations of loss functions. The following results are obtained in the same manner as section D.2 and section D.3.

Figure 11 shows the results when we train the Pix2Pix model only with our loss term (without the GAN loss). The samples of MR 1 and proxy MR 1 losses are almost similar to those with the reconstruction loss only. Since there is no reason to generate diverse images, the variances of the samples are near zero while the samples are hardly distinguishable from the output of the predictor. On the other hand, MR 2 and proxy MR 2 losses do generate diverse samples, although the variation styles are different from one another. Intriguingly, MR 2 loss incurs high-frequency variation patterns as the sample variance of MR 2 loss is much closer than proxy MR 2 loss to the predicted variance. In the case of proxy MR 2 loss, where the attraction for diversity is relatively mild, the samples show low-frequency variation.

Another experiment that we carry out is about the joint use of all loss terms: GAN loss, our losses and reconstruction loss. Specifically, we use the following objective with varying λ Rec from 0 to Figure 12 shows the results of the experiments. As λ Rec increases, the sample variance reduces. This confirms again that the reconstruction is the major cause of loss of variability. However, we find one interesting phenomenon regarding the quality of the samples. The sample quality deteriorates up to a certain value of λ Rec , but gets back to normal as λ Rec further increases. It implies that either proxy MR loss or the reconstruction loss can find some high-quality local optima, but the joint use of them is not desirable. 

:



Figures and Tables:

Figure 1: The loss mismatch in conditional GANs with examples of a Pix2Pix variant trained on Cityscapes dataset. We compare the results according to different loss functions (columns) with different noise input (rows). (c) Using the GAN loss alone is not enough to learn the complex data distribution due to its training instability. (d) Using the 1 loss only, the model generates images fairly close to the ground-truth, but they are far from being realistic. (e) The combination of the two losses enables the model to generate visually appealing samples. However, the model completely ignores noise input and generates almost identical images. (f) In contrast, the model trained with our loss term (i.e. proxy MR 2 ) generates a diverse set of images fully utilizing noise input.

Figure 2: The architecture comparison of the proposed MR-GAN and proxy MR-GAN with conventional conditional GANs. (a) In conditional GANs, G generates a sample ỹ (e.g. a realistic image) from an input x (e.g. a segmentation map) and D determines whether ỹ is real or fake. The reconstruction loss L Rec enforces a sample ỹ to be similar to a ground-truth image y. (b) In our first model MR-GAN, G generates a set of samples ỹ1:K to calculate the sample moments, i.e. mean and variance. Then, we compute the moment reconstruction loss, which is basically an MLE loss between the sample moments and y. (c)The second model proxy MR-GAN is a more stable variant of MR-GAN. The predictor P , a twin network of G, is trained with an MLE loss to estimate the conditional mean and variances of y given x. Then, we match the sample mean μ to the predicted μ and the sample variance σ2 to the predicted σ2 .

),Zhu et al. (2017),Lee et al. (2018a),Huang et al. (2018), andLee et al.  (2018b), since the latent variables can model meaningful information about the target diversity.

Figure 3: Comparison between the results of our proxy MR 2 -GAN and the state-of-the-art methods on image-to-image translation, super-resolution and image inpainting tasks. In every task, our model generates diverse images of high quality, while existing methods with the reconstruction loss do not.

Figure 4: Experiments on a synthetic 2D dataset. (a) The data distribution. (b)(c) Using the GAN loss alone or with 2 loss results in the mode collapse. (d) Training a predictor with the MLE loss in Eq.(7) produces a Gaussian distribution with the mean and variance close to real distribution. The dots are samples from the Gaussian distribution parameterized by the outputs of the predictor. (e)-(h) Generators trained with our methods successfully capture all eight modes.

Figure 9: GLCIC-CelebA Gaussian proxy MR 2 loss (success cases).

Figure 10: GLCIC-CelebA Gaussian proxy MR 2 loss (failure cases).

Figure 11: Ablation test using our loss terms only. The first row of each image is arranged in the order of input, ground truth, predicted mean, sample mean, predicted variance, and sample variance. The other two rows are generated samples. Without GAN loss, the results of proxy MR 1 loss and MR 1 loss are almost identical to the result of 2 loss (predicted mean), while proxy MR 2 loss and MR 2 loss show variations with different styles.

Figure 12: Joint use of GAN, proxy MR 2 , and reconstruction losses. Each image is presented in the same manner as figure 11. We fix the coefficients of GAN and proxy MR 2 losses and test multiple values for the coefficient of reconstruction loss.

Peer review (iclr2019):
 The paper proposes a modification to the traditional conditional GAN objective (which minimizes GAN loss as well as either L1 or L2 pixel-wise reconstruction losses) in order to promote diverse, multimodal generation of images. The modification involves replacing the L1/L2 reconstruction loss -- which predicts the first moment of a pixel-wise gaussian/laplace (respectively) likelihood model assuming a constant spherical covariance matrix -- with a new objective that matches the first and second moments of a pixel-wise gaussian/laplace likelihood model with diagonal covariance matrix. Two models are proposed for matching the first and second moments - the first one involves using a separate network to predict the moments from data which are then used to match the generator’s empirical estimates of the moments (using K samples of generated images). The second involves directly matching the empirical moment estimates using monte carlo. The paper makes use of a well-established idea - modeling pixel-wise image likelihood with a diagonal covariance matrix i.e. heteroscedastic variance (which, as explained in [1], is a way to learn data-dependent aleatoric uncertainty). Following [1], the usage of first and second moment prediction is also prevalent in recent deep generative models (for example, [2]) i.e. image likelihood models predict the per-pixel mean and variance in the L2 likelihood case, for optimizing Equation 4 from the paper. Recent work has also attempted to go beyond the assumption of a diagonal covariance matrix (for example, in [3] a band-diagonal covariance matrix is estimated). Hence, the only novel idea in the paper seems to be the method for matching the empirical estimates of the first and second moments over K samples. The motivation for doing this makes intuitive sense since diversity in generation is desired, which is also demonstrated in the results. Section specific comments: - The loss of modality of reconstruction loss (section 3.2) seems like something which doesn’t require the extent of mathematical and empirical detail presented in the paper. Several of the cited works already mention the pitfalls of using reconstruction loss. - The analyses in section 4.4 are sound in derivation but not so much in the conclusions drawn. It is not clear that the lack of existence of a generator that is an optimal solution to the GAN and L2 loss (individually) implies that any learnt generator using GAN + L2 loss is suboptimal. More explanation on this part would help. The paper is well written, presents a simple idea, complete with experiments for comparing diversity with competing methods. Some theoretical analyses do no directly support the proposition - e.g. sections 3.2 and 4.4 in my specific comments above. Hence, the claim that the proposed method prevents mode collapse (training stability) and gives diverse multi-modal predictions is supported by experiments and intuition for the method, but not so much theoretically. However, the major weakness of the paper is the lack of novelty of the core idea. === Update after rebuttal: Having read through the other reviews and the author*s rebuttal, I am unsatisfied with the rebuttal and I do not recommend accepting the paper. My rating has decreased accordingly. The reasons for my recommendation, after discussion with other reviews, are -- (1) lack of novelty and (2) weak theoretical results (some justification of which was stated in my initial review above). Elaborating more on the second point, I would like to mention some points which came up during the discussion with other reviewers: The theoretical result which states that not using reconstruction loss given that multi-modal outputs are desired is a weaker result than proving that the proposed method is actually effective in what it is designed to do. There are empirical results to back that claim, but I strongly believe that the theoretical results fall short and feel out of place in the overall justification for the proposed method. This, along with my earlier point of lack of novelty are the basis for my decision. References: [1] Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?." Advances in neural information processing systems. 2017. [2] Bloesch, M., Czarnowski, J., Clark, R., Leutenegger, S., & Davison, A. J. (2018). CodeSLAM-Learning a Compact, Optimisable Representation for Dense Visual SLAM. CVPR 2018. [3] Dorta, G., Vicente, S., Agapito, L., Campbell, N. D., & Simpson, I. (2018, February). Structured Uncertainty Prediction Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.