Title: DISTRIBUTIONAL CONCAVITY REGULARIZATION FOR

Abstract: We propose Distributional Concavity (DC) regularization for Generative Adversarial Networks (GANs), a functional gradient-based method that promotes the entropy of the generator distribution and works against mode collapse. Our DC regularization is an easy-to-implement method that can be used in combination with the current state of the art methods like Spectral Normalization and Wasserstein GAN with gradient penalty to further improve the performance. We will not only show that our DC regularization can achieve highly competitive results on ILSVRC2012 and CIFAR datasets in terms of Inception score and Fréchet inception distance, but also provide a mathematical guarantee that our method can always increase the entropy of the generator distribution. We will also show an intimate theoretical connection between our method and the theory of optimal transport.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) is a model consisting of two adversarial neural networks designed for the training of a generator distribution that mimics a target distribution defined on a (often) high dimensional space, and it has been successful in numerous applications including image and movie generations (Isola et al., 2017; Zhu et al., 2017; Saito et al., 2017) . However, it has not yet completely established itself as a wildly scientific tool because of the sheer computational difficulty of its training process. As such, there has been numerous studies seeking for a way to stabilize the training process of GANs (Gulrajani et al., 2017; Miyato et al., 2018; Karras et al., 2018) .

Mode collapse is a persistent central problem for the training of GANs, which collectively refers to the lack of diversity in generator distribution. For instance, without any countermeasure, GANs applied to multimodal mixture of Gaussians will often train a generator distribution with one mode (See Fig 22, Goodfellow (2016) for example). Mode collapse has been discussed in numerous literatures related GANs. To name a few, see Goodfellow (2016) ; Metz et al. (2016) ; Arjovsky & Bottou (2017) ; Arjovsky et al. (2017) ; Lin et al. (2017) .

In words of statistical machine learning, mode collapse can be described as a case of entropy degeneration. A naive countermeasure against mode collapse is therefore to augment the entropy of the generator distribution. Not many studies to date, however, tackled the problem of mode collapse using this direct approach. Dai et al. (2017) partially realized this idea by actually evaluating the entropy of the generator distribution with variational inference and including it into the objective function. This type of strategy requires the user to continuously produce reliable estimates of the generators based on finite samples throughout the course of the training. As such, not much further improvement can be expected from this strategy, because precise estimation of the entropy tends to be computationally heavy, and empirical estimation is an excruciatingly difficult problem on its own in high dimension, as is the case in image and movie generation. In fact, the performance of classical methods based on kernel density estimation, for example, does not scale well with respect to dimension, and the number of samples required to control the MSE can grow exponentially with the dimension (Cacoullos, 1966; Ozakin & Gray, 2009) .

In this study, we use the theory of functional gradient to develop a method that can promote the entropy of the generator distribution without directly estimating the entropy itself. All in all, the ob-jective of GANs is find a good generator. The motive of functional gradient method is pay attention to the update of the generator in the space of generator functions itself, instead of the updates in the parameter space. In the function space, the functional properties of the generators are generally easier expressed than in the parameter space. As such, the search for a generator with good functional properties may be much easier in the space of functions, as opposed to the space of parameters. Throughout, this philosophy will serve as the basis of our method.

In more general technical term, the functional gradient of an objective function with respect to a model function is an infinite dimensional gradient computed over the infinite dimensional space of all models. In our case, our interest is the functional gradient of the GANs' objective function with respect to the generator function. As we will show, all variations of GANs to date are implicitly using the discriminator function to compute the functional derivative of the objective function with respect to the generator distribution function. In other words, the discriminator determines the direction to which the algorithm should move the mass of the current generator distribution; the discriminator determines what the next (target) distribution looks like. The work of Nitanda & Suzuki (2018) is a pioneer study that explicitly incorporated this idea into the training of neural generator models.

Their study showed that one can carry out a faithful functional gradient-based update by inserting what they call gradient layer into the layers of neural generator function. Johnson & Zhang (2018) further polished this strategy by periodically distilling the networks.

A more precisely worded advantage of the functional gradient method is that, at every stage in the training process of the generator distribution, it allows the user to monitor what the next distribution (target distribution) looks like in terms of the current distribution. The ability of the functional gradient update to tell something about the next (target) distribution is a significant advantage of the functional gradient method over the conventional parametric methods, because the target distribution set forth by conventional parametric methods is usually expressed in a complicated parametric form (e.g. Deep Neural Nets (DNNs)), and its behaviors are often difficult to predict. This advantage of the functional gradient based-update suggests the possibility that we can control the update rule in order to deliberately direct the generator distribution to a distribution with preferred propertieswhich, in our case, include high entropy.

We discovered that, by locally concavifying the discriminator function, we can manipulate the functional gradient so that the next update for the generator will always target a distribution with higher entropy. From now on, we should refer to our method by Distributional Concavity (DC) regularization. Our method does not require the direct estimation of the entropy. We will not only show that our DC regularization can help improve the performance of GANs in terms of Inception score (Salimans et al., 2016) and Fréchet inception distance (FID) (Heusel et al., 2017) , but also give a mathematical guarantee that our regularization will always increase the entropy of the generator distribution. We will also show that our method greatly outperforms the method of Dai et al. (2017) that directly estimates the entropy based on classically techniques.

The regularization strategy of monotonically increasing the entropy of the generator distribution over the course of its training is not without theoretical basis. Our DC regularization has close relations to the theory of optimal transport. We will show that, when the entropy of the true distribution is higher than the current generator distribution, the functional gradient that is properly derived from the optimal mass transport always increases the entropy of the distribution. Moreover, the functional update used in our method is always a monotonic mapping, which also turns out to be one of the properties satisfied by the update with optimal transport. This is a preferred property as well, because it tends to promote the smooth training of the generator. We summarize our contributions below:

• We propose an update for the generator of GANs that promotes the entropy of the generator without the need for an explicit estimation of the actual entropy. • We show that, when the entropy of the true distribution is larger than that of the current generator distribution, the functional gradient derived from the 2-Wasserstein (W 2 ) optimal transport always increases the entropy of the distribution. • We provide a mathematical guarantee that our method increases the entropy of the generator distribution at every step. • We show that our method improves the results of GANs in terms of Inception score and FID score.

2. THEORY:



2.1. GENERATIVE ADVERSARIAL NETWORKS:

Let us first review the formulation of the original GANs. Unless otherwise noted, let us use boldface capital letters to refer to a random variable, and lower case letter for its realization. The training process of GANs (Goodfellow et al., 2014) is a two player min-max game in which the generator distribution µ θ is trained to minimize the divergence between the true distribution ν and the generator distribution µ θ measured by the current critic F , while the critic F is trained to most strongly discriminate µ θ from ν. Often, µ θ is produced by applying a paramatric generator function G θ to a random seed variable with some distribution µ z , and it satisfies

EQUATION

for all measurable statistics h. In mathematical language, µ θ is a pushforward of µ z , and is often written as G θ #µ z . In a nutshell, GANs aim to find an optimal parametric distribution µ θ that achieves the minimum value for

EQUATION

where L r and L g are the functions of user's choice. We can retrieve the formulation of the original GANs (Goodfellow et al., 2014) by letting L r (F (x)) = softplus(-F (x)) and L g (F (x)) = softplus(F (x)), where softplus(•) = log(1 + exp(•)) There is a variety of choices for (L r , L g ) (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016) . Rewritten as the optimization problem about the function G, our objective function is given by

EQUATION

2.2. FUNCTIONAL GRADIENT INTERPRETATION OF THE GANS UPDATE:

Let us elaborate further on the mechanism of the functional-gradient based update and its connection to the conventional update of GANs . For ease of notation, let us write L = L g • F in the equation (3), where the operator • designates the function-composition. Let us remind ourselves that the objective of the min part of the min-max game in each step is to find a good update of µ θ that decreases E µ θ [L(X)]. It is no exaggeration to say that the choice of the next target distribution of µ θ -or the distribution to which the µ θ will be updated-will completely determine the training efficiency of GANs. The most canonical and obvious approach is to use the information of the discriminator L in making this decision. Note that, by appealing to a standard argument based on Taylor expansion, L(x) ≥ L(x-α∇L(x)) for any value of x when α is suffiently small. Therefore, leaving the mathematical technicalities aside, one may expect that a carefully chosen α can ensure

EQUATION

One naive suggestion based on this intuition is the update of µ θ to a distribution that can be constructed by taking a sample from µ θ (say, x) and transporting it into the direction of -α∇L(x).

Using the pushforward notation, this amounts to the update from µ θ to (Id -α∇)#µ θ . The presumed relation ( 4) is equivalent to

E µz [L(G θ (X) -α∇L(G θ (X)))] ≤ E µz [L(G θ (X))],

and the corresponding suggested update in terms of the function G θ is an update from G θ to (Id -α∇L) • G θ . That is, our suggestion amounts to the update from

EQUATION

This is an update from G θ #µ z to (T α • G θ )#µ z . This update of G is called functional gradient update, and T α is called a transport function. With enough (reasonable) regularity assumption of the function space and probability space, one can justify the argument we have made above using a differential calculus in an infinite dimensional space of functions(See appendix A). As a spoiler for what we will further elaborate later, our method regularizes L so that the target distribution

(T α • G θ )#µ z has higher entropy than µ θ = G θ #µ z .

We are, however, still not done in our description of a functional gradient perspective of the GANs update. In what we formulated above, there is no guarantee that the target distribution (T α •G θ )#µ z admits the same parametric representation as G θ ; if (T α • G θ )#µ z cannot be expressed by the DNN that we have prepared for the training, all suggestions we have made above is for naught. The final remaining task in this update procedure is therefore to find the parameter θ * such that G θ * #µ z can best approximate the target distribution. Letting θ old to denote the current choice of θ for the generator function, this can be done by solving the following optimization problem about θ :

EQUATION

)

The gradient of this sub-objective function ( 6) with respect to θ evaluated at θ = θ old is given by

EQUATION

where ∂ θ designates the derivative operator with respect to θ. This formulation is practically equivalent to the one introduced in xICFG (Johnson & Zhang, 2018) . This turns out to be the familiar gradient update used in the usual GAN implementation. Indeed, in the usual implementation, the parameter for the generator is updated with the rule:

EQUATION

In general, almost all variations of GANs to date (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016) uses this type of update rule for the training of the generator. In other words, all methods to date has been implicitly doing the functional-gradient type update all along, and the choice of L has been determining the choice of the target distribution. As hinted a moment ago, this suggests that, by directly regularizing the L in the usual update scheme of GANs, one can realize a functional gradient update of the generator with a controlled target distribution. This is the very gist of our algorithm.

2.3. CHOICE OF THE TARGET DISTRIBUTION:

As inferred above, from the perspective of functional gradient, the min-max game of GANs can be decomposed into steps: (i) the target construction step that constructs the next target distribution using the functional gradient, and (ii) the distillation step that looks for a neural function that better approximates the target distribution. Now, the next natural pressing question is: "what type of L should we use to make sure that the next target distribution is nice?" As inferred in the introduction, the goal of this particular study is to create a sequence of updates that is less likely to suffer from mode collapse. Let us recall that, as long as we follow the standard update procedure of GANs, the choice of L will entirely determine the property of the next target distribution. A proposal we would like to make in this study is to simply choose the discriminator L from the set functions that are concave on the support of the current distribution; Proposition 2.1.

Let µ be a probability distribution on R d . If L is concave on the support of µ, then H(T α #µ) ≥ H(µ).

Note that this statement is independent of the step size α, because any positive scalar multiple of concave function is concave. This result ensures that any update with a concave L will always increase the entropy; that is, the target distribution will be more dispersed than the current distribution, and the mode collapse is less likely to happen.

Additionally, our choice of L guarantees that the transport function used to created the target distribution satisfies another preferable property called monotonicity. Monotonicity is a property that requires that there is no crossings in the transport:

T (x) -T (x ), x -x R d ≥ 0.

This is a preferred property because, as we will empirically show (see section 4.1) , the crossings in the transport tend to hinder the smooth training process. This is in fact somewhat intuitive, because crossings lead to wasteful transportation of the mass. In fact, this is a property that is achieved by the distributional update based on Optimal transport as well (Villani, 2008) . Indeed, by the definition, the concavity of L implies the strong convexity of x 2 2 -αL(x), which in turn implies

EQUATION

which is a stronger condition than the monotonicity. Thus, by simply making L concave, we can not only construct a target distribution whose entropy is greater than that of the current generator distribution, but also assure that the corresponding transport is monotonic. In a way, this is a statement about how complicated the function is, because the presence of the crossings imply the existence of a discontinuity or a region of one to many mappings. This is therefore a property that is likely to affect the distillation step. In fact, we will empirically show that the property of monotonicity affects the distillation step in positive way.

Indeed, we do not intend say that our suggestion for the property of L is optimal-the user has the freedom to choose the set of properties to be required for L so that the corresponding target distribution will have the desired properties that serves the purpose of the user.

2.4. RELATION TO OPTIMAL TRANSPORT:

Our strategy for the distributional update that we have discussed so far has close relations to the optimal transport. In fact, an equation of the form ( 5) is ubiquitous in the theory of optimal transport. In this section, we will show the following:

1. By choosing L to be concave, the transport T α in the equation ( 5) becomes the optimal transport from the current distribution µ to the target distribution T α #µ. 2. If the entropy of the true distribution ν is greater than that of the current distribution µ, any sequence of the updates of the distribution along the optimal transport from µ to ν increases the entropy monotonically. We can also ensure the monotonic increase in the entropy by using T α with concave L. 3. By choosing L to be concave, we can assure that the target distribution constructed from T α will always be within α distance of the current distribution in Wasserstein (W 2) sense.

In order to elaborate on these points, we would like to briefly introduce the notion of optimal transport. For a more rigorous introduction of the concept, please consult Villani (2008) . For p ≥ 1, the p-Wasserstein distance W p (µ, ν) between two arbitrary distributions µ and ν with sufficient regularity is given by inf

EQUATION

)

where the inf above is taken over all measurable maps T satisfying T #µ = ν. When p = 2, it is known that the infimum of this cost function is achieved by T * that can be expressed as x -∇L * for some L * that renders x 2 /2 -L * convex (Brenier, 1991) . The search for L * and hence T * therefore amounts to the search for the closest coupling between µ and ν in the L 2 sense.

Another surprising fact is that the movement of the particle along the direction of ∇L * monotonically decreases W 2 (Villani, 2008) . That is, if Villani, 2003) . Please compare this transport equation with the equation ( 5). This T * t is indeed the optimal transportation analogue of the functional gradient. The theorem in Brenier (1991) has a still surprising converse; any function T that can be written as a gradient of a strictly convex function turns out to be the unique optimal transport from µ to T #µ. Using this fact, we can appeal to the proposition 2.2 below to claim that the functional update based on the optimal transport satisfies the following important property that shall be intuitively fulfilled if we are in fact moving the distribution toward the true distribution: if the current distribution has lower entropy than the true distribution, the sequence of the distributions produced by the optimal transport based updates should be monotonically increasing in the entropy: Proposition 2.2. Suppose ν = T #µ for some T that can be written as a gradient of a strictly convex function.

T * t (x) = x -t∇L * (x), then W 2 (T * t #µ, ν) mono- tonically decreases with t (

If H(ν) ≥ H(µ), then H(T t #µ) is monotonically increasing on t ∈ [0, 1].

This proposition actually assures that the updates of DC regularization also satisfy the same property. Because we are designing the target distribution so that it will have higher entropy than the current distribution at every step, by letting ν in the proposition 2.2 to be the target distribution, we have a guarantee that the sequence of distributions constructed by our updates is also monotonically increasing in the entropy. To see this, simply note that we can automatically make x 2 /2 -L to be convex by making L concave, The connection between our DC regularization and optimal transport is not limited to the property we introduced above. By the theory of Brenir, the choice T α = Id -α∇L with concave and 1

-Lipschitz L satisfies W 2 (µ, T α #µ) = E[ α∇L(X) 2 2 ] = α.

Put in still other words, by constructing a target distribution with concave L, we can also assure that the target distribution is contained within the W 2 neighborhood of the current distribution function. This is a property that cannot be guaranteed with the conventional parameter-based updates. Monotonicity condition is also a property that is satisfied by the optimal transport. According the theory of Monge-Ampere (Villani (2008) ), a transport map can be optimal only if it is monotonic. Thus, while not being exactly based on the optimal transport from the generator distribution to the true distribution, our T shares many favorable properties in common with the optimal transport, and is very closely related to the theory of W 2 distance.

Reflecting on these facts, one might become tempted to say that we shall simply formulate GANs with the objective function that is solely based on W 2 distance between the generator distribution and the true distribution. However, as we will further articulate in the discussion section, the challenge remains to conduct faithful W 2 -based updates in the implementation of GANs.

3. METHOD:

If µ θ := G θ #µ z is our current parametric generator distribution and ν is the true distribution, our DC regularization method first (i) proposes a target distribution T #µ θ using a concave L, and then (ii) seek a measure µ θnew that well approximates the target distribution T #µ θ . We use the following sampling-based penalty term in order to promote the concavity of L (= L g • F ):

EQUATION

where x 1 , x 2 are samples from the support of µ θ , is a sample from the uniform distribution over [0, 1], and d is a positive scalar. Note that this term must be positive if L is concave over the support of µ θ . Our algorithm is summarized in Algorithm 1. The update rule in this algorithm uses the target distribution constructed by T (x) = x -∇L with concave L. Intuitively speaking, the transport T has an effect of moving µ θ toward T #µ θ while dispersing the mass of µ θ . See Fig 1 for a visual rendering of this interpretation.

In most practical application, the training of GANs begins by dispersing the initial distribution with small entropy and gradually molds the mass into what resembles the true distribution. Our transport ensures that this dispersion is consistently happening over the course of the training. As we will show in the result section, this regularization works in favor of the inception score without any noticeable downfall from over-dispersion, suggesting that the generator distribution created with the current state of the art techniques are still not dispersed enough.

Algorithm 1 GANs algorithm with DC regularization for each iteration do θ old ← θ the target construction step : find F * that optimizes

EQUATION

and construct T α from L := L g • F as in the equation ( 5). the distillation step : update generator by generator's objective

min θ E µz (T α • G θ old )(Z) -G θ (Z) 2 2 (15)

end for

4. EXPERIMENTAL RESULTS:

We applied DC regularization to the training of GANs on CIFAR-10 (Torralba et al., 2008) , CIFAR-100 (Torralba et al., 2008) and ILSVRC2012 dataset (ImageNet) (Russakovsky et al., 2015) in various settings and evaluated its performance in terms of Inception score (Salimans et al., 2016) and Fréchet inception distance (FID) (Heusel et al., 2017) . Inception score and FID are performance measures that are commonly used to evaluate the severity of mode collapse. For the details of the evaluation, see Appendix C.1. We also conducted an additional set of experiments with artifical The graph of -L and its gradient vector field. Each point x will be transported by T along the direction of the vector field -∇L. Over the set on which L is not concave, T may move the points in the region come closer to each other. The use of concavified L for the construction of transport function will make the points move away from each other.

dataset to investigate the properties of DC-regularization. We provide the results for additional experiments in the Appendix D as well.

4.1. INTRINSIC PROPERTIES OF DC REGULARIZATION:



Effect of DC regularization on entropy:

Using a simple Gaussian Mixture Model with five modes as the true distribution, we evaluated the sheer ability of the DC regularization to promote the entropy of the generator distribution. We used hinge loss for the objective function, and used DNNs to model both the discriminator and the generator. We trained the model with and without the DC regularization and reported the entropy of the Generator at different stages of the training by explicitly computing the determinant of the Jacobian at each layer. As one of the baseline, we trained EGAN-Ent-VI (Dai et al., 2017) , which includes into its objective function a penalty against the negative entropy of the generator. The result is illustrated in Fig 2 (a) . We see that the regularization is positively affecting the entropy at all stages of the training. Although to a less extent, we can confirm that our implementation of EGAN-Ent-VI is also preventing the degeneration of the entropy. Indeed, the persistent pressure to increase the entropy can result in over-dispsersed final product. However, this seems not to be a serious issue when it comes to the learning on big data like ImageNet. In terms of Inception score and FID score, all artificial generator distributions today are still far less diverse than the original dataset (Table 1 ), and there is still a large room left for the improvement of diversity.

Effect of monotonicity in distillation step:

Recall that each round of GANs update consists of the target construction step and the distillation step. We conducted an experiment to verify the effect of the monotonicty on the distillation step. We will show a case in which, even if the target distribution constructed from a non-monotonic map is further away from the current distribution than the target distribution constructed from a monotonic map, the projection of the latter distribution onto the parametric function-space is much easier. Let K be a positive value. Starting from an initial distribution X = G θ0 (Z) , consider the following pair of maps to be applied to X:

EQUATION

It is evident that T (2) is a monotonic mapping and T (1) is not. Denote the law of X by µ.

We used T (1) #µ and T (2) #µ as target distributions, and trained the parameterθ using

O m = E Z [ T (m) (G θ0 (Z)) -G θ (Z) 2 ]

as the objective function (Distillation). Note that this objective function takes the same value of K 2 for both m = 1, 2. Also, by the definition of the Wasserstein distance,

K = W 2 (T (2) #µ, µ) while W 2 (T (1) #µ, µ) 2 = inf T #µ=T1#µ E µ [ T (X) -X 2 ] ≤ E µ [ T (1) (X) -X 2 ] = K 2 .

A naive intuition dictates that the distillation of T (1) #µ is easier, because T (1) #µ is distributionally closer to µ while the objective function evaluated at T (1) #µ is same as the evaluation at T (2) #µ. However, as we can see in The left graph plots the transition of the entropy of the generator distribution of GANs trained for the mixture of Gaussians. The marker "k th iteration starts" designates the result for which our regularization was applied after kth step. We see that our method has the effect of preventing the entropy degeneration. The right graph plots the transitions of the energy for the distillation step when the target distributions were constructed by applying monotonic map (red) and non-monotonic map(blue) to a gaussian distribution. Notice that the distillation step converges faster when the target distribution is constructed from monotonic map.

4.2. RESULTS ON CIFAR-10 AND CIFAR-100:

Experiments with different architectures and objective functions We tested our algorithm for the training of GANs with six types of objective functions and two network architectures, and reported the performance on all 12 combinations. For the details of the objective functions and the architectures, please see Appendix C.2, C.3, C.4 For the training of the network, we applied Spectral Normalization (SN) (Miyato et al., 2018) to the full-connected layer and the convolution layer. 

Experiments with different prior dimensions:

In general, without careful selection of the dimension of the prior distribution, the training of GANs tends to suffer a serious case of mode collapse. For image generation task, this will result in low inception score. We therefore applied DC regularization to the trainings of GANs on CIFAR-10 with very low prior dimensions as well as very large prior dimensions and evaluated the performance. For this experiment, we used GAN-variant2 (Appendix C.2) for the objective function and used SNDCGAN for the architecture. As for the experimental details, please see Appendix C.4. The results are summarized in Fig 4 and Fig 10 (in Appendix) . When the prior dimension is below the inherent dimension of the dataset, the dimension of the generator distribution cannot match the dimension of the true distribution. Thus, the inception score of the generator trained with low dimensional prior is bound to be low, irrespective of the application of DC regularization. However, for dim(z) ≥ 5, the inception score is as high as 7.5, and for all choices of dim(z), the generator trained with DC regularization consistently outperformed the generator trained without the DC regularization. Similar argument applies to the performance of DC regularization for high dimensional prior. Overall, the DC regularization provides some robustness against the choice of the prior dimension. The performance of the DCGAN in terms of the inception score for CIFAR10 plotted against the dimension of µ z (standard Gaussian). The baseline is the DCGAN with spectral normalization(SN). We can see that too low a dimension and too high a dimension both negatively affect the performance. Notice that the DCGAN with DC regularization outperforms the baseline DCGAN for all extreme dimensions.

Comparison with EGAN and other methods:

We compared the performance of our algorithm against EGAN-Ent-VI (Dai et al., 2017) , another framework that can be used to control the entropy of the generator. We conducted this comparative study using SNDCGAN (Table 2 ) and SNResNet (Table 3 ), which uses Spectral Normalization that is known to have an effect of preventing the degeneration of the feature space. SNDCGAN is a method that is known to perform well on Cifar10 on its own. For the experimental setting of this study, please see the Appendix C.4. As we show in Table 1 , DC regularization outperformed EGAN-Ent-VI on both models. We reported the result of EGAN-Ent-VI with spectral normalization, because it worked better than the version without SN. As we can see in the table 1, the EGAN with SN performed worse than the vanilla SNDCGAN. For high dimensional dataset like Cifar10, the variational inference for the negative entropy can be extremely difficult. It is possible that a poor variational inference in high dimensional space backfired for EGAN-VI's performance on Cifar10. We would also like to emphasize here that EGAN-Ent-VI requires a separate decoder in addition to the generator and the discriminator, and that our algorithm is easier to implement. For the full version of the Table and the visuals of the generated samples, please see Table 8 and Table 11 in the Appendix. We compared our algorithm against other competitive methods as well. The best performance of our method is almost on par with the state-of-the-art method (Karras et al., 2018) . We are losing to progressive GAN (Karras et al., 2018 ) by a slight margin; we, however, would like to make a disclaimer that we are using much smaller architecture than Karras et al. (2018) for the performance evaluation of our method. We also confirmed that we can improve the result of Miyato et al. (2018) by using DC regularization together with SN. The results support that our method is helping the training process suppress mode collapse and is improving the overall performance. For the results of the experiments conducted with Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al., 2017) , please see the Fig 9 in Appendix.

4.3. RESULTS ON IMAGENET:

Additionaly, to evaluate our method's effectiveness on a higher dimensional dataset, we applied our method on ImageNet with 1000 classes, with each class containing approximately 1300 images. We compressed the images to 64 × 64 pixels prior to the experiments. Table 1 : Inception scores and FIDs for unsupervised image generation on CIFAR-10 and CIFAR-100. The CIFAR-10 results for the models designated with † are cited from (Miyato et al., 2018) , and the CIFAR-10 results with ‡ are cited from (Karras et al., 2018) . 

5. DISCUSSION:



Dilemma of GANs update:

As we have shown above, the experimental results support our claim that the DC regularization promotes the entropy of the generator distribution and stabilizes the training process of GANs. As mentioned briefly in the theory section 2, however, we do not have the guarantee that our update rule consistently reduces the W 2 distance between the generator distribution and the true distribution. This fault, however, is common to almost all GANs algorithms today. As we have shown, the conventional update rule is implicitly targetting a distribution of the form T #µ where T = Id -α∇L for some α and a discriminator L. As it turns out, optimal transport takes this form only when we are optimizing W 2 distance, while the common constraint of |∇L| = 1 is a condition required for dual potential when p = 1 (W 1 ) (Villani, 2008) . In other words, the conventional GANs are making the discriminator with W 1 criteria and updating the generator with W 2 criteria. If we are to faithfully create the discriminator with W 2 criteria, we must look for a Legendre-pair of dual potential functions. On the contrary, if we are to update the generator with W 1 criteria, one must look for a closed form solution for the W 1 transport. This, however, is in general a highly complex mathematical problem for which there is a separate field of study (Santambrogio, 2015) . To the authors' best knowledge, no studies have provided a solid solution to this dilemma.

Convexity vs Strong convexity:

We would like to also mention that our regularization is asking for more than what is required by W 2 theory. As mentioned above, in order for T to be the optimal transport from µ to T #µ , T only needs to be the gradient of a convex function. On the other hand, by asking L to be concave, we are in fact asking for T to be the gradient of a strongly convex function. This "overdo" is actually intentional. Recall that the parameter α in the transport T α = Id -α∇L corresponds to a step size in the update of the generator. If we train L that only guarantees the convexity of x 2 /2 -L(x), the functional update derived from such L can be non-monotonic when the step size is large, because such L only guarantees the convexity of x 2 /2 -αL(x) when α ≤ 1. Should we require L to be concave, however, x 2 /2 -αL(x) is concave for any positive α. In this context, we may therefore say that the DC regularization leaves much room to be playful about the learning schedule. Finally, as can be inferred from our proofs for the proposition 2.1 and 2.2, the strong convexity of T has an effect of diffusing the mass of pdf in every direction. We can therefore expect the DC regularization to disperse the collapsed masses in the case of mode collapse. In the light of the fact that the training of GANs usually begins with dense distribution with small support, this dispersion effect should be helpful at the early stage of the training as well.

Appendix (Supplemental Material for Distributional Concavity Regularization for GANs):

A MORE MATHEMATICAL RENDITION OF SECTION 2

In this section, we re-explain the section 2 with more mathematical details. For ease of argument, we will assume the followings from here onward. First, as is the case in many application of GANs, we will assume that ν is a probability measure on the Euclidean space R d , and µ z is a probability measure on some Euclidean space U . Let us also assume that G θ : U → R d is a parametric function of θ that is almost everywhere differentiable with respect to both input and θ. We will first consider the optimization problem of the equation ( 3) with gradient descent about θ. For ease of notation, let us write L = L g • F , where the operator • designates the function-composition. We will assume that L is a continuously differrentiable and uniformly bounded function on R d , and that the Euclidean norms of its gradient and operator norm of Hessian are also uniformly bounded. This type of regularity assumptions on the objective functions are used in other literatures as well (Nitanda & Suzuki (2018) , Johnson & Zhang (2018) ), and it is not too all unrealistic because the support of both target distributions and generator distribution are often compact throughout the course of the training in the real applications. We will also assume that the derivatives of all functions to appear are well defined and bounded on R d . The update rule of θ is given by:

EQUATION

) where ∂ θ designates the derivative operator with respect to θ. In general, almost all variations of GANs to date (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016) uses this type of update rule for the training of the generator. Let us denote the law of G θ (Z) (or X) by µ θ . Let L 2 (µ θ ) denote the Hilbert space of L 2 integrable maps from R d to itself, granted with the inner product a, b

µ θ = E µ θ [a(X) T b(X)].

We will show that we can re-derive the equation ( 18) using the theory of functional gradient. To begin with, instead of updating the parameter θ, we would like to consider directly updating the distribution µ θ by applying a T ∈ L 2 (µ θ ) to a random variable generated from µ θ . This will result in a new distribution T #µ θ , which is a unique distribution that satisfies E µ θ [h(T (X)] = E T #µ θ [h(X)] for all measurable h. Now, in order to create an update rule for T , let us consider the Gâteaux derivative of M (T ) := E T #µ θ [L(X)] with respect to T into an arbitrary direction δ ∈ L 2 (µ θ ). In terse term, we would like to consider the effect of purturbing T into the direction of δ. By the compactnesss assumption on Ω together with the regularity assumptions for the functions defined hereof, we can appeal to the dominated convergence theorem and see that

EQUATION

is uniform for all δ ∈ L 2 (µ). This way, the term ∇L(T (•)) in the expression above is a Fréchet derivative, or the type of derivative to which the usual set of derivative rules can be applied. We will refer to this derivative as functional derivative for short. Note that the directional functional derivative E µ θ [(∇L(T (X)) • δ)(•)] will always take a positive value when δ ∝ ∇L(T (•)). Thus, for α small enough, an update from the current choice of T to T -α∇L ∈ L 2 (µ θ ) will decrease the objective function. Here, we are interested in the derivative computed at T = Id, so the transformation we would like to apply to µ θ is T α (x) = x -α∇L(x).

(20) This is indeed a transportation of a mass into the direction of α∇L. Now, for the update of θ, we would like to design θ new such that µ θnew is closer to the target distribution T #µ θ old than µ θ old . That is, we would like to choose θ that minimizes min

EQUATION

)

The gradient of this sub-objective function with respect to θ is given by

2E µz ∂ θ G θ (Z) (T α • G θ old )(Z) -G θ (Z) θ=θ old (22) = -2αE µz [∂ θ G θ (Z)| θ=θ old ∇L(G θ old (Z))] .

(23) Evaluating this at θ = θ old , we recover the gradient used in the equation ( 18). This formulation is practically equivalent to the one introduced in xICFG (Johnson & Zhang, 2018) .

B THE PROOFS FOR THE PROPOSITIONS 2.1 AND 2.2

We will first prove the proposition 2.2. Let us begin with a useful lemma. We will assume that all regularity assumptions made in Section 2 holds for all variables and functions that appear in this section. Also, we will use the Df to denote the derivative of f . Also assume that, unless otherwise noted, Df (x) indicates D x f (x), the derivative of f with respect to x. Lemma B.1. Let µ and ν be probability distributions on R n , and suppose that we can write (Id -∇L)#µ = ν with one-to-one ∇L. Let us also write T = (Id -∇L), and let T t = Id -t∇L be its time-linear interpolation. Then, for all t ∈ [0, 1],

EQUATION

Proof. Let p µ be the pdf of µ. If y = T (x), by the one-to-one assumption det(DT (x)) > 0 and we may let dy = det(DT (x))dx. By appealing to the fact that

EQUATION

and we can deduce the identity p T #µ (T (x))det(DT (x)) = p µ (x). Building on this fact, with straightforward computation we can say

EQUATION

Assuming that the distributions are regular enough that we can swap the integral, we can appeal to Jacobi's formula and deduce

EQUATION

Now, let us prove the proposition 2.2. Without loss of generality, choose L so that DT = x 2 2 -L. Assuming that D 2 L is diagonalizable and that its spectrum is uniformly bounded away from 1, let spec(D 2 L) = {λ k } and write diag{λ k } k = Λ. Because x 2 2 -L is assumed to be strictly convex, D 2 x 2 2 -L = I -D 2 L is positive definite, and 1 -λ k (x) > 0 for all k. Appealing to the line equation ( 29) we see that the assumption H(ν) > H(µ) is equivalent to

E µ [log det(T (X))] = E µ log det(I -D 2 L(X)) = E µ k log(1 -λ k (X)) . > 0 (34)

Let us write max k {1 -tλ k (x)} = C t (x) > 0. Using the assumption that the support of µ is compact, we can also say 0 < max supp(µ) {C t (x)} := c t < ∞. In general, if A is positive definite and diagonalizable, with straightforward argument we can say log(detA) ≤ tr(A -I)

Using this fact, we see that

EQUATION

Suppose that D 2 L is diagonalizable with the change of basis matrix P . Then -E µ log tr (I -tD 2 L(X)) -1 D 2 L(X) = -E µ tr (P -1 (I -tΛ)P ) -1 P -1 Λ(X)P (39)

= -E µ tr P -1 (I -tΛ) -1 P P -1 Λ(X)P (40)

= -E µ tr (I -tΛ(X)) -1 Λ(X) (41) = -E µ k λ k (X) 1 -tλ k (X) (42) ≥ -E µ k λ k (X) c t (43) > 0 (44)

This concludes the proof of the proposition 2.2. The proposition 2.1 is much simpler to prove. In order to assure the H(ν) ≥ H(µ), we only need to guarantee that E µ [log det(DT (X))] ≥ 0. By requiring L to be concave, we would make x 2 /2 -L(x) to be convex so that the eigenvalues of DT will all become positive. The result then follows trivially from the argument similar to the one that leads to equation (34).

C EXPERIMENTAL SETTINGS C.1 PERFORMANCE MEASURES:

For the measure of GANs' performance on the image dataset, we used Inception score (Salimans et al., 2016) . Inception score was introduced originally as an exponentiated divergence measure based on the trained Inception convolutional neural network (Szegedy et al., 2015) , which is often called Inception Model. The dominating consensus among the machine learning community is that this score is strongly correlated with subjective human judgment of image quality. Following the procedure in Salimans et al. (2016) , we generated 5000 examples from each trained generator and calculated the Inception score on the samples. We evaluated the score 10 times with different seeds for the generation of x n and reported the average and the standard deviation of the scores.

Fréchet inception distance (FID) (Heusel et al., 2017) is another measure for the quality of the generated examples that uses 2nd order information of the final layer of the inception model. The FID is based on Frećhet distance (Dowson & Landau, 1982 ) (Not to be confused with FID), which is the 2-Wasserstein distance between two distribution multivariate Gaussian distributions, p 1 and p 2 . The Wasserstein distance for two Gaussian distributions have a closed form, and is given by

EQUATION

)

where {µ p1 , C p1 }, {µ p2 , C p2 } are the mean and covariance of samples from q and p, respectively. Now, if f is the output of the final layer of the inception model before the softmax, the Fréchet inception distance (FID) between two distributions p 1 and p 2 on the images is the Frećhet distance distance between f • p 1 and f • p 2 . We empirically computed the Fréchet inception distance between the true distribution and the generated distribution using 10000 samples from the true distribution and 5000 samples from the generator distribution.

C.2 GAN'S OBJECTIVE FUNCTION:

We used applied DC regularizaiton to the training with the following set of objective functions (softplus(x) = log(1 + exp(x))).

GAN-vanilla (Goodfellow et al., 2014 ) min

EQUATION

GAN-variant1 (Goodfellow et al., 2014) max

EQUATION

min

EQUATION

GAN-variant2 max

EQUATION

min

EQUATION

GAN-hinge

EQUATION

WGAN-GP (Gulrajani et al., 2017) max

EQUATION

LSGAN (Mao et al., 2017) min

EQUATION

)

Feature Matching (Salimans et al., 2016) max

EQUATION

F is a linear function of φ (∃w with w T φ = F ) C.3 NETWORK ARCHITECTURES Table 2 : The architecture of DCGAN (Radford et al., 2016) for image Generation experiments on CIFAR-10 and CIFAR-100. The slopes of all leaky-ReLU functions in the networks were set to 0.2. 

z ∈ R 128 ∼ N (0, I) dense → M g × M g × 512 4×4,

Comparison with other methods:

Table 8 : Inception scores and FIDs with unsupervised image generation on CIFAR-10 and CIFAR-100. CIFAR-10 results for the models designated with † are cited from (Miyato et al., 2018) , and the CIFAR-10 results results with ‡ are cited from (Karras et al., 2018) 

:



Figures and Tables:

Fig 2 (b), the training about O 2 proceeds much faster than the training about O 1 . This seemingly unintuitive observation can be supported by the theory of optimal transport. Note that T 1 is a derivative of -K|X| and T (2) is a derivative of K|X|, and the latter is a convex function. This implies that T (2) is the optimal transport from µ to T (2) #µ. This result(Fig 2 (b)) suggests that, when using the update rule similar to the equation 10, the training proceeds much faster when we choose a target distribution with monotonic mapping.

Fig 3 and Fig 8 (in Appendix)  respectively summarize Inception scores and FID for all 12 settings. We see that DC regularization is improving the performance irrespective of the choice of the architecture and the objective function.

The inception scores of different GAN methods on CIFAR-10 and CIFAR-100 (higher the better). The right group of bars in each graph represents the set of scores achieved by the implementations with our regularization. The left group in each graph represents the set of scores achieved by the implementation without the regularization. Our regularization improves the inception score for all methods.

Fig 5: Performance of DC regularization on ImageNet. The baseline used in the comparison here is the GAN implemented with SN for Hinge-loss. The training with DC regularization achieves higher score consistently for all epochs.

Using p(y|x) to denote the Inception model, the inception score is given by I({x n } N n=1 ) := exp( Ê[D KL [p(y|x)||p(y)]]), where Ê indicates the empirically approximated expectation. Often times, p(y) is approximated with 1 N N n=1 p(y|x n ).

Fig 9: Results of WGAN-GP

high prior dimension Fig 10: FID performance of DC regularization on low dimensional prior and high dimensional prior.

(a) image generation of CIFAR-10 (b) image generation of CIFAR-100 Fig 11: Image generation of CIFAR-10 and CIFAR-100 D.3 APPENDIX RESULT ON IMAGENET (a) image generation of baseline model (b) image generation of proposal model Fig 12: Image generation of ImageNet

ResNet architectures for CIFAR-10 and CIFAR-100. We used similar architectures as the ones used inGulrajani et al. (2017).Resblock model is Fig 6

Table content: <table><tr><td/><td>RGB image x ∈ R M ×M ×3</td></tr><tr><td/><td>3×3, stride=1 conv 64 lReLU</td></tr><tr><td/><td>4×4, stride=2 conv 64 lReLU</td></tr><tr><td/><td>3×3, stride=1 conv 128 lReLU</td></tr><tr><td>stride=2 deconv. BN 256 ReLU</td><td>4×4, stride=2 conv 128 lReLU</td></tr><tr><td>4×4, stride=2 deconv. BN 128 ReLU</td><td>3×3, stride=1 conv 256 lReLU</td></tr><tr><td>4×4, stride=2 deconv. BN 64 ReLU</td><td>4×4, stride=2 conv 256 lReLU</td></tr><tr><td>3×3, stride=1 conv. 3 Tanh</td><td>3×3, stride=1 conv. 512 lReLU</td></tr><tr><td>(a) Generator, Mg = 4 for CIFAR-10 and CIFAR-100</td><td>dense → 1</td></tr><tr><td colspan="2">(b) Discriminator, M = 32 for CIFAR10 and CIFAR-</td></tr><tr><td>100</td><td/></tr></table>

ResNet generator architectures (large version) for CIFAR-10 and CIFAR-100.Resblock model is Fig 6

Table content: <table><tr><td>z ∈ R 128 ∼ N (0, I)</td></tr><tr><td>dense, 4 × 4 × 128</td></tr><tr><td>ResBlock up 256</td></tr><tr><td>ResBlock up 256</td></tr><tr><td>ResBlock up 256</td></tr><tr><td>BN, ReLU, 3×3 conv, 3 Tanh</td></tr><tr><td>(a) Generator</td></tr></table>

ResNet architectures for image generation on ImageNet dataset.Resblock model is Fig 6 z ∈ R 128 ∼ N (0, I)

Table content: <table><tr><td/><td>RGB image x ∈ R 64×64×3</td></tr><tr><td/><td>ResBlock down 64</td></tr><tr><td>dense, 4 × 4 × 1024</td><td>ResBlock down 128</td></tr><tr><td>ResBlock up 1024</td><td>ResBlock down 256</td></tr><tr><td>ResBlock up 512</td><td>ResBlock down 512</td></tr><tr><td>ResBlock up 256</td><td>ResBlock down 1024</td></tr><tr><td>ResBlock up 128</td><td>ResBlock 1024</td></tr><tr><td>ResBlock up 64</td><td>ReLU</td></tr><tr><td>BN, ReLU, 3×3 conv 3</td><td>Global sum pooling</td></tr><tr><td>Tanh</td><td>dense → 1</td></tr><tr><td>(a) Generator</td><td>(b) Discriminator for uncondi-</td></tr><tr><td/><td>tional GANs.</td></tr></table>

EGAN(Dai et al., 2017)'s decoder models used in our experiments on CIFAR-10 and CIFAR-100 . The slopes of all lReLU functions in the networks were set to 0.2. Fig 6 illustrates our Resblock model.

Table content: <table><tr><td>RGB image x ∈ R M ×M ×3</td><td>RGB image x ∈ R 32×32×3 ResBlock down 64</td></tr><tr><td>3×3, stride=1 conv 64 lReLU 4×4, stride=2 conv 64 lReLU</td><td>ResBlock down 128</td></tr><tr><td>3×3, stride=1 conv 128 lReLU</td><td>ResBlock down 256</td></tr><tr><td>4×4, stride=2 conv 128 lReLU</td><td>ResBlock down 512</td></tr><tr><td>3×3, stride=1 conv 256 lReLU</td><td>ResBlock down 1024</td></tr><tr><td>4×4, stride=2 conv 256 lReLU</td><td>ResBlock 1024</td></tr><tr><td>3×3, stride=1 conv. 512 lReLU</td><td>ReLU</td></tr><tr><td>dense → 128 * 2</td><td>Global sum pooling</td></tr><tr><td>(a) Convolution decoder, M = 32 for CIFAR10 and CIFAR-100</td><td>dense → 128 * 2</td></tr><tr><td/><td>(b) ResNet decoder.</td></tr></table>

. For the details of the objective functions, see Appendix C.2.

Table content: <table><tr><td>Method</td><td colspan="4">Inception score CIFAR-10 CIFAR-100 CIFAR-10 CIFAR-100 FID</td></tr><tr><td>Real data</td><td>11.24</td><td>14.79</td><td>7.6</td><td>8.94</td></tr><tr><td>EGAN-Ent-VI(SNDCGAN)</td><td>6.95±.08</td><td>6.62±.10</td><td>29.0</td><td>33.3</td></tr><tr><td>EGAN-Ent-VI(SNResNet)</td><td>7.31±.12</td><td>6.67±.10</td><td>27.0</td><td>30.5</td></tr><tr><td>feature matching(SNDCGAN)</td><td>7.54±.10</td><td>7.71±.06</td><td>25.9</td><td>29.2</td></tr><tr><td>proposal</td><td/><td/><td/><td/></tr><tr><td>SNDCGAN + DC reg</td><td>8.08±.12</td><td>8.12±.11</td><td>24.6</td><td>25.8</td></tr><tr><td>SNDCGAN-hinge + DC reg</td><td>7.70±.11</td><td>7.99±.09</td><td>24.7</td><td>26.1</td></tr><tr><td>SNResNet + DC reg</td><td>8.27±.08</td><td>8.27±.13</td><td>24.3</td><td>24.6</td></tr><tr><td>SNResnetLarge + DC reg</td><td>8.41±.10</td><td>8.20±.08</td><td>20.6</td><td>24.8</td></tr><tr><td>SNResnetLarge-hinge + DC reg</td><td>8.29±.09</td><td>8.41±.11</td><td>19.5</td><td>23.6</td></tr><tr><td>baseline</td><td/><td/><td/><td/></tr><tr><td>SNDCGAN †</td><td>7.42±.08</td><td>7.74±.08</td><td>29.3</td><td>27.9</td></tr><tr><td>SNDCGAN-hinge †</td><td>7.58±.12</td><td>7.57±.07</td><td>25.5</td><td>28.1</td></tr><tr><td>SNResnetLarge-hinge †</td><td>8.22±.05</td><td>7.54±.13</td><td>21.7</td><td>26.6</td></tr><tr><td>Progressive GANs ‡</td><td>8.56±.06</td><td/><td/><td/></tr></table>

Peer review (iclr2019):
 Summary: This paper proposes distributional concavity regularization for GANs which encourages producing generator distributions with higher entropy. The paper motivates the proposed method as follows: - Using the concept of functional gradient, the paper interprets the update in the generator parameters as an update in the generator distribution - Given this functional gradient perspective, the paper proposes updating the generator distribution toward a target distribution which has *higher entropy and satisfies monoticity* - Then, the paper proves that this condition can be satisfied by ensuring that generator’s objective (L) is concave - Since it’s difficult to ensure concavity when parametrizing generators as deep neural networks, the paper proposes adding a simple penalty term that encourages the concavity of generator objective Experiments confirm the validity the proposed approach. Interestingly, the paper shows that performance of multiple GAN variants can be improved with their proposed method on several image datasets Strengths: - The proposed method is very interesting and is based on sound theory - Connection to optimal transport theory is also interesting - In practice, the method is very simple to implement and seems to produce good results Weaknesses: - Readability of the paper can be generally improved. I had to go over the paper many times to get the idea. - Figures should be provided with more detailed captions, which explain main result and providing context (e.g. explaining baselines). Questions/Comments: - Equation (7) has typos (uses theta_old instead of theta in some places) - Section 4.1 (effect of monoticity) is a bit confusing. My understanding is that parameter update rule of equation (3) and (6) are equivalent, but you seem to use (6) there. Can you clarify what you do there and in general this experiment a bit more? - Comparing with entropy maximization method of EGAN (Dai et al, 2017) is a good idea, but I’m wondering if you can compare it on low dimensional settings (e.g. as in Fig 2). It is also not clear why increasing entropy with EGAN-VI is worse than baselines in Table 1. Overall recommendation: The paper is based on sound theory and provides very interesting perspective. The method seems to work in practice on a variety of experimental setting. Therefore, I recommend accepting it.