Title: VARIANCE REGULARIZING ADVERSARIAL LEARNING

Abstract: We study how, in generative adversarial networks, variance in the discriminator's output affects the generator's ability to learn the data distribution. In particular, we contrast the results from various well-known techniques for training GANs when the discriminator is near-optimal and updated multiple times per update to the generator. As an alternative, we propose an additional method to train GANs by explicitly modeling the discriminator's output as a bi-modal Gaussian distribution over the real/fake indicator variables. In order to do this, we train the Gaussian classifier to match the target bi-modal distribution implicitly through metaadversarial training. We observe that our new method, when trained together with a strong discriminator, provides meaningful, non-vanishing gradients.

1. INTRODUCTION:

Generative adversarial networks (Goodfellow et al., 2014) are a framework for training a generator of some target (i.e., "real") distribution without explicitly defining a parametric generating distribution or a tractable likelihood function. Training the generator relies on a learning signal from a discriminator or discriminator, which is optimized on a relatively simple objective to distinguish between (i.e., classify) generated (i.e., "fake") and real samples. In order to match the true distribution, the generator parameters are optimized to maximize the loss as defined by the discriminator, which by analogy makes the generator and discriminator adversaries.

In recent years, GANs have attained strong recognition as being able to generate high-quality images with sharp edges in comparison to maximum-likelihood estimation-based methods (Dempster et al., 1977; Kingma & Welling, 2013; Salakhutdinov & Hinton, 2009; Berthelot et al., 2017; Zhao et al., 2016) . Despite their recent successes, GANs can also be notoriously hard to train, suffering from collapse (i.e., mapping its noise to a very small set of singular outputs), missing modes of the real distribution (Che et al., 2017) , and vanishing and/or unstable gradients (Arjovsky & Bottou, 2017) . In practice, successful learning is highly reliant on hyperparameter-tuning and model choice, and finding architectures that work with adversarial learning objectives with any / all of the above problems can be challenging (Radford et al., 2015) .

Many methods have been proposed to address learning difficulties associated with learning instability.

• The use of autoencoders or posterior models in the generator or discriminator. These have been shown help to alleviate mode collapse or stabilize learning (Larsen et al., 2015; Che et al., 2017; Dumoulin et al., 2017; Donahue et al., 2016) .

• Regularizing the discriminator, such as with input noise (Arjovsky & Bottou, 2017) , instance noise (Salimans et al., 2016) , and gradient norm regularization (Roth et al., 2017) .

• Alternate difference measures, such as integral probability metrics (IPMs, Sriperumbudur et al., 2009) metrics. The most well-known of these use a dual formulation of the Wasserstein distance. These are implemented via either weight clipping (Arjovsky et al., 2017) or gradient penalty (Gulrajani et al., 2017) . These can yeild stable gradients, high-quality samples and work on a variety of architectures.

On this last point, it is unclear whether the metric or the associated regularization used to impose Lipschitz is important, as regularization techniques have also been shown to be effective with stabilizing learning in f -divergences (Roth et al., 2017) . In this paper, we study the integral role of variance in the discriminator's output in the regime of the generated distribution and how it ultimately affects learning. In the following sections, we describe theoretical motivations, an empirical analysis from multiple variants of GANs, and finally propose a regularization scheme to combat vanishing gradients on part of the discriminator when it is well-trained.

2. METHODS:

The generative adversarial framework entails training a generating function G θ : Z → X , where z ∈ Z is noise sampled from a relatively simple prior (such as spherical Gaussian), p(z), and G θ is modeled by a deep neural network with parameters, θ. The objective is to find the optimal parameters such that the induced or generated distribution, Q θ , matches a desired target distribution, P. In order to accomplish this, adversarial models define a discriminator function, D φ,d : X → R, modeled by a deep neural network with parameters, φ. The objective of the discriminator is to estimate a measure of distance or divergence, d(P, Q θ ), between the two distributions, P and Q θ . In practice, which divergence is best to estimate in this context is still a subject of debate, but it suffices to say that the most common distances come from either the family of f -divergences (Nguyen et al., 2010; Nowozin et al., 2016) , such as the Jensen-Shannon divergence (JSD) or χ-squared divergence (Mao et al., 2016) or Integral Probability Metrics (IPMs, Sriperumbudur et al., 2009; Mroueh & Sercu, 2017) , such as the Wasserstein distance (Arjovsky et al., 2017; Gulrajani et al., 2017) and maximum mean discrepancy (MMD, Sutherland et al., 2016; Li et al., 2017) . Generally, the estimator given by the family of functions defined by D φ,d represent a family of lower-bounds of the respective divergence:

EQUATION

In Goodfellow et al. (2014) 

min θ max φ V JSD (D φ,d , G θ , P) = min θ max φ E x∼P [log D(x)] + E x∼Q θ [log (1 -D(x))]. (2)

where V JSD (D φ,d , G θ , P) is a lower bound to JSD = 2 * JSD -2 log 2. Here, D = σ •φ1 is the composition of the sigmoid function, σ(y) = 1 1+exp(-y) , and a neural network, φ. More generally, Nowozin et al. (2016) more formally define a GAN objective in terms of a family of f -divergences:

EQUATION

where f † is a convex conjugate to the f -divergence generator function and ψ f = g f • φ is the composition of a nonlinearity and neural network, such that g f respects the domain of the domain of the corresponding f -divergence.

2.1. STRONG OR WEAK METRIC?:

The above family of f -divergence estimators such as the JSD one above are powerful and can produce generators with high quality samples on complex datasets (Radford et al., 2015) , but there is some evidence that they provide poor learning signals for the generator parameters, θ. As shown in Arjovsky & Bottou (2017) , the JS divergence will be flat everywhere important if P and Q θ both lie on low-dimensional manifolds (as is likely the case with real data) and do not prefectly align. In principle, one would think that a better divergence estimate (tighter bound) to a strong divergence metric would provide the most meaningful learning signal for the generator. In practice, however, as we optimize the JSD discriminator, it will have zero gradients on X and hence provide almost no learning signal for the generator. In addition, the proxy loss,

E x∼Q θ [log D(x)]

, which is commonly used as an alternative to the GAN objective in training the discriminator can be highly unstable, and is generally plagued by missing modes and model collapse. Finally, as we will show below, other f -divergences (e.g., Mao et al., 2016) suffer different types of instabilities that can make training difficult to impossible. Some of the above observations motivate the use of a "weaker" metric that use weaker topologies than that of f -divergences, notably the earth-movers distance or Wasserstein distance (Arjovsky et al., 2017) . Besides being a useful information metric for computing the distance between two distributions, the Wasserstein distance has the nice property of being differentiable nearly everywhere, and thus provides a better convergence properties on the generator. The Kantorovich-Rubinstein dual problem of estimating the Wasserstein distance involves the GAN objective:

EQUATION

where D is constrained to be in the set of Lipschitz-1 functions2 . This GAN objective is a lowerbound for the Wasserstein, as we saw with f -GANs and the f -divergences, and adversarial models that follow this metric are called Wasserstein GANs (WGAN Arjovsky et al., 2017) . The Lipchitz constraint should ensure a smoothness or continuity or the gradients being given to the generator. In principle then, under this constraint, one can train the discriminator to optimality without having to worry about any catastrophic effects on the generator.

The difficulty, however, is in constraining D to be Lipschitz, and it is conventional to use some sort of regularization (Arjovsky et al., 2017; Gulrajani et al., 2017) . However, questions remain regarding said regularization and whether the supremum of the resulting family of functions is close enough to the true Wasserstein distance. In addition, there is strong evidence that regularization on f -GANs can improve training stability (Roth et al., 2017) , indicating that having a looser bound over a constrained family of discriminator functions may in principle be a better strategy for training a GAN.

2.2. THE IMPORTANCE OF VARIANCE:

Taking the perspective provided in the previous section, our goal becomes to train a discriminator to near-optimality in the hopes that this will improve learning in the generator. A well-known scenario is one in which the discriminator can perfectly discern samples from P from samples from Q θ . Moreover, (Arjovsky & Bottou, 2017) show that the discriminator is likely to be constant in the regime of Q θ , thus emitting zero gradients and no learning signal for the generator. Consider such a discriminator; if x, y ∼ Q θ are samples, then as the generator attempts to minimize any discrepancy between P and Q θ , the slope of the tangent hyperplane connecting (x, φ(x)) and (y, φ(y)) likely obeys

|φ(x) -φ(y)| d X (x, y) = 0 (5)

where d X (•, •) is a metric defined on X . This roughly corresponds to a flat discriminator over Q θ which doesn't emit any gradients and impedes the generator from learning. As a result, we find

EQUATION

Alternatively, a flat discriminator in the regime of Q θ can be interpreted as having minimal variance (see Figure 1 ). That is, when the discriminator is mostly constant over P and Q θ , the discriminator's unnormalized output distribution is peaked and tends to mimic a bi-modal Dirac distribution. This is because, in a sense, the discriminator acts as a bi-modal mapping: real examples from P are mapped to, for example, 1 while generated examples from Q θ (x) are mapped to 0. In contrast, when the discriminator is non-constant over the data/generated manifolds and exhibits variance in its output over within P and/or Q θ , its unnormalized output distribution is spread out and not peaked.

2.3. VARIANCE REGULARIZING GANS:

Following these observations, we propose a regularization technique targetting the discriminator via meta-adversarial learning. We hypothesize that constraining the discriminator's output distribution Figure 1 : The red line represents the discriminator's output, the blue and green lines represent the real and fake distributions, respectively. Top: When the discriminator is near-perfect, it's unnormalized output distribution is peaked. Bottom: Variance in discriminator output within the data and/or generated distribution will result in variance in its unnormalized output distribution. Note that the discriminator's output need not be in the range (0, 1), we provide just an example here.

to follow a mixture of Gaussians will have the same effect as the use of different metrics and/or regularization as seen in works discussed above.

Let p(y|l = 0) and p(y|l = 1) be two univariate Gaussian distributions with unit variance and means µ f and µ r , respectively, and where the indicator variable as values, l = 0 that corresponds to "fake" or generated and l = 1 that corresponds to real. Define the mixture:

EQUATION

Depending on the location of the means, this Gaussian classifier has an associated classification error that is directly related to their overlap. In principle, the classifier defines a bound on a difference or distance between two distributions defined by the values of the indicator variable, l = 0 and l = 1. As the discriminator is really a classifier that in principle is trying to minimize the Bayes risk, under the above constraint, the bound is maximized when the means, µ f and µ r , are as far apart as possible. This is obviously undesirable as this can lead to the same problems mentioned in the previous sections. Here we will take a different strategy: keep the means and variance fixed so that the associated classifier has a fixed error rate. This will enforce a sort of fixed "weakness" on the discriminator, which corresponds to a fixed "looseness" on the lower bound defined by the classifier.

We ensure the discriminator's output follows p φ (y) through adding two additional discriminators which are trained in a similar was as the original GAN formulation:

EQUATION

where F and R are the are the fake and real output discriminators, respectively, each parametrized by a two-layer MLP. F's goal is to discern a batch of real values {y i } i drawn from a Gaussian with mean µ f and unit variance from a batch taken from the discriminator's output over fake samples. Likewise, R follows a similar protocol for real samples. It is important to note that comparisons are performed batch-wise rather than per sample. In essence, F and R are meta-discriminators which instill a regularizing effect on the discriminator.

Meanwhile, the original GAN formulation is only slightly perturbed, as the generator tries to match Q θ to P while the discriminator attempts to differentiate real and fake samples. The discriminator's loss objective combines equations 8 and 9:

EQUATION

The generator's loss function is:

EQUATION

standard of IPMs. In practice, however, we find that driving the discriminator's output towards µ r using a least squares loss also works well:

EQUATION

)

Algorithm 1 Variance Regularizing Adversarial Learning (VRAL) using meta-discriminators. Default parameter values are α = 10 -4 , β 1 = 0.5, β 2 = 0.999, = 3 × 10 -6 , m = 64. n batches is determined by the chosen dataset. Input: n F ,R,φ , the number of updates to perform on F, R and φ per update to G θ . 1: while θ G not converged do 2:

for i = 1 . . . n batches do 3:

for k = 1 . . . n F ,R,φ do 4:

Sample{y (i) } m i=1 ∼ N f (y), {y (i) } m i=1 ∼ Q θ 5: g F ← ∇ θ F 1 m m i=1 log F({y (i) } i ) + log (1 -F({φ(y (i) )} i )) 6: θ F ← θ F + α • Adam(g F , β 1 , β 2 , ) 7: Sample{y (i) } m i=1 ∼ N r (y), {x (i) } m i=1 ∼ P 8: g R ← ∇ θ R 1 m m i=1 log R({y (i) } i ) + log (1 -R({φ(x (i) )} i )) 9: θ R ← θ R + α • Adam(g R , β 1 , β 2 , ) 10: Sample{x (i) } m i=1 ∼ P, {y (i) } m i=1 ∼ Q θ 11: g φ ← ∇ θ φ 1 m m i=1 log F({φ(y (i) )} i ) + log R({φ(x (i) )} i )]

12:

θ φ ← θ φ + α • Adam(g φ , β 1 , β 2 , ) 13: Sample{x (i) } m i=1 ∼ P, {y (i) } m i=1 ∼ Q θ 14: g G ← ∇ θ G 1 m m i=1 φ(y (i) ) -φ(x (i) ) 15: θ G ← θ G -α • Adam(g G , β 1 , β 2 , )

The training procedure for our proposed GAN with regularizing networks is similar to standard GAN training, however requiring each of the meta-discriminators to be updated prior to the discriminator update. In practice, we update F, R and D the same n F ,R,D number of times per generator update.

The full training procedure is presented in algorithm 1.

Our current formulation of Variance Regularizing GANs aims to fix

E x∼P [φ(x)] and E x∼Q θ [φ(x)]

at values µ r and µ f , respectively. However, this can become problematic during optimization as the discriminator's goal is to maximize the difference between these two means. To accommodate, we subtract the mean of the discriminator's output over a batch of fake samples. This useful insight results in the modified objective in which the discriminator's output over fake examples is no longer fixed to a distribution centered at µ f , but can move around and preserves unit variance. Consequently, each occurrence of {φ( ) ]. As we will show in subsequent sections, this modification proves extremely helpful.

x (i) )} i where x ∼ Q θ in equation 10 is replaced with {φ(x (i) )} i -E x∼Q θ [φ(x

2.4. META-ADVERSARIAL LEARNING:

In future sections, we show our method which uses meta-discriminators to regularize the discriminator's output distribution to be successful. However, this same technique can be applied to encourage GANs exhibit other desirable properties. To this end, we demonstrate how we can enforce the discriminator to approximate a K-Lipschitz function. Consider a similar setup to Variance Regularizing GANs in which only one meta discriminator M is used, and its objective is

EQUATION

where µ * = K and σ * can vary, but is typically close to 0 (e.g. 0.1). The discriminator's objective is almost identical to equation 10, but its goal is to now maximize the second term in the above equation. In essence, the single meta-discriminator M is distinguishing samples drawn from a Gaussian distribution fixed at µ * versus the slope of the hyperplane between x and y, where x and y are real and generated examples, respectively.

3. RELATED WORK:

Linear Discriminant Analysis (LDA) is a commonly used technique for dimensionality reduction and/or binary classification. Consider two sets of classes, P and Q θ , and a set of observations {φ(x)}, where each x ∈ X (i.e., φ(x) may be interpreted as the result of dimensionality reduction on x). The objective is to preserve the discrepancy between classes while maintaining intra-class variance. Let p φ (φ(x); x ∼ P) and p φ (φ(x); x ∼ Q θ ) be the probability densities of each class, assumed to follow a Gaussian distribution with means µ r and µ f and variances σ 2 r and σ 2 f , respectively. For the sake of simplicity, assume σ 2 r = σ 2 f = 1. The LDA classification rule is then to predict φ(x) belonging to the class of samples from

EQUATION

otherwise P, and T is a chosen threshold. Notice that when φ(x) is near µ r , the LHS of equation 14 becomes small (assuming φ(x) is far from µ f ) and vice-versa when φ(x) is close to µ f .

McGan (Mroueh et al., 2017) employs mean and covariance feature matching between P and Q θ . To achieve this, a function Φ : X → R m , parametrized as a deep neural network, which maps x ∈ X to an embedding space is learned. The objective is to minimize the mean and covariance between Φ(P) and Φ(Q θ ). More recently, Fisher GANs (Mroueh & Sercu, 2017) constrain the second order raw moments of the discriminator such that 1 2 (E x∼P [φ 2 (x)] + E x∼Q θ [φ 2 (x)]) = 1 using Lagrange multipliers. This approach slightly differs from Variance-Regularizing GANs, which enforce that variance, the second order central moments of the discriminator, are equal to 1. Nonetheless, Fisher GANs are similar to Variance Regularizing GANs as they aim to stabilize variance in the discriminator's output, although not explicitly defining a distribution which it should follow.

4. EMPIRICAL RESULTS:



4.1. GAN ARCHITECTURE:

We run experiments on the CIFAR-10 ( Krizhevsky & Hinton, 2009) and CelebA (Liu et al., 2015) datasets for image generation. We use a DCGAN (Radford et al., 2015) model in all experiments and perform all parameter updates using the Adam optimizer (Kingma & Ba, 2015) . For our proposed method, the meta-discriminators F and R are MLPs with 28 hidden units with a sigmoid output.

4.2. ANALYSIS:

Having established theoretical motivations for a well-trained discriminator, we show how many GAN algorithms fail to preserve variance in the discriminator output when trained at large ratios, such as 50 discriminator updates per generator update. We train the standard GAN, standard GAN using proxy loss (i.e., -log D loss), Least Squares GAN (Mao et al., 2016) , Wasserstein GAN and our proposed method (both with and without subtracting means) on CIFAR-10 with a varying number of discriminator updates per generator update. We find that variance in the discriminator's unnormalized output distribution directly correlates with the generator's ability to learn if the discriminator is well-trained. Figure 2 shows samples from updating the discriminator 50 times per generator update (which we refer to as 50:1 training), while figure 3 demonstrates the discriminator's expected real-valued output over the training set. Interestingly, the discriminators in standard and Least Squares GANs exhibit almost no output variance within P or Q θ , suggesting the discriminator is constant over these regimes in image space and would emit zero gradients w.r.t. any generated sample, hence hindering learning in the generator.

Notice that these methods fail to learn at a 50:1 ratio Furthermore, the generated samples also posit that some amount of variance is necessary; GANs using -log D and Wasserstein losses are able to generate sharp samples while the discriminator in these instances is imperfect and exhibits some level of uncertainty amongst the generated samples.

Our hypothesis of (near) zero gradients is evidently supported as we contrast the discriminator's gradient norms for any particular GAN at varying update ratios. Figure 4 indicates that as the discriminator achieves perfection by being updated more frequently than the generator, ||∇ x∼Q θ L φ || approaches zero. However, the standard GAN with -log D loss and Wasserstein GAN are much more robust to large update ratios. The discriminators of these methods don't map all samples, real or fake, to a single value, hence providing a meaningful learning signal for the generator and avoiding the vanishing gradient scenario. Our proposed method follows this line of reasoning, as the meta-discriminators prevent the discriminator from achieving perfection, regardless of update ratio.

4.3. 1-LIPSCHITZ DISCRIMINATORS:

Adhering to our earlier discussion of using a single meta-discriminator to force the discriminator to approximate a K-Lipschitz function, we trained a meta-discriminator M to perform batch-wise discrimination between samples drawn from N (y; µ * = 1, σ * = 0.1) and |φ(x)-φ(y)| d X (x,y)

where x ∼ P, y ∼ Q θ . See figure 6 for results. More work is needed to robustly train GANs using meta-adversarial learning.

5. CONCLUSIONS:

In this paper, we have demonstrated the importance of intra-class variance in the discriminator's output. In particular, our results show that methods whose discriminators tend to map inputs of a class to single real values are unable to provide a reliable learning signal for the generator. Furthermore, variance in the discriminator's output is essential to allow the generator to learn in the presence of a well-trained discriminator. We proposed a technique, conceptually in line with LDA, which ensures the discriminator's output distribution follows a specified prior. Taking a broader perspective, we also introduced a new regularization technique called meta-adversarial learning, which can be applied to ensure enforce various desirable properties in GANs.

:

Here, we introduce a slight abuse of notation, using φ to represent a deep neural network and its parameters.

Though the K-Lipschitz condition suffices, in practice, it is common to use 1-Lipschitz as this does not change the optimization in any meaningful way.

Figures and Tables:

Figure 2: Generated CIFAR-10 samples using the standard GAN, GAN with -log D loss, Least Squares GAN, Wasserstein GAN, and two versions of VRAL (with and without subtracting means) over various training ratios. The top row shows results from training each method at a 1:1 training ratio. Likewise, the second, third, and fourth shows show results from 10:1, 25:1 and 50:1 training. The standard and least squares GANs are not robust to such large training ratios while others continue to generate sharp images.

Figure6: Generated images on CelebA by forcing the discriminator to be 1-Lipschitz using a single meta-discriminator.

, V d is called the value function. Optimization of the discriminator function by maximizing the value function roughly corresponds to finding the supremum over all D φ,d . The GAN objective is the central quantity of interest in the adversarial saddle-point problem, given P, p(z), G θ , and D φ,d :

Table content: <table/>

Peer review (iclr2018):
 This paper studies how the variance of the discriminator affect the gradient signal provided to the generator and therefore how it might limit its ability to learn the true data distribution. The approach suggested in this paper models the output of the discriminator using a mixture of two Gaussians (one for “fake” and the other for “not fake”). This seems like a rather crude approximation as the distribution of each “class” is likely to be multimodal. Can the authors comment on this? Could they extend their approach to use a mixture of multimodal distributions? The paper mentions that fixing the means of the distribution can be “problematic during optimization as the discriminator’s goal is to maximize the difference between these two means.“. This relates to my previous comment where the distribution might not be unimodal. In this case, shifting the mean doesn’t seem to be a good solution and might just yield to oscillations between different modes. Can you please comment on this? Mode collapse: Can you comment on the behavior of your approach w.r.t. to mode collapse? Implementation details: How is the mean of the two Gaussians initialized? Relation to instance noise and regularization techniques: Instance noise is a common trick being used to train GANs, see e.g. http://www.inference.vc/instance-noise-a-trick-for-stabilising-gan-training/ This also relates to some regularization techniques, e.g. Roth et al., 2017 that provides a regularizer that amounts to convolving the densities with white Gaussian noise. Can you please elaborate on the potential advantages of the proposed solution over these existing techniques? Comparison to existing baselines: Given that the paper addresses the stability problem, I would expect some empirical comparison to at least one or two of the stability methods cited in the introduction, e.g. Gulrajani et al., 2017 or Roth et al., 2017. Relation to Kernel MMD: Can the authors elaborate on how their method relates to approaches that replace the discriminator with MMD nets. e.g. - Training generative neural networks via Maximum Mean Discrepancy optimization, Dziugaite et al - Generative models and model criticism via optimized maximum mean discrepancy, Sutherland et al More explicitly, the variance in these methods can be controlled via the bandwidth of the kernel and I therefore wonder what would one use a simple mixture of Gaussians instead?