Title: ON THE ANOMALOUS GENERALIZATION OF GANS

Abstract: Generative models, especially Generative Adversarial Networks (GANs), have received significant attention recently. However, it has been observed that in terms of some attributes, e.g. the number of simple geometric primitives in an image, GANs are not able to learn the target distribution in practice. Motivated by this observation, we discover two specific problems of GANs leading to anomalous generalization behaviour, which we refer to as the sample insufficiency and the pixel-wise combination. For the first problem of sample insufficiency, we show theoretically and empirically that the batchsize of the training samples in practice may be insufficient for the discriminator to learn an accurate discrimination function. It could result in unstable training dynamics for the generator, leading to anomalous generalization. For the second problem of pixel-wise combination, we find that besides recognizing the positive training samples as real, under certain circumstances, the discriminator could be fooled to recognize the pixel-wise combinations (e.g. pixel-wise average) of the positive training samples as real. However, those combinations could be visually different from the real samples in the target distribution. With the fooled discriminator as reference, the generator would obtain biased supervision further, leading to the anomalous generalization behaviour. Additionally, in this paper, we propose methods to mitigate the anomalous generalization of GANs. Extensive experiments on benchmark show our proposed methods improve the FID score up to 30% on natural image dataset.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) have great potential in modeling complex data distributions and have attracted significant attention recently. A great number of techniques (Goodfellow et al., 2014; Miyato et al., 2018; Arjovsky et al., 2017; Gulrajani et al., 2017; Salimans et al., 2016; Brock et al., 2018) and architectures (Radford et al., 2015; Karras et al., 2018; Zhang et al., 2018; Mirza & Osindero, 2014) have been developed to make the training of GANs more stable and to generate high fidelity, diverse images. The corresponding generated samples are authentic and difficult for human to distinguish from the real ones. Despite these improvements, recent work (Zhao et al., 2018) reported a surprising phenomenon of anomalous generalization of GANs on a geometry dataset, raising new questions about the generalization behaviour. By anomalous generalization it means that several seemingly easy attributes are shown to be learned poorly by GANs, including numerosity (number of objects) and color proportion, which are important for human perception. For example, as shown in Figure 1 , for a geometric-object training dataset where the number of objects for each training image is fixed (e.g. every training image has exactly two rectangles), it is observed that most generated images after training have very different numbers of objects than the training images (e.g. rectangle numbers of most generated images are not two). Mathematically speaking, with regard to the number of objects, the learned distribution of GANs differs significantly from the target distribution, which fails to achieve the goal of modeling the target data distribution faithfully.

Several works have developed theories for GANs. The original work proves the convergence to equilibrium under ideal conditions (Goodfellow et al., 2014) . Further extensions include (Arjovsky et al., 2017; Miyato et al., 2018; Nagarajan & Kolter, 2017; Mescheder et al., 2018; Bai et al., 2018; Heusel et al., 2017) . Arora et al. (2017) points out that GANs may not have good generalization when the discriminator has finite capacity, e.g., neural networks. But they show generalization occurs for GANs under the weak metric of neural net distance. Although these theories provide Motivated by this observation, we discover and investigate two specific problems of GANs, namely sample insufficiency and pixel-wise combination, which cause GANs to have anomalous generalization behaviour. Moreover, we propose methods to improve the generalization of GANs.

For the problem of sample insufficiency, we show theoretically and empirically that the batchsize in practice could be insufficient for GANs to model the target data distribution. As a typical setting of GANs, the discriminator learns to separate the fake data distribution of the generator from the real data distribution approximated by the training dataset. In practice, the discriminator learns such a separation function based on the mini-batches sampled from the training dataset and the generated samples of the generator. However, since the size of the mini-batch is much smaller than all the possible samples in the high-dimensional data distribution, the separation function of the discriminator learned based on the mini-batch samples could be noisy. With this noisy discriminator as reference, the generator would learn noisy generation function too and the training dynamics become unstable. As a result, GANs would have anomalous generalization behaviour.

For the problem of pixel-wise combination, in some situations, we find that the positive training samples and their pixel-wise combinations (pixel-wise average or pixel-wise logical-and) are both recognized as real by the discriminator during training. However, the pixel-wise combinations of the positive training samples could have very different properties from the real samples themselves, indicating that the discriminator is unable to differentiate those seemingly easy attributes (e.g. number of objects). With this fooled discriminator as reference, the generator could be fooled further to generate those pixel-wise combinations of training samples, which may not belong to the target distribution. As a result, the data distribution learned by the generator could be very different from the target data distribution and the generalization of GANs becomes anomalous.

To summarize, our contributions are:

• We show that in certain circumstances the discriminator tends to recognize the pixel-wise combinations of the positive training samples as real, which could fool the generator to have anomalous generalization behaviour.

• We demonstrate theoretically and empirically that the sample insufficiency in practice could result in unstable training dynamics and anomalous generalization of GANs.

• We show that the anomalous generalization reported in Zhao et al. (2018) is caused by the two problems (sample insufficiency and pixel-wise combination). We then propose novel methods to mitigate anomalous generalization behaviour. Figure 1 shows that our proposed methods improve the proportion of correct generated images by almost 80%. Our methods also improve the FID up to 30% on natural image datasets.

2. BACKGROUND:



2.1. GENERATIVE ADVERSARIAL NETWORKS:

In most cases of GANs, the generator learns to map a prior distribution (e.g. standard Gaussian) to a fake distribution to approximate the target real data distribution. The discriminator learns a function to separate the real and fake distributions. They define the following min-max game:

EQUATION

where P r and P z denote the real and prior distributions respectively. f 1 and f 2 are the critic functions for the positive and negative training samples (e.g.

f 1 (x) = log(x), f 2 (x) = log(1-x)).

2.2. ANOMALOUS GENERALIZATION BEHAVIOUR OF GANS:

Some anomalous generalization behaviours of GANs have been observed recently. In Zhao et al. (2018) , several seemingly easy attributes are shown to be learned poorly by GANs, including numerosity (number of objects) and color proportion, which are important for human vision systems.

The phenomenon shows that the learned distribution of the generator fails to approximate the target distribution, raising new questions about the training dynamics and generalization behaviour of GANs.

3. SAMPLE INSUFFICIENCY:

In this section, we first discuss the problem of sample insufficiency in general in Section 3.1. Its empirical observation is shown in 3.2, followed by the theoretical analysis in Section 3.3.

3.1. SAMPLE INSUFFICIENCY IN THE GENERAL TRAINING OF GANS:

In the training of GANs, the generator learns to fake the target distribution. The discriminator learns to separate the fake distribution of the generator from the real data distribution. To learn a good separation function between the fake and the real distributions, the discriminator needs to have sufficient information of them. But in practice, such information of the distribution is provided by the positive or negative training samples in the mini-batch. Since the batchsize is often much smaller than the size of all possible data in the high-dimensional data distribution, the information is insufficient and the separation function of the discriminator learned based on the mini-batch samples is noisy. With this noisy discriminator as reference, the generator could also learn a noisy generation function and the training dynamics of GANs become unstable. The smaller the batchsize is, the more unstable the training dynamics are. Since the training of the generator is unstable and the learned generation function is noisy, it is difficult for the learned distribution to approximate the target distribution. As a result, the generalization of GANs would become anomalous. We will show both empirically and theoretically that the sample insufficiency leads to anomalous generalization behaviour of GANs in the following subsections.

3.2. EMPIRICAL VERIFICATION:

We do experiments to show that the problem of sample insufficiency could lead GANs to anomalous generalization, both for geometric and natural image generation.

We establish a geometry dataset consist of 64 images (while the experiment also applies to larger size) that all images (32 by 32 with 0/255 binary pixel value) in it have exactly two rectangles (8 by 8). The prior distribution is the discrete uniform distribution. Size of its support set is the same as the dataset. More details can be found in Appendix E. We compare the mini-batch gradient descent (MGD) with the full-batch gradient descent (FGD). The batchsize for MGD and FGD is 16 and 64 respectively.

As shown in Figure 2 (middle), caused by the sample insufficiency, the training dynamics for the mini-batch gradient descent (MGD) are highly unstable. Both the gradient and the loss go up and down frequently, suggesting it is difficult for GANs to model the target distribution if trained with ) change frequently during training with anomalous images of false rectangle numbers. But for the full-batch gradient descent (FGD), where the sample insufficiency is avoided and the separation function of the discriminator between the real and fake data distributions can be learned accurately at each step, the loss and gradients are stable. The learned distribution converges smoothly to the target distribution in a short time.

The problem of sample insufficiency is also observed to influence the natural image generation for datasets like CELEB-A (Liu et al., 2015) . As shown in Figure 2 (right), after trained with the same amount of data, the FID score is better for the larger batchsize, where the problem of sample insufficiency is relatively less severe, than the smaller batchsize, where the sample insufficiency is more problematic. In brief, the experiments show that the sample insufficiency makes the training dynamics unstable, both for the discriminator and the generator. Since the training of the generator and the discriminator are unstable and the learned generation function is noisy, it is hard for the learned distribution to approximate the target distribution. As a result, the final generalization behaviour of GANs becomes anomalous.

3.3. THEORETICAL ANALYSIS: A WGAN MODEL:

In this subsection, we introduce a simple yet prototypical model which shows that the insufficient batchsize will result in unstable training dynamics and smaller batch leads to poorer performance than that of larger batch. As a result, the generalization of GANs would become anomalous when the batchsize is small. Assume that the real data distribution is a d-dimensional multivariate normal distribution centered at the origin, N (0, I d ). The latent distribution of the generator is

p z ∼ N (0, I d ). The generator is defined as G θ (x) = θ + x. The discriminator is a linear function D w (x) = w x.

We consider the WGAN model (Arjovsky et al., 2017) , whose value function is constructed using the Kantorovich-Rubinstein duality (Villani, 2008) as

EQUATION

where f L denotes the Lipschitz constant of the function f , namely, f L = inf{L : f is L-Lipschitz}. And P θ REAL denotes the real data distribution, P θ denotes the distribution of the generator G θ , and the supremum is taken over all the linear functions f w (x) = w x since the optimal classifier f * is absolutely linear. And when f w is a linear function, f w L = w , so

EQUATION

The generator is trained to minimize

W (P θ REAL , P θ ). Denote F (w, θ) = w (E x∼N (0,I d ) x - (E y∼N (0,I d ) y + θ)).

When we use stochastic gradient descent, the training procedure can be de-scribed as

EQUATION

where η t and µ t are the step size. We present our result for gradient flow (Du et al., 2018a; b) , i.e., gradient descent with infinitesimal time interval, whose behaviour can be described by the following differential equations:

EQUATION

(A detailed explanation of gradient flow and stochastic gradient flow is in Appendix A.) We will show that when the WGAN in (3) is trained using stochastic gradient flow, the batchsize will impact the behaviour of the training dynamics. That is, compared with large batchsize (or even full batch), when the batchsize is small, the training dynamics of WGAN suffer from a large variance, thus more unstable. Theorem 1 tells us that when the WGAN model is trained using constant step size stochastic gradient flow, then the variance of the output will increase as t grows, and is of order Θ( 1 m ). Namely, the variance will be large if the batchsize is small. Theorem 1. [Variance of WGAN, constant step size] Denote [θ t ] i as the i-th component of the vector θ t . Suppose we train the WGAN model in (3) using constant step size stochastic gradient flow with batchsize m, then the parameter of the generator θ t satisfies

EQUATION

where

EQUATION

and e sA Σ d+i is the (d + i)-th row of the matrix e sA Σ ,

The proof is in Appendix B. The main idea is that due to the special dynamics of optimization of mini-max problems, the bias caused by the randomness of the batch sampling in each epoch will accumulate, which will lead to large variation after many steps of training. Note that although in traditional optimization problems the variance of SGD caused by the randomness of samples will not affect the convergence (Bubeck et al., 2015; Brutzkus et al., 2017) , the variance will damage the convergence properties in GANs.

Next we consider the vanishing step size case, in which η t = µ t = 1/t (without loss of generosity assume t ≥ 1. This step size is commonly used in convex optimization in practical problems (Bubeck et al., 2015) . Theorem 2 shows that the problem still exists in such case, whose proof is in Appendix C. Theorem 2. [Variance of WGAN, vanishing step size] Suppose we train the WGAN model in (3) using 1/t step size stochastic gradient flow with batchsize m, then the parameter of the generator θ t satisfies

EQUATION

Remark 1. Here we consider only the WGAN because it is usually more stable than typical GANs. We believe that other forms of GANs will have a similar problem caused by the insufficient batchsize. Remark 2. Although our theoretical analysis only considers the effect of noise in distribution, the same reason also applies to the effect of other irrelevant features especially when the generator cannot learn the real data distribution perfectly. 

4.1. PIXEL-WISE COMBINATION IN THE GENERAL TRAINING OF GANS:

When GANs are trained on image datasets, we find that under certain situation pixel-wise combinations (e.g. pixel-wise average or pixel-wise logical-and) of the real samples could fool the discriminator, although the generated combinations could have inconsistent properties. e.g. the pixel-wise average of two animal images in CIFAR10 could be unrecognizable for human. Take a simple case for illustration, suppose the discriminator is a linear classifier, given two real images, the pixel-wise average of those two real images are likely to be predict as positive sample by the linear discriminator. Since those pixel-wise combinations are recognized as real by the discriminator, the generator correspondingly tends to generate them. Therefore, the generated images could be different from the expected ones and the generalization of GANs becomes anomalous.

4.2. ILLUSTRATION ON TOY AND REAL DATASETS:

We demonstrate the problem of pixel-wise combination, which leads GANs to anomalous generalization, by a toy dataset. reference, the generator is fooled further to generate unexpected samples. As a result, the learned data distribution could differ a lot from the target data distribution.

Beyond the toy dataset, the problem of pixel-wise combination does exist in the training of GANs in practice. For the natural image dataset CELEB-A, some generated images are exactly the same as the pixel-wise averages of the training data (Figure 3 bottom). Also, the pixel-wise average of certain structurally similar images could generate realistic samples (Figure 4 ). In brief, the experiments show that the problem of pixel-wise combination exists for GANs and makes it hard to model the target data distribution faithfully, which leads to anomalous generalization behaviour.

4.3. THEORETICAL ANALYSIS:

In this section we give a possible explanation to the problem of pixel-wise combination by a theoretical analysis. Note that this phenomenon is most remarkable on the geometric or the facial datasets.

The samples in those datasets are structurally similar, namely, for a facial dataset, the eyes, noses and other features of the human faces appear at fixed positions in the images with high probability. So we assume that the l 2 distance between most positive training samples in the dataset is small.

For the discriminator f (x), without loss of generosity, assume that f (x) > 0 if the sample x is classified as real. We make the following assumptions. Assumption 1. Assume that f is L-Lipschitz. Assumption 2. Assume that the discriminator classifies all the positive training samples as real with a large margin (i.e. there exists > 0 such that for all the positive training samples x, f (x) > ).

Most classifiers satisfy the Lipschitz condition (for example, the softmax classifier). Assumption 2 means that the discriminator classifies the positive training samples as real with high confidence.

Our theorem shows that under these assumptions, the pixel-wise convex combination of any two positive training samples will be classified as real with high probability. The proof is in Appendix D. Theorem 3. If the discriminator f (x) satisfies Assumption 1-2. Then for any two samples x 1 and x 2 in the training dataset satisfying x 1 -x 2 2 ≤ δ and any λ ∈ (0, 1), we have

f (λx 1 + (1 -λ)x 2 ) ≥ max{f (x 1 ) -L(1 -λ)δ, f (x 2 ) -Lλδ} (11) Moreover, if > Lδ min{λ, 1 -λ}, then f (λx 1 + (1 -λ)x 2 ) > 0.

5. FIXING THE ANOMALOUS GENERALIZATION OF GANS:

In this section, we propose novel methods to mitigate the two problems. We present the Pixel-wise Combination Regularization (PCR) to mitigate the problem of pixel-wise combination in Section 5.1. For the problem of sample insufficiency, we present the Sample Correction (SC) in Section 5.1. The results show that the anomalous generalization for the geometric dataset is avoided entirely (Section 5.2). For the natural image dataset, the training modifications could improve the FID score up to 30% (Section 5.3).

5.1. APPROACH:

Pixel-wise Combination Regularization For the training of vanilla GANs, the positive training samples for the update of the discriminator come from the training dataset and the negative training samples are generated by the generator. Since we think that the discriminator tends to recognize the pixel-wise combinations of the images in the training dataset as real even though they are not in the target distribution, we define a dataset:

D com = {x 0 , x 1 , ...x n-1 |x k = y i ⊕ y j 2 y i , y j ∈ D training i = j} (12)

and use the images in D com as additional negative training samples to restrict this tendency. The ⊕ in Eqn. 12 is the pixel-wise combination operation, it could be the pixel-wise average or pixelwise logical-and/or. The samples in D com are the combinations of every two images in the training dataset. The loss term for training with the Pixel-wise Combination Regularization can be written as:

L = E x∼Pr [f 1 (D(x))] positive samples + 1 2 E x∼Pg [f 2 (D(x))] + E x∼Pcom [f 2 (D(x))]

negative samples (13) where P r , P g and P com are the data distributions approximated by the training data, the generator and D com . In this way, the tendency to generate those combination images is restricted. We refer to this addition of the negative training samples for the training of the discriminator as Pixel-wise Combination Regularization (PCR).

Sample Correction:

We introduce a general framework to mitigate the problem of sample insufficiency. We assume that to model the target distribution, the discriminator is required to separate accurately the real samples in the target data distribution from the others not in it, which the sample insufficiency makes it difficult to achieve. For that goal, the realistic samples in the negative training batch are not useful. Intuitively, if the realistic samples appear in both the positive and the negative training batches, it would be ambiguous for the discriminator to learn the correct separation function. Therefore, we replace the realistic samples in the negative training batch with less realistic ones by a certain pre-defined measure of reality. In this way, the discriminator could efficiently learn an accurate separation function with limited batchsize. The Sample Correction approach is a general framework and the measure of reality could differ for different datasets. We present our experiments on the geometry and the CELEB-A datasets as examples in the next subsections.

5.2. EXPERIMENTAL RESULTS ON GEOMETRIC DATASET:

As shown before, caused by the two problems, when trained on a geometry dataset where all the training images have exactly two rectangles, there would be a lot of anomalous generated samples with different number of rectangles. We use the two proposed methods to mitigate this anomalous generalization. We do experiments to verify the effects of our methods. The training dataset consists of 25600 binary images, all of which have exactly two rectangles. For the Pixel-wise Combination Regularization (PCR), the pixel-wise logical-and/or of the positive training images are precomputed (details in Appendix F). They are used as additional negative training samples. For the Sample Correction (SC), the generated realistic samples in the negative training batch of the discriminator are discarded. The realistic samples are those with exactly two rectangles, the same as the positive training samples.

As shown in Figure 1 (right), compared to the vanilla approach, the SC approach (SC) almost eliminates the anomalous generalization and the proportion of correct images (rectangle number is 2) goes up to 97%. The Pixel-wise Combination Regularization (PCR) approach also improves the proportion but is stuck at 70%. We think this is caused by the still existence of the problem of sample insufciency. Combining these two methods, the SC+PCR approach converges to 99%, much more quickly than other approaches, showing the existences of the two problems and the effects of our methods. (Gulrajani et al., 2017; Mao et al., 2017; Zhang et al., 2018) . Three natural image datasets are involved: CIFAR10, CELEB-A and M-IMAGENET (Liu et al., Krizhevsky & Hinton, 2009) . M-IMAGETNET is the validation set of the IMAGENET dataset. We train the network unsupervisedly with the Adam optimizer (α = .0002, β 1 = .5, β 2 = .9). As shown in Table 1 , the performances improve in most cases after applying the Pixel-wise Combination Regularization. The FID scores of the baselines are consistent with that reported in Lucic et al. (2018) . For WGANGP trained on CIFAR10, the achieved best FID score improves up to 30%, showing the potentials of our regularization method. Interestingly, the improvements are more remarkable for the CIFAR10 and M-IMAGENET than the CELEB-A dataset. We hypothesize this is because the objects in CELEB-A tend to appear at regular or fixed positions. Therefore, the average of real images is likely to give a data point in the target data manifold. For example, it is very possible that the average of two human face images is still a realistic human face image (data in CELEB-A). But it is less possible for the average of images of a car and a dog to be a realistic image (data in CIFAR10). For the Sample Correction method. We randomly select a small portion of the images as training data (M-CELEB-A and M-CIFAR). Dataset size is kept small (e.g. 32) so that the learned data distribution could approximate the target data distribution well. The measure of reality of a sample in this case is the normalized minimum L 2 distance of the sample to all the training data (DIF, between 0 and 1). We train GANs in two different ways, namely Vanilla and Sample Correction. The latter approach differs in that the realistic samples in the negative batch (DIF less than 0.1) are replaced with less realistic ones. The batchsize is kept small to better demonstrate the problem of sample insufficiency (e.g. 2). As shown in Figure 5 , the Sample Correction approach outperforms the vanilla one by a large margin. FID and DIF scores are both better during training. This is because for the Sample Correction approach, the problem of sample insufficiency is restricted and the separation function of the discriminator is more accurate. The improvements can also be found visually (Figure 5 left). For the vanilla approach, some generated samples degenerate to noises. But the Sample Correction approach can generate real ones with authentic details.

6. CONCLUSION AND FUTURE WORK:

In this paper, we discuss two specific problems of GANs, namely sample insufficiency and pixelwise combination. We demonstrate that they make it difficult to model the target distribution and lead GANs to anomalous generalization. Specific methods are introduced to prevent them from misleading the generalization of GANs, which improve the performance. We hope the two specific problems and the methods to restrict them can help the future work to better understand the generalization behaviour of GANs.

A AN EXPLANATION OF GRADIENT FLOW AND STOCHASTIC GRADIENT FLOW Gradient flows, or steepest descent curves, are a very classical topic in evolution equations: given a functional F defined on R d , and we want to look for points x minimizing F (which is related to the statical equation ∇F (x) = 0). To do this, we look, given an initial point x 0 , for a curve starting at x 0 and trying to minimize F as fast as possible. Since the negative gradient direction is the steepest descent direction, we will solve equations of the form

EQUATION

On the curve of the solution of this equation (t, x(t)), when t increases, at every point x(t) the point goes along the negative gradient direction.

If we write down the discrete form of ( 14), it becomes

EQUATION

which is the Euler method of the differential equation. And if we take δt = 1, we get the expression of gradient descent. So we can view the gradient flow as gradient descent of infinitesimal time interval.

And if we add a stepsize term in the equation, it becomes

EQUATION

where η t denotes the step size. And in the discrete form, it falls into the familiar gradient descent formula with step size {η t }:

EQUATION

In most machine learning problems, the function F can be written as

F (x) = 1 n n i=1 f i (x)

, and compute the gradient of F exactly can be computationally exhaustive. So instead we often use an approximation of ∇F (for example, we can randomly select i ∈ {1, ..., n} and use ∇f i to approximate ∇F ). This can be described formally as follows. If ∇F (x(t)) = g(x(t)) + EY , where Y is a random vector with distribution p(y), then we can sample Y t ∼ p and update x as

EQUATION

Especially, if p is a Gaussian distribution with mean 0 and covariance matrix I d , then ( 18) is equivalent to x(t + 1) -x(t) = -η t (g(x(t)) + (W (t + 1) -W (t))) (19) where W(t) is a Wiener process (Wiener, 1923) . And this is the Euler-Maruyama scheme (Kloeden & Platen, 2013) of the following stochastic differential equation:

dX t = -η t g(X t )dt -η t dW t X 0 = x 0 (20)

the solution, if exists, is a stochastic process {X t = X(t, x 0 )}. We call this solution the stochastic gradient flow of F at point x 0 . And when we say that F is trained using stochastic gradient flow, we mean that the curve {x t } is a sample path of {X t = X(t, x 0 )}.

Finally, when we say that F is trained using stochastic gradient descent with batchsize m, we mean that for each t, we sample Y 1 t , ..., Y m t ∼ p and update x as

EQUATION

This is the Euler-Maruyama scheme of the following stochastic differential equation:

EQUATION

And when we say that F is trained using stochastic gradient flow with batchsize m, we mean that the curve {x t } is a sample path of {X m t = X m (t, x 0 )}.

B PROOF OF THEOREM 1:

Proof. First we give a detailed description of the training dynamics of our model in Section 3.3 using gradient flow.

The framework of training the WGAN in Section 3.3 is as follows. Denote F (w, θ) = w (E x∼N (0,I d ) x -(E y∼N (0,I d ) y + θ)). For each epoch t = 1, 2, ..., given the parameter of the generator θ t , the target of the discriminator is

EQUATION

The gradient of w is

EQUATION

And for one-step gradient upscent the update of w is

EQUATION

where η t is the step size.

After w t is updated to w t+1 , given the parameter of the discriminator w t+1 , the target of the generator becomes

EQUATION

And the gradient of θ is ∇ θ F (w t+1 , θ) = -w t+1 .

(27) For one-step gradient descent the update of θ is

EQUATION

Now we consider the case when η t and µ t are constants, η t = µ t = C. Without loss of generality assume C = 1. And when the time interval is infinitesimal, the discrete dynamics converge to the gradient flow with constant step size, which can be written in the form of ordinary differential equations (ODEs):

EQUATION

Under the full-batch condition, which means that E x∼N (0,I d ) x = 0 and E y∼N (0,I d ) y = 0 can be precisely calculated, the solution to the above ODEs is

w t = w 0 cos t -θ 0 sin t θ t = θ 0 cos t + w 0 sin t (31)

Notice that the solution implies that w t ≤ w 0 + θ 0 , which means that the 1-Lipschitz condition on w in WGAN is automatically satisfied if w 0 and θ 0 are initialized sufficiently small. The parameters lie on a circle around the equilibrium point (0, 0), which are stable though do not converge.

When the batchsize is m, which means that we randomly draw i.i. where e sA Σ i denotes the i-th row of e sA Σ.

Since

e sA Σ i 2 2 ≥ 0, for all t 1 ≤ t 2 we have Var [X(t 1 )|X(0)] ii ≤ Var [X(t 2 )|X(0)] ii .

So the variance of the OU process will increase as t grows. And because the elements in Σ is of order 

Θ( 1 √ m ), we have Var [X(t)|X(0)] = Θ( 1 m ). From the definition of X(t) we have Var([θ t ] i ) = Var[X(t)|X(0)] d+i,d+i = t 0 e sA Σ d+i

C PROOF OF THEOREM 2:

Proof. Now we consider the vanishing step size situation, namely η t = µ t = 1/t. And when the time interval is infinitesimal, the discrete dynamics converge to the gradient flow which can be written in the form of ordinary differential equations (assume t ≥ 1):

EQUATION

Under the full-batch condition, which means that E x∼N (0,I d ) x = 0 and E y∼N (0,I d ) y = 0 can be precisely calculated, the solution to the above ODEs is

EQUATION

Notice that the solution implies that w t ≤ w 1 + θ 1 , which means that the 1-Lipschitz condition on w in WGAN is automatically satisfied if w 0 and θ 0 are initialized sufficiently small. The parameters lie on a circle around the equilibrium point (0, 0), which are stable though do not converge.

When the batchsize is m, which means that we randomly draw i.i. The samples with true rectangle number is discarded. The selection can be implemented efficiently by counting the number of rectangle using straight forward counting algorithm. For PCR approach, the pixel-wise averages of the training data (pixel-wise logic-and and logic-union) are precomputed. Specifically, the way we generate training geometry data can be utilized for this precomputation. To generate 2-number-rectangle training geometry data, we first randomly generate several 3-numberrectangle images. After that, for each 3-number-rectangle image, we randomly remove one rectangle out of the three. The remove is done twice for each 3-number-rectangle image. By this we get two different 2-number-rectangle images out of one 3-number-rectangle image. Because of this construction method, we could obtain the pixel-wisely average images easily, i.e. pixel-wise logic-or or logic-and, which are 3-number-rectangle or 1-number-rectangle respectively. These precomputed additional images are used as additional negative training samples for the PCR approach. All images are 32 by 32 with one channel. All rectangles are 8 by 8.

In experiments, SC and PCR can both improve the proportion of correct generated images. Combining the SC and PCR, the proportion goes to 100% quickly. We show there is no mode collapses for these three training modifications: SC, PCR and SC+PCR. We randomly draw 30 training images. 

:

PIXEL-WISE COMBINATIONIn this section, We first discuss the problem of pixel-wise combination in Section 4.1, followed by the illustration on toy and real datasets in Section 4.2. Finally, we provide a theoretical analysis in Section 4.3.

Figures and Tables:

Figure 1: Left: The generated images have different rectangle numbers (e.g. one, two or three), while the rectangle numbers of all the training data are exactly two (the anomalous marked red). Right: The proportion of the correct generated images (rectangle number is two) for different training approaches. The training dataset consists of 25600 images, all of which have exactly two rectangles.

Figure 2: Middle: The loss and gradients for MGD/FGD. Training of MGD is unstable while FGD converges quickly. Left: The generated samples of the generator by four latent codes during training. MGD is unstable and anomalous images are generated (z M GD i , rectangle number is not two). FGD is stable and converges (z F GD i ). Right: The FID scores during training for different batchsizes (on CELEB-A). Larger batchsize is better after trained with the same amount of data.

Figure 3: Top: For the geometry dataset, the proportion curve of the correct generated images (number of rectangle is three) is plotted (upper right solid line). The scores of the discriminator for the positive training data and the pixel-wise logical-and/or of the positive training data are also plotted respectively (upper right dash line). During training, the generated images which are pixelwise logical-and/or of the positive training samples prevail (highlighted with colored frame on upper left). They get high scores similar to the positive training images in the earlier stage. Bottom: Certain generated images are shown for human face generation(Liu et al., 2015). They are exactly the same as the pixel-wise averages of the training data.

Figure 4: Generated images of the generative average method (GAM) and GANs. The performances are comparable both visually and by the FID score.

Figure 5: Left: The final generated samples for the M-CIFAR10 and M-CELEB-A, where the Sample Correction (SC) approach achieves better performance. Right: The FID and DIF curves during training for the vanilla and the Sample Correction (SC) approaches.

ds. Hence we complete our proof.

Figure 6: The generated images for certain fixed latent codes. For FGD (left), training is stable and converges smoothly. For MGD, where the insufficient problem is severe, training is unstable and gives rise to anomalous images (rectangle number is not two).

For each sample, we find the closest image in 256000 generated samples. Results are shown in Figure 7. For three training designs extended from the Vanilla (SC, PCR and SC+PCR), most training images are represented by the learned distribution of the generator, meaning the high performance is achieved (up to 99% correct generated images) without mode collapse.

Figure 7: For a random training image, the closest generated image in 256000 generated samples is found. For three training method (SC, PCR and SC+PCR), most training images are represented by the learned distribution of the generator, meaning the high performance is achieved (up to 99% correct generated images) without mode collapse.

The achieved best FID scores of three runs are reported after 50000 steps for different models and datasets. In most cases, the FID score improves after applying the Pixel-wise Combination Regularization (PCR). Positive improvement rates are highlighted in bold. The baseline scores are consistent with that reported inLucic et al. (2018).

Table content: <table><tr><td colspan="2">Vanilla/PCR/boost</td><td/><td>Model</td></tr><tr><td/><td/><td>WGANGP</td><td>LSGAN</td><td>SAGAN</td></tr><tr><td/><td>CELEB-A</td><td>20.9 / 21.7 / -3.4%</td><td>17.7 / 16.0 / 9.7%</td><td>28.0 / 24.3 / 13.2%</td></tr><tr><td>Dataset</td><td>CIFAR10</td><td colspan="3">45.4 / 31.6 / 30.4% 57.6 / 51.0 / 11.4% 39.6 / 33.7 / 14.7%</td></tr></table>

to estimate the true mean E x∼N (0,I d ) x and E y∼N (0,I d ) y, the gradients become ∇ w F = xm,tȳm,t -θ and ∇ θ F = -w. Since x 1 t , ..., x m t and y 1 t , ..., y m t are independent for different t, we have xm,t , ȳm,t ∼ N (0, 1 And the parameters w and θ satisfy the following stochastic differential equations:dw t = -θ t dt + 2 m dW t dθ t = w t dt (32)where W t is a d -dimensional standard Wiener process. Denote X t = (w t , θ t ) ∈ R 2d , then X t is a multidimensional OU process(Uhlenbeck & Ornstein, 1930) with expectation

Table content: <table><tr><td/><td/><td/><td colspan="3">E [X(t)|X(0)] = e tA X(0)</td><td>(33)</td></tr><tr><td colspan="2">and variance</td><td/><td/><td>t</td></tr><tr><td/><td colspan="3">Var [X(t)|X(0)] =</td><td colspan="2">e tA e -sA ΣΣ e tA e -sA ds</td><td>(34)</td></tr><tr><td/><td/><td/><td/><td>0</td></tr><tr><td>where</td><td/><td/><td/><td/></tr><tr><td/><td/><td>A =</td><td colspan="2">0 -I d I d 0</td><td>, Σ =</td><td>2 m I d 0 0 0</td><td>(35)</td></tr><tr><td>So e tA =</td><td colspan="2">(cos t)I d -(sin t)I d (sin t)I d (cos t)I d</td><td>and</td><td/></tr><tr><td/><td>e tA e -sA =</td><td colspan="4">(cos t)I d -(sin t)I d (sin t)I d (cos t)I d</td><td>(cos -s)I d -(sin -s)I d (sin -s)I d (cos -s)I d</td><td>(36)</td></tr><tr><td/><td>=</td><td colspan="4">(cos (t -s))I d -(sin (t -s))I d (sin (t -s))I d (cos (t -s))I d</td><td>= e (t-s)A</td><td>(37)</td></tr><tr><td colspan="3">So the variance can be written as</td><td/><td/></tr><tr><td/><td colspan="3">Var [X(t)|X(0)] =</td><td/></tr><tr><td/><td/><td/><td/><td/><td>(40)</td></tr><tr><td/><td/><td/><td>t</td><td/></tr><tr><td/><td/><td/><td>=</td><td colspan="2">e sA ΣΣ e sA ds</td><td>(41)</td></tr><tr><td/><td/><td/><td>0</td><td/></tr><tr><td colspan="6">And the variance of the i-th component of X(t) is</td></tr><tr><td/><td/><td/><td/><td/><td>t</td></tr><tr><td/><td colspan="4">Var [X(t)|X(0)] ii =</td><td>0</td><td>e sA Σ i e sA Σ i ds</td><td>(42)</td></tr><tr><td/><td/><td/><td/><td/><td>t</td></tr><tr><td/><td/><td/><td/><td>=</td><td>0</td><td>e sA Σ i</td><td>2 2 ds</td><td>(43)</td></tr><tr><td colspan="6">and y 1 t , ..., y m t ∼ N (0, I d ) and use the sample mean xm,t = 1 m</td><td>d. samples x 1 t , ..., x m t ∼ N (0, I d ) m i=1 x i t and ȳm,t = 1 m m i=1 y i t</td></tr></table>

to estimate the true mean E x∼N (0,I d ) x and E y∼N (0,I d ) y, the gradients become ∇ w F = xm,tȳm,t -θ and ∇ θ F = -w. Since x 1 t , ..., x m t and y 1 t , ..., y m t are independent for different t, we have xm,t , ȳm,t ∼ N (0, 1 m I d ) and they are independent for different t. So xm,t -ȳm,t ∼ N (0, 2 m I d ). And the parameters w and θ satisfy the following stochastic differential equations: dw t = -1 t θ t dt + 1 So e Ã(t) = (cos ln t)I d -(sin ln t)I d (sin ln t)I d (cos ln t)I d and e Ã(t) e -Ã(s) = (cos ln t)I d -(sin ln t)I d (sin ln t)I d (cos ln t)I d (cos -ln s)I d -(sin -ln s)I d (sin -ln s)I d (cos -ln s)I d

Table content: <table><tr><td colspan="7">and y 1 t , ..., y m t ∼ N (0, I d ) and use the sample mean xm,t = 1 m</td><td>d. samples x 1 t , ..., x m t ∼ N (0, I d ) m i=1 x i t and ȳm,t = 1 m m i=1 y i t</td></tr><tr><td/><td/><td colspan="2">t w t dt dθ t = 1</td><td>t</td><td colspan="2">2 m dW t</td><td>(47)</td></tr><tr><td/><td/><td/><td/><td/><td/><td>(48)</td></tr><tr><td>and variance</td><td/><td/><td/><td/><td/></tr><tr><td colspan="3">Var [X(t)|X (1)] =</td><td/><td/><td/><td>(49)</td></tr><tr><td>where</td><td/><td/><td/><td/><td/></tr><tr><td/><td>A(t) =</td><td>0 t I d 1</td><td>-1 t I d 0</td><td colspan="2">, Σ(t) =</td><td>√ √ mt I d 0 2 0 0</td><td>(50)</td></tr><tr><td/><td colspan="3">Ã(t) = A(t) =</td><td>0 (ln t)I d</td><td/><td>-(ln t)I d 0</td><td>(51)</td></tr><tr><td/><td/><td/><td/><td/><td/><td>(52)</td></tr><tr><td>=</td><td colspan="4">(cos ln t s )I d -(sin ln t s )I d (sin ln t s )I d (cos ln t s )I d</td><td/><td>(53)</td></tr><tr><td>=</td><td colspan="4">(cos ln s t )I d (sin ln s t )I d -(sin ln s t )I d (cos ln s t )I d</td><td/><td>(54)</td></tr><tr><td colspan="2">=e Ã( s t )</td><td/><td/><td/><td/><td>(55)</td></tr></table>

Peer review (iclr2020):
This paper tries to understand and tackle what they call the anomalous generalization of GANs. They first show that GANs suffer from instability because of sample insufficiency, they show experimentally that when decreasing the batch-size on both a toy dataset and CelebA that the training become unstable and the performance drops, leading to the generation of outliers, they also show theoretically in a simplified setting that small batch-size can lead to instability. They then show that the anomalous generalization can also come from the discriminator not being able to discriminate pixel-wise combination of the real samples. They show on some toy dataset and CelebA that some of the generated samples are a pixel-wise combination of two training examples and that those combination can fool the discriminator, they also show provide a theorem confirming this observation. Finally they provide two methods to mitigate the previous issues: 1) They propose to add a loss term for the discriminator and force it to also learn to discriminate pixel-wise combinations of images. 2) They propose to remove the most realistic generated samples from the mini-batch and replace them with less realistic samples Overall I find the observations of the paper interesting and worth investigating. However I find the paper a bit weak especially about sample efficiency. Indeed the instability of GANs and how it relates to batch-size as already been studied, and the theoretical results provided are quite limited. Furthermore I*m not convinced by the proposed trick to fix that issue. Thus I*m in favour of rejecting this paper. Main argument: - I think the results in section 3.3 should be put into context and mention related work. First of all this bilinear example as already been studied in the literature, in the full-batch and stochastic setting. In full batch it was already known that simultaneous gradient descent doesn*t converge see for example [1]. In the stochastic setting it was also studied and an algorithm was proposed to fix that see [2]. I think this related work and other need to be mentioned in this section and the results need to be contrasted with existing results. In the current state it*s not clear if this theorems bring anything novel. - I have also some concerns about section 4.3. At the beginning of training, it*s clear that it*s probably easy for the discriminator to classify real samples with a large margin, but as the training progress it becomes harder and harder for the discriminator to classify real samples so it*s not clear to me that assumption 2 would hold. Can you discuss it ? it would be interesting to actually check experimentally how that assumption holds. - I*m also not convinced by the Sample Correction trick proposed. - PCR really seems to improve the performance, it would be nice to run all the methods with several seeds and also show the standard deviation. - Why is SC only computed on a really small subset of CelebA and CIFAR10 ? why not try on the full dataset ? It makes the conclusion not very convincing. Minor comment: - Figure 1 (right): what are the axis ? in particular does the axis represent the number of training iterations ? If so what is number of generated samples used to compute the proportion of correct generated samples. Also you don*t measure if there is any trade-off between quality and mode dropping. Could you measure the number of different samples that are generated to have some measure of mode dropping ? - For the toy task how do you count the number of rectangles ? - Can you define what does DIF stands for and how it is computed ? References: [1] Gidel, G., et al. *A variational inequality perspective on generative adversarial networks.* arXiv (2018). [2] Chavdarova, T., et al. *Reducing noise in gan training with variance reduced extragradient.* arXiv (2019).