Title: IS THERE MODE COLLAPSE? A CASE STUDY ON FACE GENERATION AND ITS BLACK-BOX CALIBRATION

Abstract: Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism. One concern raised is whether the state-of-the-art GAN's learned distribution still suffers from mode collapse. Existing evaluation metrics for image synthesis focus on low-level perceptual quality. Diversity tests of samples from GANs are usually conducted qualitatively on a small scale. In this work, we devise a set of statistical tools, that are broadly applicable to quantitatively measuring the mode collapse of GANs. Strikingly, we consistently observe strong mode collapse on several state-of-the-art GANs using our toolset. We analyze possible causes, and for the first time present two simple yet effective "black-box" methods to calibrate the GAN learned distribution, without accessing either model parameters or the original training data.

1. INTRODUCTION:

Generative adversarial networks (GANs) (Goodfellow et al., 2014) have demonstrated unprecedented power for various image generation tasks. However, GANs have also been suffering from generation bias and/or loss of diversity. The underlying reasons could be compound, ranging from the data imbalance to the training difficulty of GANs, and more:

• First of all, the training data for GANs, especially for the typical unconditional/unsupervised generation tasks (Karras et al., 2017; 2018) , might possess various subject or attribute imbalances. As a result, GANs trained with them might be further biased towards the denser areas, similarly to the classifier bias towards the majority class in imbalanced classification. • More intrinsically, even when the training dataset "looks" balanced, training GANs is notoriously more unstable (sometimes even uncontrollable) than training classifiers, potentially constituting another source of mode collapse. One most common hurdle of GANs is the loss of diversity due to mode collapse (Goodfellow, 2016) , wherein the generator concentrates too large a probability mass on a few modes of the true distribution. Another widely reported issue, known as co-variate shift (Santurkar et al., 2017) , could be viewed as a nuanced version of mode collapse.

This paper seeks to explore: do the state-of-the-art GANs still suffer from mode collapse? Can we have a toolkit to detect that? And if the mode collapse happens, is there any "easy and quick" remedy for calibrating the GAN's learned distribution to alleviate the mode collapse?

Evaluation of Mode Collapse There are several popular metrics for GAN evaluation, e.g. Inception Score (IS) (Salimans et al., 2016) , Fréchet Inception Distance (FID) (Heusel et al., 2017) , MODE (Che et al., 2016) and birthday paradox based diversity test (Arora & Zhang, 2017) . IS, FID and MODE score takes both visual fidelity and diversity into account. Birthday paradox based diversity test gives a rough estimation of support size under the assumption of uniform sampling. Recently, a classification-based metric (Santurkar et al., 2017) was proposed for a quantitative assessment of the mode distribution learned by GANs. However, their approach hinge on a classifier trained on the original (balanced) GAN training set, with class labels known, available and well-defined (e.g., object classes in CIFAR-10, or face gender in CelebA), making it non-straightforward to extend to data subjects where classes are hard to be defined, and/or are not enumerable (e.g, open set problems).

To tackle this problem, we propose a hypothesis test method by analyzing the clustering pattern of samples. We exploit a statistical tool from spatial analysis, called Ripley's K function, to quantitatively measure the mode collapse. We demonstrate the application of our tool set in analyzing the bias in unconditional face image generation: a popular benchmark task nowadays for GANs, yet remaining rather unclear how to measure its mode collapse using existing tools since every generated identity is expected to be new. The study of face identity generation bias has profound practical values for understanding facial privacy (Filipovych et al., 2011) and fairness (Holstein et al., 2018) . Using our tools, we find the mode collapse still a prevailing problem in state-of-the-art face generation GANs (Karras et al., 2018; 2017) , and further analyze several possible causes.

Calibration Approaches on GAN Many approaches have been proposed to alleviate mode collapse problem, ranging from better optimization objectives (Arjovsky et al., 2017; Mao et al., 2017) , to specialized builing blocks (Durugkar et al., 2016; Ghosh et al., 2018; Liu & Tuzel, 2016) . However, they require either tedious (re-)training, or at least the access to training data, as well as to model parameters: we refer to the existing methods as white-box approaches.

In contrast, we are interested in an almost unexplored aspect: assuming some generation bias is known, how can be calibrate the GAN models are often observed to suffer from the mode collapse problem (Salimans et al., 2016) ; (Sutskever et al., 2015) , where only small modes subsets of distribution are characterized by the generator. The problem is especially prevalent for high-dimensional data, e.g. face image generation, where the training samples are low-density w.r.t. the high-dimensional feature space. Salimans et al. (2016) presented the popular metric of Inception Score (IS) to measure the individual sample quality. IS does not directly reflect the population-level generation quality, e.g., the overfitting and loss of diversity. It also requires pre-trained perceptual models on ImageNet or other specific datasets (Barratt & Sharma, 2018) . Heusel et al. (2017) propose the Fréchet Inception Distance (FID), which models the distribution of image features as multivariate Gaussian distribution and computes the distance between the distribution of real images and the distribution of fakes images. Unlike IS, FID can detect intra-class mode dropping. However, the multivariate Gaussian distribution assumption hardly holds very well on real images, and low FID score cannot rule out the possibility of the generator's simply copying the training data. Besides the two most popular metrics, (Che et al., 2016) develop an assessment for both visual quality and variety of samples, known as the MODE score and later shown to be similar to IS (Zhou et al., 2017) . (Arora et al., 2018) and (Arora & Zhang, 2017) proposed a test based upon the birthday paradox for estimating the support size of the generated distribution. Although the test can detect severe cases of mode collapse, it falls short in measuring how well a generator captures the true data distribution. It also heavily relies on human annotation, making it challenging to scale up to larger-scale evaluation. (Santurkar et al., 2017 ) took a classification-based perspective and view loss of diversity as a form of covariate shift. As we discussed above, their approach cannot be straightforwardly extended to data subjects without pre-known and closed-set class definition, in addition to the need of training an extra classifier on the original labeled training set.

2.2. MODEL CALIBRATION APPROACHES OF GANS:

There are many efforts to address the mode collapse problem in GANs. Some focus on discriminators by introducing different divergence metrics (Metz et al., 2016) and optimization losses (Arjovsky et al., 2017; Mao et al., 2017) . The minibatch discrimination scheme allows the discriminator to discriminate between whole mini-batches of samples instead of between individual samples. (Durugkar et al., 2016) adopted multiple discriminators to alleviate mode collapse. ModeGAN (Che et al., 2016) and VEEGAN (Srivastava et al., 2017) enforce the bijection mapping between the input noise vectors and generated images with additional encoder networks. Multiple generators (Ghosh et al., 2018) and weight-sharing generators (Liu & Tuzel, 2016) are developed to capture more modes of the distribution. However, these approaches are designed to easily calibrating trained GANs.

A handful of existing works attempt to combine GANs with sampling methods to improve generation quality. (Turner et al., 2018) introduced the Metropolis-Hastings generative adversarial network (MH-GAN). The MH-GAN uses the learned discriminator from GAN training to build a wrapper for the generator for improved sampling, at the generation inference stage. With a perfect discriminator, the wrapped generator can sample from the true distribution exactly even with a deficient generator. (Azadi et al., 2018) proposed discriminator rejection sampling (DRS) for GANs, which performs rejection sampling on the outputs of the generator by using the probabilities given by the discriminator, to approximately correct errors in the generator's distribution. Yet still, these approaches are white-box calibration since both require access to trained discriminators (which might be even less available/accessible than the generator after a GAN is trained).

3. METHOD:

We intend to study the bias of the most representative features of the generated faces, i.e.the face identity distribution, since almost all face attributes can be derived based on this representations. To detect face identity collapse, we are aiming to detect high-density regions in features space caused by any possible attribute non-diversified. Or, if being slightly imprecise in terms, (Santurkar et al., 2017) examined the marginalized distribution through some discrete categorical attributes' lens, while ours looks at the joint distribution of all possible attributes in the continuous feature space holistically. 

I S i N I S i ← N (I S i , S\I S i , d 0 ) R obs ← { ĨS 1 , • • • , ĨS p }

// Find the region for observation by selecting the top p face images in S with largest

N I S i R ref ← { ÎS 1 , • • • , ÎS q }

// Find the region for reference by randomly selecting q face images from S

T ← {I T 1 , • • • , I T M } // Randomly sampled M face images (M m) for each d in [d b , d e ] ds do for each ĨS i ∈ R obs do // Count neighbors within d distance for each ĨS i in R obs N d ĨS i ← N ( ĨS i , T , d) for each ÎS i ∈ R ref do // Count neighbors within d distance for each ÎS i in R ref N d ÎS i ← N ( ÎS i , T , d) Compute the pointwise confidence regions of [N d ÎS i | 1-α 2 , N d ÎS i | α 2 ] for each d ∈ [d b , d e ]

ds , at confidence level of α (default 0.05). The intervals between the upper and lower confidence bounds for all samples in R ref define the confidence band (Eubank & Speckman, 1993) .

Reject the hypothesis that the clustering pattern of R obs is the same as that of R ref , if the curve of N d ĨS i falls outside of the confidence band.

Given an unconditional face generator G and an identity descriptor f id , we sample images I = G(z) using a random distribution z∼N (0, Σ). The unit vector f id (I) describes the identity feature in the face embedding space. The normalized cosine distance between image I 0 and I 1 is defined as:

EQUATION

For a given anchor face image I 0 , a distance threshold d 0 and a collection of randomly sampled face images S, the neighboring function N (I 0 , S, d 0 ) is defined to compute the number of neighbors within d 0 distance of I 0 , among all images in S:

EQUATION

We refer to the tool of Ripley's K function (Dixon, 2014) , a spatial analysis method used to describe point patterns over a given area of interest. Ripley's K function can be used to determine if the points of interest appears to be dispersed, clustered, or randomly distributed throughout the area. Our defined neighboring function N (I 0 , S, d 0 ) serves as a surrogate of the Ripley's K function K(d).

Hypothesis Testing Given an observed high-identity-density region R obs and a reference region 

R

4. EMPIRICAL STUDY AND ANALYSIS:

We choose two state-of-the-art GANs: PGGAN (Karras et al., 2017) and StyleGAN (Karras et al., 2018) , as our model subjects of study. Both are known to be able to produce high-resolution, realistic and diverse images. We find that the observations below drawn from the two models also generalize to a few other GAN models. We choose the CelebAHQ benchmark (Karras et al., 2017) and FFHQ benchmark (Karras et al., 2018) as our data subject of study. Both benchmarks are composed of diverse and realistic face images. All images are 1024 × 1024 resolution unless otherwise specified.

We use ensemble model of InsightFace (Deng et al., 2019b; Guo et al., 2018; Deng et al., 2018; 2019a) , FaceNet (Schroff et al., 2015) and CosFace (Wang et al., 2018) as f id to serve as the face identity descriptor. We emphasize that the due diligence of "sanity check" has been performed on those classifiers, e.g., their face recognition results are manually inspected one-by-one and confirmed to be highly reliable on the generated images. q (|R ref |) is set to be 1000. We empirically set d b , d e and d s are set to be 0.1, 0.5 and 0.01 respectively.

4.1. OBSERVATION OF THE MODE COLLAPSE:

Mode Collapse Analysis For both StyleGAN and PGGAN, despite of the observed diversity and high quality of their generated images, we empirically find some high-density regions in both learned distributions. Figure 1 shows that the clustering pattern of R obs is significantly different from that of R ref , showing that even the learned distributions of two currently best models have strong dense regions towards some specific identities. For simplicity, our study target is the worst-case dense mode, i.e. the identity with the largest number of neighbors within a given distance threshold.

Consistency of the Dense Mode The dense region R obs is obtained by selecting the top p images in S with the largest number of neighbors. In order to test the consistency of the worst-case dense mode I m against sampling, we visualize the I m w.r.t. different size of S in Figure 2 . We consistently observe roughly the same identity as the sampling size increases. I m can be reliably obtained even when |S|= 1k. The consistency of I m demonstrate that the support size of I m is unnegligible.

4.2. EMPIRICAL STUDY OF THE CAUSE OF MODE COLLAPSE:

We hypothesize multiple factors that may potentially lead to the observed dense mode of face identity. We perform additional experiments, aiming to validate one by one: unfortunately, none of them was observed to reduce the observed mode collapse. That implies the existence of some more intrinsic reason for the mode collapse in GAN, which we leave for future exploration.

Imbalance of Training Data? CelebAHQ is a highly imbalanced dataset: among its 30, 000 highresolution face images of 6, 217 different celebrities, the largest identity class has 28 images and the smallest one has only 1. Would a balanced dataset alleviate the mode collapse? We turn to the Flickr-Faces-HQ Dataset (FFHQ), a high-quality human face dataset created in (Karras et al., 2018) , consisting of 70, 000 high-resolution face images, without repeated identities (we manually examined the dataset to ensure so. It is thus "balanced" in terms of identity, in the sense that each identity class has one sample. We train StyleGAN on FFHQ: somehow surprisingly, the mode collapse persists and seems no less than StyleGAN on CelebAHQ, as shown in Figure 1c and 1d . Unfitting/Overfitting in Training? We train StyleGAN on CelebAHQ (128 × 128) again, and store model checkpoints at iteration 7707 (FID = 7.67, same hereinafter), 8307 (7.02), 8908 (6.89), 9508 (6.63), 10108 (6.41), and 12000 (6.32). We plot their corresponding neighboring function curves in Figure 3b . Similarly, despite the variances, the identity mode collapse persists due to the consistent large gap between R obs and R ref curves. We propose two calibration approaches: reshaping latent space via Gaussian mixture models and importance sampling. They operate on the latent codes, and require no modification of the trained model, nor even any access to the model parameters or training data, making them "black-box".

Both approaches are evaluated with StyleGAN trained on CelebAHQ-128. For simplicity, we only target to eliminating the worst-case dense mode I m , i.e. the identity with the largest number of neighbors within a specified distance threshold.

5.1. RESHAPING LATENT SPACE VIA GAUSSIAN MIXTURE MODELS:

Since we consistently observe close neighbors to I m , when interpolating near I m , we hypothesize that the latent codes of a dense mode I m lay on a smooth manifold. Based on this assumption, we attempt to re-shape that into a Gaussian mixture.

5.1.1. METHOD DESCRIPTION:

The original latent space distribution φ(z; θ 0 ) can be approximated with a mixture of Gaussian distributions K i=1 w i φ(z; θ i ). We randomly sample N latent code and use K-means to explore Starting from a StyleGAN model M pretrained on CelebAHQ-128, we aim at alleviating the collapse on the worst-case dense mode I m in R obs with the largest number of neighbors. We reshape the latent space of M via Gaussian mixture models to get the new model M . We get new worst-case dense mode I m in the new region R obs with the largest number of neighbors. We next randomly sample 10 6 images from the original Gaussian distribution and new GMM distribution, to form T and T respectively. We then plot the neighboring function curves for I m in T and T , and I m in T and T respectively. We expect the reshaping latent space via Gaussian mixture models to alleviate the worst-case dense mode with the minimal sacrifice of generated image quality and diversity.

θ i = (µ i , σ i ).

As shown in Figure 4 , the latent space reshaping could suppress the clustering of I m (indicated by a large gap between the two red curves) without intensifying the clustering of I m (indicated by a little gap between the two green curves), resulting in a reduction of mode collapse on I m . Such an alleviation is achieved with an unnoticeable degradation of generation quality, with FID increasing from 5.93 (M) to 5.95 (M ). The large overlapping between confidence bands

N M R ref and N M R ref

shows that the diversity of generation is not sacrificed either.

5.2. IMPORTANCE SAMPLING:

Under the same hypothesis of smooth manifold in section 5.1, the high-density region corresponding to the worst-case dense mode I m can be approximated with a convex hull.

Importance sampling is a variance reduction strategy in the Monte Carlo method. Let the estimated neighboring function densities for the dense and sparse regions be p 1 and p 2 respectively. We accept the samples from G falling in the high-density region with a probability of p 2 /p 1 , so that the calibrated densities can match.

We approximate the high-density region with a convex hull formed by the collection of latent codes Z Im corresponding to the identities similar to I m :

Conv(Z Im ) = { |Z Im | k=1 α k z k | (∀k : α k ≥ 0) ∧ |Z Im | k=1 α k = 1, z k ∈ Z Im } (3) 5.2.2 EXPERIMENT

The experiment setting is mostly similar to the reshaping latent space via the Gaussian mixture models case. We integrate importance sampling to the latent code generation stage. Given the dense mode I m , we can find the collection of latent codes Z Im via sampling: Z Im is obtained from the top 10 2 latent codes whose corresponding images have the smallest distances (1) to I m , among the 10 6 random samples. We randomly sample 10 6 images from M and M to form T and T respectively. We plot the neighboring function curves for I M in T and T , and I M in T and T respectively. As shown in Figure 5 , the mode collapse is again alleviated (indicated by a gap between the two red curves), without intensifying the clustering of I m (indicated by a little gap between the two green curves), while FID only marginally increases from 5.93 (M) to 5.94 (M ). The confidence band N M R ref is overlapped with N M R ref , showing no loss of the diversity. Additionally, in the appendix, we show a white-box counterpart to the importance sampling approach, where the latent codes Z Im are obtained via explicit optimization (accessing and altering model parameters). The white-box approach does not seem to notably outperform than our above black-box way, implying the relative effectiveness of the latter.

EQUATION

6. DISCUSSIONS AND FUTURE WORK:

This paper is intended as a pilot study on exploring the mode collapse issue of GANs. Using face generation as a study subject, we quantify the general mode collapse via statistical tools, discuss and verify possible causes, as well as propose two black-box calibration approaches for the first time to alleviate the mode collapse. Despite the preliminary success, the current study remains to be limited in many ways. First, there are inevitably prediction errors for the identity descriptors from generated images, even we have performed our best efforts to find the three most accurate descriptor predictions. Moreover, the fundamental causes of GAN mode collapse demand deeper understandings. Besides, the two calibration approaches only handle one worst-case dense mode, leaving much improvement room open for future work.

:

BLACK-BOX CALIBRATION APPROACHESGiven a pre-trained generator G and target dense mode for alleviation, the goals of calibration are three-fold: (1) the density of the mode is maximally alleviated; (2) the diversity and quality of the

Figures and Tables:

Identity Clustering Pattern Analysis via Sampling and Neighboring Function N Given a pre-trained generator G, an identity descriptor f id , a random distribution N (0, Σ), a neighbor distance threshold d 0 and a face embedding space distance range [d b , d e ] ds (d s : step size) S ← {I S 1 , • • • , I S m } // Randomly sampled m face images for each I S i ∈ S do // Count neighbors within d0 distance for each sampled

Figure 1: Identity clustering pattern analysis on StyleGAN and PGGAN, both trained on CelebAHQ. The blue region is a confidence band formed by the pointwise intervals between the upper and lower confidence bounds for all identities in R ref . The red curve is the neighboring function curve for identity in R obs , the worst-case dense mode. We empirically set m (|S|) to be 100, 000 and M (|T |) to be 10, 000, 000. To study the worst-case dense mode, p (|R obs |) is set to be 1.

Figure 2: Visualization of the worst-case dense mode I m w.r.t. different size of the S. S is a collection of randomly sampled images.

Figure 3: Empirical study on possible causes of the mode collapse. The shaded areas denote the variances of neighboring statistics for different experiments (caused by re-initialization/training; running different iterations; and varying architectures: see the texts for details). We empirically set m (|S|) to be 100, 000 and M (|T |) to be 1, 000, 000. To study the worst-case dense mode, p (|R obs |) and is set to be 1.

Figure 4: Identity clustering pattern analysis of Style-GAN on CelebA, before/after latent space reshaping.

Figure 5: Identity clustering pattern analysis of Style-GAN on CelebA, before/after importance sampling.

GAN, without accessing either the training data or the current model parameters? Such black-box calibration is desirable due to many practical demands: the training data might be protected or no longer available; the GAN model might be provided as a black box and cannot be altered (e.g., as APIs); or we simply want to adjust the generated distribution of any GAN with minimized re-training efforts. For the first time, we explore two "black-box" approaches to calibrate the GAN learned distribution, i.e., latent space reshaping via Gaussian mixture models, and importance sampling. They are observed to alleviate the mode collapse without re-touching training data, nor even needing any access to model parameters.

Table content: <table><tr><td>2 RELATED WORKS</td></tr><tr><td>2.1 EVALUATION METRICS OF MODE COLLAPSE IN GANS</td></tr></table>

ref , we want to test the hypothesis that the clustering pattern of R obs is the same as R ref . We use N to get the clustering pattern for the anchor images in R obs and R ref respectively. We can reject the hypothesis if the clustering pattern of R obs is significantly different from R ref . The detailed algorithm is outlined in Algorithm 1.

Table content: <table/>

Model Architecture Differences? Both StyleGAN and PGGAN progressively grow their architectures that can generate images of different resolutions: 128, 256, 512 and 1024. Utilizing this property, we train StyleGAN and PGGAN on CelebAHQ-128, CelebAHQ-256, CelebAHQ-512 and CelebAHQ-1024 respectively, and plot the neighboring function curves correspondingly. According to Figures3c and 3d, varying the architectures does not eliminate the mode collapse either. generated images (measured by FID) are minimally sacrificed; and (3) the calibration is black-box, which does not require access to training data or model parameters.

Table content: <table/>

Peer review (iclr2020):
The direction of this work, to evaluate whether mode collapse exists without using label information, is very good. Rather than using the labels, the authors use an off-the-shelf model (on faces) to provide a space on which to measure distances between generated images. They use this distance to test the hypothesis that samples are highly concentrated around some modes, thus under representing others. The paper could benefit from more clarity. For instance, the methods would be better illustrated through some toy figures. In addition, some explanations in this work are very hard to parse, e.g., the first paragraph of the methods section. While this work is focusing on black box methods for evaluating and palliating mode dropping (aka collapse), it*s a bit disappointing that these results are at least also evaluating on white-box type methods in settings where mode dropping is clearer, e.g. PACGAN on stacked MNIST or even normal CelebA. Unfortunately, no white-box methods are covered in this work, so there is no strong point of comparison, which would be helpful to establish the validity of this work. Finally, the authors demonstrated that there exist a high-density mode, but not whether some modes might be missing. How can this method be used to find missing modes if the generator isn*t generating them without the real data? If my generator is only generating a few digits, but each of them represented similarly by the generated distribution, what would this measure do? It wont detect that the generator is missing modes: you*d need to know those modes existed (e.g., have examples of them). Generally I like the experiments, though I wish there were more qualitative results looking at more than just the existence of one worst-case mode. Other comments page 1 I*m not sure the connection between mode collapse and instability is well-established. What motivates connecting to instability in this work? The statement about co-variate shift is a little vague, and it*s not clear what the connection to mode collapse is. page 2 I*m surprised PACGAN isn*t mentioned in the white-box methods. It was one of the big SOTA methods for palliating mode dropping. I*m not sure why ignore white-box methods: at least it would be good verification that this method works (e.g., across methods, common measures of collapse used in those settings, etc) page 3 The first paragraph of Section 3 is very difficult to understand. f is normalized? page 4 Why not use Ripley*s K? (this is not explained, and should be) The main link missing in the test proposed is that to mode dropping. The problem is that this measure wont detect mode dropping if there*s aren*t samples from those modes to measure anything against. You need real samples as well. How were these face models chosen? Why not use the discriminator of the GAN (at least test what it does)? How were these hyper parameters chosen? page 7: 5.1.2 could use an accompanying toy figure demonstrating what*s going on. ---- Update --- Thanks for your responses and the updated experiment on the white-box calibration. I still think that comparisons to existing methods that palliate mode-dropping is essential for this work, and I would encourage you to bring experiments that allow for comparison to PACGAN et al into the main text for future versions, as the work / direction as value. I will keep my score as-is.