Is There Mode Collapse? A Case Study on Face Generation and Its Black-box Calibration
Authors: Zhenyu Wu
Ye Yuan
Zhaowen Wang
Jianming Zhang
Zhangyang Wang
Hailin Jin
Venue:  (2020)
Keywords: Generative Adversarial Networks
Mode Collapse
Calibration

Abstract: Generative adversarial networks (GANs) nowadays are capable of producing im-ages of incredible realism.   One concern raised is whether the state-of-the-artGAN’s learned distribution still suffers from mode collapse. Existing evaluation metrics for image synthesis focus on low-level perceptual quality. Diversity tests of samples from GANs are usually conducted qualitatively on a small scale. In this work, we devise a set of statistical tools, that are broadly applicable to quantitatively measuring the mode collapse of GANs. Strikingly, we consistently observe strong mode collapse on several state-of-the-art GANs using our toolset.  We analyze possible causes, and for the first time present two simple yet effective “black-box” methods to calibrate the GAN learned distribution, without accessing either model parameters or the original training data.

Under review as a conference paper at ICLR 2020
Is THERE MODE COLLAPSE? A CASE STUDY ON FACE GENERATION AND ITS BLACK-BOX CALIBRATION
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) nowadays are capable of producing im- ages of incredible realism. One concern raised is whether the state-of-the-art GAN's learned distribution still suffers from mode collapse. Existing evaluation metrics for image synthesis focus on low-level perceptual quality. Diversity tests of samples from GANs are usually conducted qualitatively on small scale. In this work, we devise set of statistical tools, that are broadly applicable to quantitatively measuring the mode collapse of GANs. Strikingly, we consistently observe strong mode collapse several state-of-the-an GANs using our toolset. We analyze possible causes, and for the first time present two simple yet effective "black-box box" methods to calibrate the GAN learned distribution, without accessing either model parameters or the original training data.
INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) have demonstrated unprecedented power for various image generation tasks. However, GANs have also been suffering from generation bias and/or loss of diversity. The underlying reasons could be compound, ranging from the data imbalance to the training difficulty of GANs, and more:
First of all, the training data for GANs, especially for the typical unconditionallusspprriied generation tasks (Karras et al.) 2017 2018), might possess various subject or attribute imbalances. As a result, GANs trained with them might be further biased towards the denser areas, similarly to the classifier bias towards the majority class in imbalanced classification.
More intrinsically, even when the training dataset "looks" balanced, training GANs is notoriously more unstable (sometimes even uncontrollable) than training classifiers, potentially constituting another source of mode collapse. One most common hurdle of GANs is the loss of diversity due to mode collapse (Goodfellow) (2066), wherein the generator concentrates too large a probability mass on a few modes of the true distribution. Another widely reported issue, known as co-variate shift (Santurkar al. 2017). could be viewed nuanced version of mode collapse.
This paper seeks to explore: do the state-of-the-art GANs still suffer from mode collapse? Can we have a toolkit detect that? And if the mode collapse happens, there any "easy and quick" remedy for calibrating the GAN's learned distribution to alleviate the mode collapse?
Evaluation of Mode Collapse There are several popular metrics for GAN evaluation, e.g. Inception Score (IS) Salimann et al. 2016), Fréchet Inception Distance (FID) (Heusel et al. 2017). MODE (Che et al. 2016) and birthday paradox based diversity test (Arora & Zhang 2017). IS. FID and MODE score takes both visual fidelity and diversity into account. Birthday paradox based diversity test gives a rough estimation of support size under the assumption of uniform sampling. Recently, classification-basse metric (Santurkar al.) (017)) was proposed for quantitative assessment of the mode distribution learned by GANs. However, their approach hinge on a classifier trained on the original (balanced) GAN training set, with class labels known, available and well-defined (e.g., object classes in CIFAR-10, 10, or face gender in CelebA), making it non-straighfforward to extend to data subjects where classes are hard be defined, and/or not enumerable (e.g, open problems).
To tackle this problem, we propose hypothesis test method by analyzing the clustering pattern of samples. We exploit statistical tool from spatial analysis, called Ripley's K function, quantitatively

---

Under review as a conference paper at ICLR 2020
measure the mode collapse. We demonstrate the application of our tool set in analyzing the bias in unconditional face image generation: a popular benchmark task nowadays for GANs, yet remaining rather unclear how to measure its mode collapse using existing tools since every generated identity is expected to be new. The study of face identity generation bias has profound practical values for understanding facial privacy (Filipovych
al. 201T) and fairness (Holstein al., 2018). Using our tools, we find the mode collapse still a prevailing problem in state-of-the-ar face generation GANs (Karras et al. 2018: 2017), and further analyze several possible causes.
Calibration Approaches on GAN Many approaches have been proposed alleviate mode col- lapse problem, ranging from better optimization objectives (Arjovsky et al. 2017: Mao et al. (017). to specialized builing blocks (Durugkar et al., 2016 Ghosh et al. 2018 Liu & Tuzel 2016). However, they require either tedious (re-) Jtraining or at least the access to training data, well as to model parameters: we refer to the existing methods as white-box approaches.
In contrast, are interested in an almost unexplored aspect: assuming known, how can be calibrate the GAN, without accessing either the training data or the current model parameters? Such black-box calibration is desirable due to many practical demands: the training data might be protected or no longer available; the GAN model might be provided as a black box and cannot be altered (e.g,, as APIs); or we simply want adjust the generated distribution of any GAN with minimized re-training efforts. For the first time, we explore two "black-box" approaches to calibrate the GAN learned distribution, i.e., latent space reshaping via Gaussian mixture models, and importance sampling. They are observed to alleviate the mode collapse without re-touching training data, nor even needing any access to model parameters.
generation bias is
2 RELATED WORKS
2.1 EVALUATION METRICS OF MODE COLLAPSE IN GANS
GAN models are often observed to suffer from the mode collapse problem (Salimans et al. 2016); (Sutskever et al. 2015), where only small modes subsets of distribution are characterized by the generator. The problem is especially prevalent for nigh-dimensional data, e.g. face image generation, where the training samples are low-density w.r.t. the high-dimensional feature space.
Salimans et al. (2016) presented the popular metric of Inception Score (IS) measure the individual sample quality. IS does not directly reflect the population-level generation quality, e.g., the overfitting and loss of diversity. It also requires pre-trained perceptual models on ImageNet other specific datasets (Barratt & Sharma 2018). [Hesel et al. (2017) propose the Fréchet Inception Distance (FID), which models the distribution of image features multivariate Gaussian distribution and computes the distance between the distribution of real images and the distribution of fakes images. Unlike IS, FID can detect intra-class mode dropping. However, the multivariate Gaussian distribution assumption hardly holds very well on real images, and low FID score cannot rule out the possibility of the generator's simply copying the training data. Besides the two most popular metrics, (Chel et al.) 2016) develop an assessment for both visual quality and variety of samples, known as the MODE score and later shown to be similar to IS (Zhou et al., 2017). (Arora et al. 2018)) and (Arora & Zhang 2017) proposed test based upon the birthday paradox for estimating the support size of the generated distribution. Although the test can detect severe cases of mode collapse, it falls short in measuring how well generator captures the true data distribution. It also heavily relies on human annotation, making it challenging to scale up to larger-scale evaluation.
(anttrrarr et al. 2017) took classification-baat covariate shift. As we discussed above, their approach cannot be straightforwardly extended to data subjects without pre-known and closed-set class definition, in addition to the need of training extra classifier on the original labeled training set.
perspective and view loss of diversity form of
2.2 MODEL CALIBRATION APPROACHES OF GANS
There many efforts to address the mode collapse problem in GANs. Some focus on discriminators by introducing different divergence metrics (Metz et al., 2016) and optimization losses (Arjovsky
et al. 2017 Mao et al. 2017). The minibatch discrimination scheme allows the discriminator
2

---

Under review as a conference paper at ICLR 2020
to discriminate between whole mini-batches of samples instead of between individual samples. (Durugkar et al. 2016) adopted multiple discriminators to alleviate mode collapse. ModeGAN (Che) et al. 2016) and VEEGAN (Srivastava et al., 2017) enforce the bijection mapping between the imput noise vectors and generated images with additional encoder networks. Multiple generators (Ghosh) al. 2018) and weight-sharing generators (Liu & Tuzel 2016) are developed to capture more modes of the distribution. However, these approaches are designed to easily calibrating trained GANs.
handful of existing works attempt to combine GANs with sampling methods to improve generation quality. (Turner et al. 2018) introduced the Metropolis-Hastiinne generative adversarial network (MH-GAN). The MH-GAN uses the learned discriminator from GAN training to build a wrapper for the generator for improved sampling, at the generation inference stage. With a perfect discriminator, the wrapped generator can sample from the true distribution exactly even with a deficient generator. (Azadi et al., (0188) proposed discriminator rejection sampling (DRS) for GANs, which performs rejection sampling on the outputs of the generator by using the probabilities given by the discrimi- nator, to approximately correct errors in the generator's distribution. Yet still, these approaches are white-box calibration since both require access to trained discriminators (which might be even less available/accessible than the generator after a GAN is trained).
3
METHOD
We intend to study the bias of the most representative features of the generated faces, i.e.the face identity distribution, since almost all face attributes can be derived based on this representations. To detect face identity collapse, we are aiming to detect high-density regions in features space caused by any possible attribute non-diversified. Or, if being slightly imprecise in terms, (Santurka et al., 2017) examined the marginalized distribution through some discrete categorical attributes' lens, while ours looks at the joint distribution of all possible attributes in the continuous feature space holistically.
Algorithm 1 Identity Clustering Pattern Analysis via Sampling and Neighboring Function .V'
> Given a pre-trained generator G, an identity descriptor fia. a random distribution (0, 2.), a neighbor distance threshold do and face embedding space distance range [do, dela, (dg: step size) I/ Randomly sampled m face images // Count neighbors within do distance for each sampled T,$
> S f {1$ ................................ . for each 1 E S do
D NN1 + , do)
Robs +- [i .... Is}
// Find the region for observation by selecting the top p face images in S with largest Ny. Find the region for reference by randomly selecting g face images from S // Randomly sampled M face images (M >> m))
Rres t- -is,..... 15} > TT + IIT. .I...... 1 for each in (do, dela, do
for each IS E Robs do
I/ Count neighbors within d distance for each i in Robs
D .dd. - ~ d.
for each is E Ryef do > D is + (i d.
// Count neighbors within d distance for each 1 in Riet
fs b Compute the pointwise confidence regions of [Nd 11 q, Nd [g] for each d E [db, dela, at
confidence level of a (default 0.05). The intervals between the upper and lower confidence bounds for all samples in Rref define the confidence band (Eubank & Speckman 1993).
& Reject the hypothesis that the clustering pattern of Robs is the same as that of Ryef, ,. if the curve of Nd falls outside of the confidence band. is
Given an unconditional face generator G and an identity descriptor fia, we sample images I= G(2) using a random distribution >~N (0. E). The unit vector fia(I) describes the identity feature in the face embedding space. The normalized cosine distance between image To and Jn is defined as:
d(To, In) Lcos 1os--1 -( Sia (To) fta(T)) >)
3

---

Under review as a conference paper at ICLR 2020
For given anchor face image Tu, a distance threshold dg and collection of randomly sampled face images S, the neighboring function N (Io, S, do) is defined to compute the number of neighbors within do distance of Lo, among all images in S:
(oo, S, do)
Σ IES
1 (1 sgn(do d(Jo,1)))
(2)
We refer to the tool of Ripley's K function (Dixon 2014), a spatial analysis method used to describe point patterns over given area of interest. Ripley's ss K function can be used to determine if the points of interest appears to be dispersed, clustered, or randomly distributed throughout the area, Our defined neighboring function M(oo S, do) serves as surrogate of the Ripley's K function K (d).
Hypothesis Testing Given an observed iee--iddeettdennnn region Robs and a reference region RTef- we want test the hypothesis that the clustering pattern of Roshs is the same RTef- We use W to get the clustering pattern for the anchor images in Robs and Rref respectively. We can reject the hypothesis if the clustering pattern of Robs is significantly different from Ryes- The detailed algorithm is outlined in Algorithm
4 EMPIRICAL STUDY AND ANALYSIS
We choose two state-of-the-arr GANs: PGGAN (Karras et al. 2017) and StyleGAN (Krraa et al. 2018), as our model subjects of study. Both are known to be able produce high-resolution, realistic and diverse images. We find that the observations below drawn from the two models also generalize to few other GAN models. We choose the CelebAHQ benchmark (Karras al. (017) and FFHQ benchmark (Karras et al.) 2018) as our data subject of study. Both benchmarks are composed of diverse and realistic face images. All images 1024 1024 resolution unless otherwise specified.
We use ensemble model of InsightFace (Deng et al. 2019h Guo et al. 2018 Deng et al. 2018 2019a), FaceNet (Schroff et al. 2015) and CosFace (Wang al. 2018) as fat to serve as the face identity descriptor. We emphasize that the due diligence of "sanity check" has been performed on those classifiers, e.g., their face recognition results are manually inspected one-by-one and confirmed to be highly reliable on the generated images, ((Rre is set to be 1000. We empirically set do, de and d, are to be 0.1, 0.5 and 0.01 respectively.
4.1 OBSERVATION OF THE MODE COLLAPSE
Mode Collapse Analysis For both StyleGAN and PGGAN, despite of the observed diversity and high quality of their generated images, we empirically find some high-density regions both learned distributions. Figure shows that the clustering pattemn of Robs is significantly different from that of Rres, showing that even the learned distributions of two currently best models have strong dense regions towards some specific identities. For simplicity, our study target is the worst-case dense mode, i.e. the identity with the largest number of neighbors within given distance threshold.
Consistency of the Dense Mode The dense region Robs is obtained by selecting the top p images in S with the largest number of neighbors. In order to test the consistency of the worst-case dense mode Im against sampling, we visualize the Im w.r.t. different size of S in Figure We consistently observe roughly the same identity as the sampling size increases. Im can be reliably obtained even when S 1k. The consistency of I demonstrate that the support size of T,, is unnegligible.
4.2 EMPIRICAL STUDY OF THE CAUSE OF MODE COLLAPSE
We hypothesize multiple factors that may potentially lead to the observed dense mode of face identity. We perform additional experiments, aiming to validate one by one: unfortunately, none of them was observed to reduce the observed mode collapse. That implies the existence of some more intrinsic reason for the mode collapse in GAN, which we leave for future exploration.
Imbalance of Training Data? CelebAHQ is a highly imbalanced dataset: among its 30, 000 high- resolution face images of 6, 217 different celebrities. the largest identity class has 28 images and the smallest one has only 1. Would balanced dataset alleviate the mode collapse?
4

---

Under review as a conference paper at ICLR 2020
6.5
6.0
6.0
5.5
5.5
4.5
log (ull
4.0
4.0
3.5
3.5
NRa
3.0
Niw
3,0
No.
2.5
2.0
1.5
1.5
1.0
1.0
0.0
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
(a) PGGAN-CclebAHO
(b) StyleGAN-CclebAHO
6.5
5.5
5.5
5.0
5.0
4.5
4.5
Ma
3.5
NR.
3.0
New
3.0
New
2.5
2.5
1.5
1.0
0.5
0.5
0.10
0.15
0.20
0.25
0.30
0.50
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
d
d
(c) PGGAN-FFHQ
(d) StyleGAN-FFHQ
Figure 1: Identity clustering pattern analysis StyleGAN and PGGAN, both trained CelebAHQ- The blue region is a confidence band formed by the pointwise intervals between the upper and lower confidence bounds for all identities in Ryef- The red curve is the neighboring function curve for identity in Robs. the worst-case dense mode. We empirically set ([S)) to be 100, 000 and M ([T)) be 10, 000, 000. To study the worst-case dense mode, p (Robb is set be 1.
(a) | 1k
(b) S| 10k
(c) |S| 100k
(d) |S|= 1m
(e) |S = 10m
Figure 2: Visualization of the worst-case dense mode Im w.r.t. different size of the S. S is a collection of randomly sampled images.
We turn to the Flickr-Faces-Ho Dataset (FFHQ), high-quality human face dataset created in (Karras) et al. 2018). consisting of 70, 000 high-resolution face images, without repeated identities (we manually examined the dataset to ensure so. It is thus "balanced" in terms of identity, in the sense that each identity class has one sample. We train StyleGAN on FFHQ: somehow surprisingly, the mode collapse persists and seems less than StyleGAN CelebAHQ, shown in Figure Te ld
5

---

Under review as a conference paper at ICLR 2020
6.5
6.5
5.0
5.0
4.5
4.5
Na.
Nxw
New
2.5
25
2.0
2.0
0.5
0.5
0.0
0.0
0.10
0.15
0,20
0.25
0.35
0.40
0.45
0.50
0.10
0.15
0,20
0.35
0.45
0.50
(a) BanleGAN-Randomnesss.
(b) SyyeGAANOOverfiiinnnnneeeeriiii
6.5
6.5
6.0
6.0
4.5
4.5
4.0
4,0
3.5
35
New
2.0
2.0
1.5
15
0.0
0.0
0.10
0.20
0.25
0.30
0.35
0.45
0.50
0.10
0.20
0.25
0.35
0.45
(c) StyleGAN-Arcchteeture
(d) PGGAN-Archilecture
Figure 3: Empirical study possible causes of the mode collapse. The shaded areas denote the variances of neighboring statistics for different experiments (caused by ce-mitialization/raaiinng running different iterations; and varying architectures: ((S) to be 100, 000 and M ((T) to be 1, 000, 000. To study the worst-case dense mode, (R and is set be
the texts for details). We empirically set m
Randomness during Intialiizatoo/Oppiiiizatooo We repeat training StyleGAN on CelebAHQ (128 x 128) for 10 times. The experimental results are shown in Figure Ba with the shaded areas denoting the variances. Despite the variance for the neighboring function curves plotted for repeated experiments, large gap between the curves of Rabs and Ryef can be consistently observed.
Unfitting/Overrittinng in Training? We train StyleGAN on CelebAHQ (128 x 128) again, and store model checkpoints at iteration 7707 (FID = 7.67, same hereinafter), 8307 (7.02), 8908 (6.89), 9508 (6.63), 10108 (6.41), and 12000 (6.32). We plot their corresponding neighboring function curves in Figure 3b Similarly, despite the variances, the identity mode collapse persists due the consistent large gap between Robe and Rref curves.
Model Architecture Differences? Both StyleGAN and PGGAN progressively grow their archi- tectures that can generate images of different resolutions: 128, 256, 512 and 1024. Utilizing this property, train StyleGAN and PGGAN on CelebAHQ- 128 CelebAHQ-25
CelebAHQ-511 and CelebAHQ-1022 1024 respectively, and plot the neighboring function curves correspondingly According to Figures 3c and 3d varying the architectures does not eliminate the mode collapse either.
BLACK-BOX CALIBRATION APPROACHES
Given a pre-trained generator G and target dense mode for alleviation, the goals calibration are three-fold: (1) the density of the mode maximally alleviated; (2) the diversity and quality of the

---

Under review as a conference paper at ICLR 2020
generated images (measured by FID) minimally sacrificed; and (3) the calibration is black-box, which does not require access to training data or model parameters.
We propose two calibration approaches: reshaping latent space via Gaussian mixture models and importance sampling. They operate on the latent codes, and require no modification of the trained the model parameters training data, making them "black-box".
model, even any
Both approaches are evaluated with StyleGAN trained on CelebAHQ- 128. For simplicity, we only target to eliminating the worst-case dense mode I i.e. the identity with the largest number of neighbors within a specified distance threshold.
5.1 RESHAPING LATENT SPACE VIA GAUSSIAN MIXTURE MODELS
Since we consistently observe close neighbors to Im when interpolating near Ims we hypothesize that the latent codes of dense mode In lay on a smooth manifold. Based on this assumption, we attempt to re-shape that into a Gaussian mixture.
5.1.1 METHOD DESCRIPTION
The original latent space distribution <(=00)) can be approximated with mixture of Gaussian distributions wib(2;; (2; 0.). We randomly sample N latent code and K-means explore
0, (Mi, oz). We denote p(lm) the probability of sampling the worst-case dense mode Im-
Σ wi p p(1ml2)00(2 0.)22. If p(4m(0)) is large, reduce 1=1 W/ to make the overall p(Tm) small. p( 10:) is estimated by the number of neighbors within do distance to Im in cluster C, i.e. ( C₁ do).
p(mm) p([ml=))(2 (2))((2;))))2))))
5.1.2 EXPERIMENTS
Starting from StyleGAN model M pre- trained on CelebAHQ-128, we aim at al- leviating the collapse on the worst-case dense mode Im in Robs with the largest number of neighbors. We reshape the latent space of M via Gaussian mixture models to get the new model M'. We get the new worst-case dense mode I'm in the new gion R' with the largest number of neigh- bors. We next randomly sample 10 images from the original Gaussian distribution and new GMM distribution, form T and T' respectively. We then plot the neighboring function curves for 1m and T', and I'm in T and T' respectively. We expect the re- shaping latent space via Gaussian mixture models to alleviate the worst-case dense mode with the minimal sacrifice of gener- ated image quality and diversity.
6.5
5.0
3.0
No.
Nr.
Ni n
1.5
Ni, . Nif.
1.0
.Rl.
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
Figure 4: Identity clustering pattern analysis of Style- GAN CelebA, before/after latent space reshaping.
As shown in Figure the latent space reshaping could suppress the clustering Im (indicated by a large gap between the two red curves) without intensifying the clustering of T (indicated by little gap between the two green curves), resulting in reduction mode collapse on Im- Such an alleviation is achieved with an unnoticeable degradation of generation quality, with FID increasing from 5.93 (M) 5.95 (M'). The large overlapping between confidence bands N and M shows that the diversity of generation sacrificed either.
R.ef
R
7

---

Under review as a conference paper at ICLR 2020
5.2
IMPORTANCE SAMPLING
Under the same hypothesis of smooth manifold in section 5.1, the high-density region corresponding the worst-case dense mode In. can be approximated with convex hull.
5.2.1 METHOD DESCRIPTION
Importance sampling is variance reduction strategy in the Monte Carlo method. Let the estimated neighboring function densities for the dense and sparse regions pi and Pa respectively. We accept the samples from G falling in the high-density region with a probability of p2/P1, that the calibrated densities can match.
We approximate the high-density region with a convex hull formed by the collection of latent codes Zim corresponding to the identities similar to I'
12
Zim
Conv (Z
(vk: on 2 0)
Σ
O = 1, ZK 7smm.
5.2.2 EXPERIMENT
The experiment setting is mostly similar to the reshaping latent space via the Gaussian mixture models case. We integrate importance sampling the latent code generation stage. Given the dense mode m, we can find the collection of latent codes Zr,, via sampling:
Zi. = [21mm G(=)) s do, 2 ~ N(O,
(4)
Zr is obtained from the top 102 latent codes whose corresponding images have the smallest distances (T) to I. among the 10c random samples. We randomly sample 10t images from M and M to form T and T' respectively. We plot the neighboring function curves for 7a in T and T', and I in and T' respectively. As shown in Figure [S] the mode collapse is again al- leviated (indicated by a gap between the two red curves), without intensifying the clustering of I'm (indicated by little gap between the two green curves), while FID only marginally increases from 5.93 (M) to 5.94 (MM'.. The confidence band N M is overlapped with N N'M
5.3
5.0
3.5
3.0
No. UM.
2.0
1.5
Agg.
ARt.
0.0
0.15
0.25
0.30
0.35
0.40
0.45
0.50
Figure 5: Identity clustering pattern analysis of Style-
GAN on CelebA, before/after importance sampling. Ryes" showing no loss of the diversity.
Roe!
Additionally, in the appendix, we show a white-box counterpart to the importance sampling approach, where the latent codes Zmm are obtained via explicit optimization (accessing and altering model parameters). The white-box approach does not seem to notably outperform than our above black-box way, implying the relative effectiveness of the latter.
DISCUSSIONS AND FUTURE WORK
This paper is intended as pilot study on exploring the mode collapse issue of GANs. Using face generation as a study subject, we quantify the general mode collapse via statistical tools, discuss and verify possible causes, as well propose two black-box calibration approaches for the first time to alleviate the mode collapse. Despite the preliminary success, the current study remains be limited in many ways. First, there are inevitably prediction errors for the identity descriptors from generated images, even we have performed our best efforts to find the three most accurate descriptor predictions. Moreover, the fundamental causes of GAN mode collapse demand deeper understandings. Besides, the two calibration approaches only handle one worst-case dense mode, leaving much improvement room open for future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-114-22 2017.
Sanjeev Arora and Yi Zhang. Do gans actually learn the distribution? an empirical study, arXiv preprint arXiv:/7066 706.0822400..4.. 2017.
Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do GANs learn the distribution? some theory and empiries. In International Conference on Learning Representations, 2018.
Samaneh Azadi, Catherine Olsson, Trevor Darrell, lan Goodfellow, and Augustus Odena. Discrimi- nator rejection sampling. arXiv preprint 181.:0880000758, 2018.
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint arXiv: 1oXiv:8801.01977 2018.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. arXiv preprint 1rXiv:16/2.021366 2016.
Jiankang Deng. Anastasios Roussos, Grigorios Chrysos, Evangelos Ververas, Irene Kotsia, Jie Shen, and Stefanos Zafeiriou. The menpo benchmark for multi-pose 2d and 3d facial landmark localisation and tracking. IJCV, 2018.
Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019a.
Jiankang Deng, Jia Guo, Zhou Yuxiang, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-stage dense face localisation the wild. In arxiv, 2019b.
Philip M Dixon. Ripley's & k function. Wiley StutsRef: Statistics Reference Online, 2014.
Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multi-adversarial networks, arXiv preprint arXiv:1611.01673 2016.
Randall L Eubank and Paul L Speckman. Confidence bands in nonparametric regression. Journal of the American Statistical Association, 124724):22771134) 1993.
Roman Filipovych, Christos Davatzikos, Alzheimer's Disease Neuroimaging Initiative, et al. Semi- supervised pattern classification of medical images: application to mild cognitive impairment (mci).
Neurolmage, 55(3):1109-111 1119, 2011.
Arnab Ghosh, Viveka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K Dokania. Multi- agent diverse generative adversarial networks. In Proceedings of the IEEE Conference Computer 2018.
Vision and Pattern Recognition, pp.
Ian Goodfellow. Nips 2016 tutorial: Generative adversarial networks.
arXiv preprint
170100170..00166, 2016.
Ian Goodfellow, Jean Pouget-Abadie. Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014.
Jia Guo, Jiankang Deng, Niannan Xue, and Stefanos Zafeiriou. Stacked dense u-nets with dual transformers for robust face alignment. In BMVC. 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, and Hanna Wallach. Improving faimness in machine learning systems: What do industry practitioners need? arXiv preprint arXiv: 18/2.008 2018

---

Under review as a conference paper at ICLR 2020
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:/710.10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv: 04948, 2018.
Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. Advances neural information processing systems, pp. 469-477, 2016.
Xudong Mao, Qing Li. Haoran Xie, Raymond YK Lau, Zhen Wang. and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794-2802 2017.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks, 2016.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, pp. 2234-2243 2016.
Shibani Santurkar, Ludwig Schmidt, and Aleksander Mądry, A classification- study of covariate shift in gan distributions. arXiv preprint arXiv: 171.v01711.00970, 2017.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Information Processing Systems, pp. 3308-3318, 2017.
Ilya Sutskever, Rafal Jozefowicz, Karol Gregor, Danilo Rezende, Tim Lillierap, and Oriol Vinyals. Towards principled unsupervised learning. arXiv preprint 1 2015.
Ryan Turner, Jane Hung, Yunus Saatci, and Jason Yosinski. Metropolis-hasting generative adversarial networks. arXiv preprint 181.
2018
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5265-5274, 2018
Zhiming Zhou, Weinan Zhang, and Jun Wang. Inception score, label smoothing, gradient vanishing and-log (d (x)) alternative. arXiv preprint arXiv: 2017.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
toy figure to explain our evaluation procedure
Robh (Top p foces with the loreest number neighbore within distance at dg
R R
Randomly selected g faces
Sampling
ITI M(n M)
Neighzoring onalysis on Rata and Reef for eon de 10 ddle
Bampling
Figure 6: Illustration of our Algorithm 1: Identity Clustering Pattern Analysis via Sampling and
Neighboring Function N. See details in Algorithm !
Obtaining Latent Codes by Optimization (White-box Approach) ing Zr,,. is latent code recovery via optimization:
The second approach of find-
Zim = { min Lugg (m G(z)) + + - G(2)laa N(0,2))
(5)
6.5
6.0
5.0
4.5
log ((ld))
3.5
3.0
Na.
2.5
N
Na. n
1.5
Nuw
1.0
Naf.
0.0
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Figure 7: Clustering pattern analysis of StyleGAN on CelebA, before/after importance sampling.
Here we are using combination of perceptual loss, loss and Gaussian prior. a and y are coefficients respectively. We run the optimization 102 times with different random initialization of get 102 latent code. We then sample 10 images from M and M' to form T and T'' respectively. We plot the neighboring function curves for the worst-case dense mode I., in T and T and I', in T shown in Figure the bias again alleviated (indicated by gap between the two red curves), while FID only marginally increases from 5.93 (M) to 5.96 (M'). The confidence band N is overlapped with NMM" V showing loss of the diversity.
and T' respectively.
We can observe that the FID score only marginally increases after white-box calibration. Meanwhile, the solid-red line (before calibration) is shifted to the right dash-red line (after calibration), indicating that the white-box approach alleviate mode collapse problem and severer mode is introduced. Besides, the solid- -ideengeennnnn line and dash-green line are almost overlaid, suggesting that the white-box approach has minimal impact other modes.
11

---

Under review as a conference paper at ICLR 2020
Evaluation results different percentile instead of worst-case In the current evaluation metric, we only consider worst case because the worst case is the most important mode collapse problem care in generative models, i.e., if the model always generates similar faces, we can not the generated dataset for other training purposes due to fairness/privacy concern. However, our evaluation metric be easily adapted to detect median/mean dense regions by using faces with different percentiles instead of top k dense faces.
l0g (
Mha-th
4.0
- Akg-
3.5
3.0
2.5
2.0
1.5
Aha-Th
1.0
Mhn N0s
0.5
0.0
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Figure 8: Identity clustering pattern analysis of StyleGAN on CelebAHQ using different percentile clustering anchor
In this figure, we show the clustering pattern of choosing different percentiles in step mode detection. We can observe that R overlays with Robs 20% and above, which demonstrats that around 20% of the generated images are highly clustered.
Mode collapse between training data and testing data Our approach also have the capability to detect missing modes between training data and the generated data. If we simply change the first step to random sample images from the training data, which could hopefully capture all modes in the training data, then we could find the missing mode by comparing the clustering pattern between training data and testing data.
6.5
NR 0.00%
6.0
NR 0.3%
5,5
NR
5.0
NR
NRa 20%
(M(
NRar
NR.
NR 50%
NR 63%
NR
NR.
1.0
NRas 90% Ne..
0.0
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Figure 9: Identity clustering pattern analysis StyleGAN on CelebAHQ using training data as clustering anchor
We can observe that the Rref overlays with Rose and we can not reject the hypothesis. Generated images not severe clustered around anchors from training data, indicating that GAN models have the capability to generate "new faces". The width of the Ryef band Fig. 8 is similar to that in Fig. 9 suggesting that mode dropping is not severe between training data and generated images.
12

---

Under review as a conference paper at ICLR 2020
Applying Our Proposed Metric FFHQ FFHQ is public face dataset contains 56, images. without repeating identities. We first randomly pick 1k images to form the set and sort the S set according to the number of neighbors within distance 0.3. We choose the sample at percentile 0. 0. 0.1%, 1%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%. We conduct the neighboring analysis these selected samples. As shown in Figure T0 still observe gap between Ros and Ryef. which demonstrates that FFHQ dataset has dense mode, even without repeating identities. Furthermore, we would like clarify that our metric proposed to measure the collapse of GAN's learned distribution. We have empirically shown in the paper that the mode collapse still occurs despite balanced training data.
6.5
6.0
5.0
4.5
Mac
Nu
Maa
Maa
3.5
3.0
Map
2.0
1.5
0.5
0.0
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Figure Identity clustering pattern analysis of StyleGAN on FFHQ
13


Reviews:
[{"review_id": "SJlp3GIRtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The direction of this work, to evaluate whether mode collapse exists without using label information, is very good. Rather than using the labels, the authors use an off-the-shelf model (on faces) to provide a space on which to measure distances between generated images. They use this distance to test the hypothesis that samples are highly concentrated around some modes, thus under representing others.\n\nThe paper could benefit from more clarity. For instance, the methods would be better illustrated through some toy figures. In addition, some explanations in this work are very hard to parse, e.g., the first paragraph of the methods section.\n\nWhile this work is focusing on black box methods for evaluating and palliating mode dropping (aka collapse), it's a bit disappointing that these results are at least also evaluating on white-box type methods in settings where mode dropping is clearer, e.g. PACGAN on stacked MNIST or even normal CelebA. Unfortunately, no white-box methods are covered in this work, so there is no strong point of comparison, which would be helpful to establish the validity of this work.\n\nFinally, the authors demonstrated that there exist a high-density mode, but not whether some modes might be missing. How can this method be used to find missing modes if the generator isn't generating them without the real data? If my generator is only generating a few digits, but each of them represented similarly by the generated distribution, what would this measure do? It wont detect that the generator is missing modes: you'd need to know those modes existed (e.g., have examples of them).\n\nGenerally I like the experiments, though I wish there were more qualitative results looking at more than just the existence of one worst-case mode.\n\nOther comments\npage 1\nI'm not sure the connection between mode collapse and instability is well-established. What motivates connecting to instability in this work?\nThe statement about co-variate shift is a little vague, and it's not clear what the connection to mode collapse is.\npage 2\nI'm surprised PACGAN isn't mentioned in the white-box methods. It was one of the big SOTA methods for palliating mode dropping.\nI'm not sure why ignore white-box methods: at least it would be good verification that this method works (e.g., across methods, common measures of collapse used in those settings, etc)\npage 3\nThe first paragraph of Section 3 is very difficult to understand.\nf is normalized?\npage 4\nWhy not use Ripley's K? (this is not explained, and should be)\nThe main link missing in the test proposed is that to mode dropping. The problem is that this measure wont detect mode dropping if there's aren't samples from those modes to measure anything against. You need real samples as well.\nHow were these face models chosen? Why not use the discriminator of the GAN (at least test what it does)? How were these hyper parameters chosen?\npage 7:\n5.1.2 could use an accompanying toy figure demonstrating what's going on.\n\n---- Update ---\nThanks for your responses and the updated experiment on the white-box calibration. I still think that comparisons to existing methods that palliate mode-dropping is essential for this work, and I would encourage you to bring experiments that allow for comparison to PACGAN et al into the main text for future versions, as the work / direction as value. I will keep my score as-is.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xrU4J2FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a set of statistical tools, that are applicable to quantitatively measuring the mode collapse of GANs. The authors consistently observe strong mode collapse on several state-of-the-art GANs using the proposed toolset. The authors analyze possible causes, and for the first time present two simple yet effective “black-box” methods to calibrate the GAN learned distribution, without accessing either model parameters or the original training data.\n\nThe writing and presentation are good.\n\nMy concerns regarding this paper are as below.\n1) I wonder if the proposed method work for most GAN models, more experiments evaluated on more recent GAN-based  models should be added to verify the superiority claimed in this paper, e.g., TP-GAN [Huang et al., ICCV 2017], PIM [Zhao et al., CVPR 2018], DR-GAN [Tran et al., CVPR 2017], DA-GAN [Zhao et al., NIPS 2017], MH-Parser [Li et al., 2017], 3D-PIM [Zhao et al., IJCAI 2018], SimGAN [Shrivastava et al., CVPR 2016], AIM [Zhao et al., AAAI 2019].\n2) The main contributions of this paper are not quite clear to me.\n3) Typos need to be corrected in next version, e.g., all equations should have punctuation mark at the end, all e.g., i.e., et al., etc. should be italic, format of references should be consistent.\n\nBased on my comments above, I decide to give the rate of WA for this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlStebC5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work addresses the important problem of generation bias and a lack of diversity in generative models, which is often called model collapse. It proposed a new metric to measure the diversity of the generative model's \"worst\" outputs based on the sample clustering patterns. Furthermore, it proposed two blackbox approaches to increasing the model diversity through resampling the latent z. Unlike most existing works that address the model collapse problem, a blackbox approach does not make assumptions about having access to model weights or the artifacts produced during model training, making it more widely applicable than the white-box approaches.  \nIn terms of experiment setup, the authors chooses face generation as the area to investigate and measures the diversity by detecting the generated face identity. With the proposed methods, the authors showed that most STOA methods have a wide gap between the top p faces of the most popular face identities and randomly sampled faces. It further showed that the proposed blackbox approaches increases the proposed diversity metric without sacrificing image quality.\n\nThe proposed diversity measuring metric is lacking both in terms of experimental proofs and intuitive motivations. While the black-box calibration of a GAN model may be attractive under specific settings, the authors did not consider the restrictions under those situations and their design may be hard to implement as a result. For those reasons, I propose to REJECT this paper.\n\nMissing key experiments that will provide more motivation that 1. the new metric reflects human perception of diversity 2. the new metric works better than existing ones:\n1. Please provide experiments and/or citation for using the face identity as a proxy for face image diversity. this is important since all your experiments rely on that assumption.\n2. Were there experiments that applies your metric to the training datasets like CelebA and FFHQ? In theory your metric should show no gap between N_R_obs and N_R_ref measured on the training dataset since that's the sampled ground truth.\n\nMissing assumptions about blackbox calibration approaches:\n1. If we do not have access to the model parameter, the training data, or the artifacts during training like the discriminator, what are some of the real world situations that fit this description? In those cases, is it too much to assume that we can control the random seed input to G? \n2. Is it reasonable to assume some constraints on how much data we can get from the blackbox generator? A website that just exposes the image generation API may not allow you to ping their service 100k times to improve the generation diversity. If you are allowed to do that, it may be reasonable to assume that you can contact the API provider to get access to the rest of the model.\n\nMinor improvements that did not have a huge impact on the score\n1. I found the argument about FID in section 2.1 unconvincing. Are there proofs or citations for the claim that real images don't follow multivariate gaussian distribution after applying FID? Copying is indeed an issue that FID cannot detect, but it may be tangential to model collapse for real world concerns like privacy. \n2. The statement \"IS, FID and MODE score takes both visual fidelity and diversity into account.\" under \"Evaluation of Mode Collapse\" is contradictory to the description in sec 2.1 that IS in fact does not measure diversity. \n3. You may want to consider stating the work as \"a pilot study\" (sec 6.) earlier in the abstract or in the introduction, so that the reader knows what to expect.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of mode collapse in GANs. The authors present new metrics to judge the model's diversity of the generated faces. The authors present two black-box approaches to increasing the model diversity. The benefit of using a black box approach is that the method does not require access to the weights of the model and hence it is more easily usable than white-box approaches. However, there are significant evaluation problems and lack of theoretical and empirical motivation on why the methods proposed by the paper are good. The reviewers have not changed their score after having read the response and there is still some gaps in evaluation which can be improved in the paper. Thus, I'm recommending a Rejection.
Decision: Reject