Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This work addresses the important problem of generation bias and a lack of diversity in generative models, which is often called model collapse. It proposed a new metric to measure the diversity of the generative model's "worst" outputs based on the sample clustering patterns. Furthermore, it proposed two blackbox approaches to increasing the model diversity through resampling the latent z. Unlike most existing works that address the model collapse problem, a blackbox approach does not make assumptions about having access to model weights or the artifacts produced during model training, making it more widely applicable than the white-box approaches.  
In terms of experiment setup, the authors chooses face generation as the area to investigate and measures the diversity by detecting the generated face identity. With the proposed methods, the authors showed that most STOA methods have a wide gap between the top p faces of the most popular face identities and randomly sampled faces. It further showed that the proposed blackbox approaches increases the proposed diversity metric without sacrificing image quality.

The proposed diversity measuring metric is lacking both in terms of experimental proofs and intuitive motivations. While the black-box calibration of a GAN model may be attractive under specific settings, the authors did not consider the restrictions under those situations and their design may be hard to implement as a result. For those reasons, I propose to REJECT this paper.

Missing key experiments that will provide more motivation that 1. the new metric reflects human perception of diversity 2. the new metric works better than existing ones:
1. Please provide experiments and/or citation for using the face identity as a proxy for face image diversity. this is important since all your experiments rely on that assumption.
2. Were there experiments that applies your metric to the training datasets like CelebA and FFHQ? In theory your metric should show no gap between N_R_obs and N_R_ref measured on the training dataset since that's the sampled ground truth.

Missing assumptions about blackbox calibration approaches:
1. If we do not have access to the model parameter, the training data, or the artifacts during training like the discriminator, what are some of the real world situations that fit this description? In those cases, is it too much to assume that we can control the random seed input to G? 
2. Is it reasonable to assume some constraints on how much data we can get from the blackbox generator? A website that just exposes the image generation API may not allow you to ping their service 100k times to improve the generation diversity. If you are allowed to do that, it may be reasonable to assume that you can contact the API provider to get access to the rest of the model.

Minor improvements that did not have a huge impact on the score
1. I found the argument about FID in section 2.1 unconvincing. Are there proofs or citations for the claim that real images don't follow multivariate gaussian distribution after applying FID? Copying is indeed an issue that FID cannot detect, but it may be tangential to model collapse for real world concerns like privacy. 
2. The statement "IS, FID and MODE score takes both visual fidelity and diversity into account." under "Evaluation of Mode Collapse" is contradictory to the description in sec 2.1 that IS in fact does not measure diversity. 
3. You may want to consider stating the work as "a pilot study" (sec 6.) earlier in the abstract or in the introduction, so that the reader knows what to expect.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The direction of this work, to evaluate whether mode collapse exists without using label information, is very good. Rather than using the labels, the authors use an off-the-shelf model (on faces) to provide a space on which to measure distances between generated images. They use this distance to test the hypothesis that samples are highly concentrated around some modes, thus under representing others.

The paper could benefit from more clarity. For instance, the methods would be better illustrated through some toy figures. In addition, some explanations in this work are very hard to parse, e.g., the first paragraph of the methods section.

While this work is focusing on black box methods for evaluating and palliating mode dropping (aka collapse), it's a bit disappointing that these results are at least also evaluating on white-box type methods in settings where mode dropping is clearer, e.g. PACGAN on stacked MNIST or even normal CelebA. Unfortunately, no white-box methods are covered in this work, so there is no strong point of comparison, which would be helpful to establish the validity of this work.

Finally, the authors demonstrated that there exist a high-density mode, but not whether some modes might be missing. How can this method be used to find missing modes if the generator isn't generating them without the real data? If my generator is only generating a few digits, but each of them represented similarly by the generated distribution, what would this measure do? It wont detect that the generator is missing modes: you'd need to know those modes existed (e.g., have examples of them).

Generally I like the experiments, though I wish there were more qualitative results looking at more than just the existence of one worst-case mode.

Other comments
page 1
I'm not sure the connection between mode collapse and instability is well-established. What motivates connecting to instability in this work?
The statement about co-variate shift is a little vague, and it's not clear what the connection to mode collapse is.
page 2
I'm surprised PACGAN isn't mentioned in the white-box methods. It was one of the big SOTA methods for palliating mode dropping.
I'm not sure why ignore white-box methods: at least it would be good verification that this method works (e.g., across methods, common measures of collapse used in those settings, etc)
page 3
The first paragraph of Section 3 is very difficult to understand.
f is normalized?
page 4
Why not use Ripley's K? (this is not explained, and should be)
The main link missing in the test proposed is that to mode dropping. The problem is that this measure wont detect mode dropping if there's aren't samples from those modes to measure anything against. You need real samples as well.
How were these face models chosen? Why not use the discriminator of the GAN (at least test what it does)? How were these hyper parameters chosen?
page 7:
5.1.2 could use an accompanying toy figure demonstrating what's going on.

---- Update ---
Thanks for your responses and the updated experiment on the white-box calibration. I still think that comparisons to existing methods that palliate mode-dropping is essential for this work, and I would encourage you to bring experiments that allow for comparison to PACGAN et al into the main text for future versions, as the work / direction as value. I will keep my score as-is.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents a set of statistical tools, that are applicable to quantitatively measuring the mode collapse of GANs. The authors consistently observe strong mode collapse on several state-of-the-art GANs using the proposed toolset. The authors analyze possible causes, and for the first time present two simple yet effective “black-box” methods to calibrate the GAN learned distribution, without accessing either model parameters or the original training data.

The writing and presentation are good.

My concerns regarding this paper are as below.
1) I wonder if the proposed method work for most GAN models, more experiments evaluated on more recent GAN-based  models should be added to verify the superiority claimed in this paper, e.g., TP-GAN [Huang et al., ICCV 2017], PIM [Zhao et al., CVPR 2018], DR-GAN [Tran et al., CVPR 2017], DA-GAN [Zhao et al., NIPS 2017], MH-Parser [Li et al., 2017], 3D-PIM [Zhao et al., IJCAI 2018], SimGAN [Shrivastava et al., CVPR 2016], AIM [Zhao et al., AAAI 2019].
2) The main contributions of this paper are not quite clear to me.
3) Typos need to be corrected in next version, e.g., all equations should have punctuation mark at the end, all e.g., i.e., et al., etc. should be italic, format of references should be consistent.

Based on my comments above, I decide to give the rate of WA for this paper.
review_title: Official Blind Review #1

Meta-review: This paper studies the problem of mode collapse in GANs. The authors present new metrics to judge the model's diversity of the generated faces. The authors present two black-box approaches to increasing the model diversity. The benefit of using a black box approach is that the method does not require access to the weights of the model and hence it is more easily usable than white-box approaches. However, there are significant evaluation problems and lack of theoretical and empirical motivation on why the methods proposed by the paper are good. The reviewers have not changed their score after having read the response and there is still some gaps in evaluation which can be improved in the paper. Thus, I'm recommending a Rejection.
Decision: Reject