Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper defines a goodness of fit measure F for generative networks, that reflects how well a model can generate the training data. F allows to detect mode collapse: as long as it is strictly positive, mode collapse is observed as parts of the training data have not been memorized. It aims at providing an alternative to the Fréchet Inception Distance and the Inception Score that rely on pretrained neural networks (whereas this new measure does not). It also provides insight into the DCGAN and WGAN networks in that regard, observing for instance that data subsampling helps decrease F, which motivates the use of a mixture of GANs.

This paper brings an interesting contribution to the evaluation of generative networks. However:

1.	The use of the square distance in the image space is not obvious and not justified.
2.	Computation of this metric is not straightforward: there is no theoretical guarantee and it is computationally expensive.
3.	The theoretical properties of this measure and its robustness are not investigated.
4.	Typos are obscuring the reading of the paper.

- Post rebuttal: I have read the authors' response and am maintaining my weak reject rating.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This work proposed a new goodness of fit measure for generative network evaluations, which is based on how well the network can generate the training data. The measure is zero if the network could perfectly recover the training data, and would represent how far it is from generating the training set in the average manner of the total least square sense, where the one-to-one mapping between the generated data and the training sample is constructed through latent space optimization. Using the proposed measure, the authors showed an interesting trend present in the DCGAN training and the impact of the residual connection. The authors might want to add some discussion in Section 4.2 regarding why the residual connection is detrimental for covering the support.  Increasing the model complexity through larger latent space dimension and learning mixtures is proposed as solutions to improve the measure as well.

With all the interesting results presented, I still have the concerns about the sensitivity of the proposed measure:
- It is an average over the training data or the selected sample. Above Section 4, the authors argued that "\hat{F}(G) > 0 meaning that we do not observe any memorization". This seems overly assertive. Since the measure is an average over the training data, it has difficulty to differentiate between one network which has almost zero value for part of the training data but large values for the rest, and another network with roughly the same \hat{F}(G) value but small values for all training data. The variance could help, but can not resolve this issue. This would be more important when the training data contains noise or outliers.
- It only concerns the generation of the training data, but not the sampled data from the network (at least not directly). Therefore it has no direct control of the fidelity of the generated samples.
- As shown by the authors, the proposed measure can be considered as the approximation of the true probability support not covered by the generative models, which also defines a necessary condition to avoid mode collapse. But what about the other part? It would have difficulty comparing two models with the same support but different high-density areas. Indeed, there are existing works which consider both the precision and recall of the generative models [1, 2, 3], and directly work with the generated samples instead of the training data. These should be discussed and compared with, not just the FID scores which have already been shown to have issues [3]. 

Some notations:
- In the last equation on Page 2,  should it be L_{G} instead of L_{D}?
- In the first equation on Page 3, should the denominator be N_{B} instead of N_{N}?
- "Optimality" in terms of generative models may depend on the downstream tasks. I do not think there exists a universal definition of "optimality" for generative models.

[1] M.S.M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. NeurIPS 2018.
[2] L. Simon, R. Webster, and J. Rabin. Revisiting precision and recall definition for generative model evaluation. ICML 2019.
[3] T. Kynkaanniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. Arxiv:1904.06991.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper proposes a new goodness of fit measure for generative models, and uses it to get insight into GAN's. While this is an important topic and a novel approach, I do not think the paper delivers on what it promises.

I think this paper should be rejected. First, while it claims to be a general method for generative models it, it is limited to only  GANs and even for GANs it is limited. Second, most of the observations are nice but trivial, e.g. larger latent space leads to larger image.

Detailed remarks:
- The main point that the training set points x must have p(x)>0 under the model is naturally satisfied for almost all models except GANs such as VAEs, autoregressive model and flow models with standard implementations as the support is the whole space. This is in contrast to the claim in the paper that "its applications can be extended to other generative networks such as Variational Autoencoders.".
- Even for GANs as this measure only looks at the support and not the distribution it is not clear if this measure does more then evaluate mode collapse. While this is an important task, it falls short of the promises the authors claim.
- The authors claim that "We demonstrate that our measure being minimized is a necessary and sufficient condition to detect mode collapse." but only show that it is necessary.
- Proposition 1 is a trivial statement.
- The authors claim that " mode collapse happens if P(x) > 0 but minz ||G(z) − x|| > 0". This is a main point by the authors, but it ignores the probability and only looks at the support. It has been shown that mode collapse happens even in 2d distributions, e.g. veegan paper, where it is easy to get the support to be the whole distribution.
- The results in sec. 5 are quiet obvious, with a larger latent space you can naturally get a larger support, same as with a mixture model.


In general the method only looks at the support, ignoring the distribution over the support and is therefore very limited in evaluating generative models.  


minor details:
- In eq. 3 the integration should be w.r.t dP(x) for it to be monte-carlo approximated as it is in eq. 4.
- Not 100% I understand what the authors try to say here - "we pick the latent variable z and error ||G(z) − x||2 that corresponds to the smallest error instead of picking the latent variable that Adam Kingma and Ba (2014) finds."
review_title: Official Blind Review #2

Meta-review: This paper proposes to measure the distance of the generator manifold to the training data. The proposed approach bears significant similarity to past studies that also sought to analyze the behavior of generative models that define a low-dimensional manifold (e.g. Webster 2019, and in particular, Xiang 2017). I recommend that the authors perform a broader literature search to better contextualize the claims and experiments put forth in the paper.

The proposed method also suffers from some limitations that are not made clear in the paper. First, the measure depends only on the support of the generator, but not the density. For models that have support everywhere (exact likelihood models tend to have this property by construction), the measure is no longer meaningful. Even for VAEs, the measure is only easily applicable if the decoder is non-autoregressive so that the procedure can be applied only to the mean decoding. 

In this current state, I do not recommend the paper for submission.

Xiang (2017). On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
Webster (2019). Detecting Overfitting of Deep Generative Networks via Latent Recovery

Decision: Reject