Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work addressed an interesting problem and a very relevant one - how to audit generative models? The most popular metric to evaluate generative models is FID but it is extremely opaque and population based. This paper proposes new fidelity metrics that overcome both these challenges. While the definition of these metrics are well grounded, their measurement process itself - which is done via trained NN classifiers - is a bit unconvincing. The authors demonstrate that the proposed metric can recover the real ranking of generative models on a synthetic data generation task and can discover issues like mode collapse.
main_review: Strengths: The paper addresses an important topic that is time critical and relevant to the generative modeling community and the broader sphere of AI. It is well written and motivated and a pleasure to read overall. The authors clearly point out the issues that are lacking with existing metrics and motivate the need to define the new versions of precision / recall metrics. Moreover, the metrics enable a new use case of model auditing which can be very relevant is safety critical applications.

Weaknesses: 
1. The paper promises a lot in terms of the defined metrics but the evaluations are bit underwhelming. The experiment with the predictive model depends on the downstream classifier as well - the performance of the generative model is tied to the performance of the classifier - hence the hyper-parameter settings and training settings for the classifier should be explained clearly. 
2. Another aspect that is missing from the evaluation is the application of model auditing on real life critical data. The idea of model auditing becomes useful mainly in cases where we need to filter at the sample level. When a generative model is trained on medical data (imaging) how do the metrics help in model auditing?
3. An interesting question to probe is how to compare different generative models that have comparative expertise? For instance, one model could be an expert in color and another could be in shape and so on - can we use the defined metrics to identify such behaviors?
4. Please provide error bars on the predictive modeling experiments - as in, account for the randomness in the generative model training process.
summary_of_the_review: Overall, the paper does a good job in addressing a relevant problem in a well motivated manner. Even though there are some open concerns regarding the evaluation, the fact that the current approach is one of the first to facilitate model auditing for generative models makes this a good contribution. Authors, please address my concerns in the weaknesses listed above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a methodology assessing the performance of a generative model in a domain agnostic fashion.  A 3-dimensional metric space is proposed: fidelity (output quality), diversity (coverage of expected variability of output) and generalization (to what degree model avoids memorizing training data i.e. is truly generative) is proposed.  Particularly the latter element is novel and the manner in which it is may be evaluated by developing an `authenticity’ measure for the task. Three illustrative use cases in image (mnist) and medical patient data (COVID-19, `Hide and Seek’ seq2seq data) domains are provided.
main_review: The paper contributes useful thought and discussion for how best to argue the holistic performance of a generative model, contrasting with FID and other domain-specific approaches.  There are novel perspectives in this paper that are worth sharing with the community, particularly those contributing to discussion around generalization vs. memorization (copying) where it is shown that prior metric ignoring this aspect otherwise indicate performant models (c.f. time-series NeurIPS challenge data).  The paper could be improved by drawing deeper relationship with prior thought in this area e.g. Adlam et al. (2019), Meehan et al. (2020) who are cited in Sec 3.2.2 but not discused.  Indeed the brief literature review is relegated to the paper Appendix and was not helpful in arguing for the novel contribution of the paper in context of such work.  The paper also lacks any reflection or conclusion on the limitations of the proposed metrics.  It was unclear to me how the proposed authenticity classifier would work for multi-modal distribution given the assumptions of noise within a hypersphere.  It did not seem like it would be practical for realistic, complex data problems.  A use case on high-dimensional data e.g. images beyond a toy MNIST example would have better demonstrated the practical utility of this work for data where the need of such performance metrics is clearer.
summary_of_the_review: I am on the borderline tending toward accepting the paper given the interesting discussion around authenticity and memorization, but I am not fully convinced this is a practically useful paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper targets to a 3-dim metric: alpha-Precision, beta-Recall, and Authenticity, that quantifies the fidelity, diversity, and generalization performance of a generative model. The proposed metric serves as soft-boundary classifiers between real and generated spherical-shaped supports, and improves the robustness against outliers and generation failure cases. This sample-wise metric can audit models by judging individual synthetic samples by their quality. 
main_review: Strengths:
+ Clear writing and demonstrations.
+ Meaningful research topic.
+ Technically reproducible.

Weaknesses:
- The technical novelty is marginally revised from Sajjadi et al. 2018.
- The experiments are incomplete and therefore unconvincing.
  - The testing datasets are a bit toy-like. Like Sajjadi et al. 2018, please also experiment on CelebA [1] or FFHQ [2]. For mode collapse evaluation, please consider using attribute classifiers pre-trained on CelebA, as well as IvOM [3].
  - The testing generative models are out of date. Results on old-fashioned architectures are not conclusive for cutting-edge research works. Imagine our community works on a new GAN paradigm and targets to outperforming StyleGAN3 [1], are we convinced to use the proposed metric which has only been validated on toy data/models? For GAN, please consider validating on StyleGAN3 [4]. For VAE, please consider validating on VQ-VAE-2 [5].

[1] Liu, Ziwei, et al. "Deep learning face attributes in the wild." ICCV 2015.

[2] Karras, Tero, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks." CVPR 2019.

[3] Metz, Luke, et al. "Unrolled generative adversarial networks." ICLR 2017.

[4] Karras, Tero, et al. "Alias-free generative adversarial networks." NeurIPS 2021.

[5] Razavi, Ali, Aaron van den Oord, and Oriol Vinyals. "Generating diverse high-fidelity images with vq-vae-2." NeurIPS 2019.
summary_of_the_review: See the main review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a mechanism to calculate the "goodness" of a generative model on a per-sample level; this method is model agnostic. This can be used to audit the model post-hoc. 
main_review: Strengths:

1. Well written, easy to follow.
2. Timely topic.
3. Discussion of NeurIPS challenge is important.

Weaknesses:

1. I did not learn something substantial from reading the paper.
2. The claims are too strong.

Detailed comments:

1. The generalized definitions of precision and recall are effective markers of understanding synthetic data quality. However, empirically they do not outperform other distribution measuring mechanisms (such as FID, PW, W) and I suspect methods like KL divergence or MMD, or TV distance will also perform comparably. 
2. To my understanding, approaches like FID, KL Divergence, MMD, TV distance are also “model agnostic”; I humbly request the authors to rethink the positioning of their work along this particular claim.
3. The appeal in this work lies in the fact that the proposed measures can be calculated on a per-sample basis, unlike prior works. 
4. Empirically, the authors consider few issues which may result in poor fidelity/diversity (mode collapse being one of them). If the models did not generalize during training, how do these methods capture them?
5. Could the authors clarify how the proposed method provides interpretability? The authors observe correlations between induced failures and their scores, but they can’t really attribute the score to a particular failure event?
6. The proposed measure for authenticity relies on the ability to detect overfitting. However, recent work by Feldman et al. argues that overfitting (or memorization, in some cases) is essential for generalization. Can the authors comment on the same?

summary_of_the_review: Key requirements: I would urge the authors to empirically validate if overfitting has occurred to validate their claims (maybe through MI attacks).
review_title: null

Meta-review: The authors propose a new set of metrics for evaluation of generative models based on the well-established precision-recall framework, and an additional dimension quantifying the degree of memorization. The authors evaluated the proposed approach in several settings and compared it to a subset of the classic evaluation measures in this space. The reviewers agreed that this is an important and challenging problem relevant to the generative modeling community at large. The paper is well-written and the proposed method and motivation are clearly explained. 

The initial reviews were borderline, and after the discussion phase we have 2 borderline accepts, one strong accept, and one strong reject. After reading the manuscript, the rebuttal, and the discussion, I feel that the work should not be accepted on the grounds of insufficient empirical validation. Establishing a new evaluation metric is a very challenging task -- one needs to demonstrate the pitfalls of existing metrics, as well as how the new metric is capturing the missing dimensions in a thorough empirical validation. While the former was somewhat shown in this work (and in many other works), the latter was not fully demonstrated. The primary reason is the use of a non-standard benchmark to evaluate the utility of the proposed metrics. I agree that covering a broader set of tasks and models makes sense in general, but it shouldn’t be done at the cost of existing, well-understood benchmarks. I expected to see a thorough comparison with [1], one of the most practical metrics used today which can be easily extended to all settings considered in this work (notwithstanding the drawbacks outlined in [2]). What are the additional insights? What is [1] failing to capture in practical instances? Does the rank correlation change with respect to modern models across classic datasets (beyond MNIST and CIFAR10)? This would remove confounding variables and significantly strengthen the paper.

My final assessment is that this work is borderline, but below the acceptance bar for ICLR. I strongly suggest the authors to showcase the additional improvements over methods such as [1] in practical and well-understood settings commonly used to benchmark generative models (e.g. on images). The experiments suggested by the reviewers are a step in the right direction, but not sufficient.

[1] Improved Precision and Recall Metric for Assessing Generative Models. Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, Timo Aila. NeurIPS ’19

[2] Evaluating generative models using divergence frontiers. 
Josip Djolonga, Mario Lučić, Marco Cuturi, Olivier Frederic Bachem, Olivier Bousquet, Sylvain Gelly. AISTATS ‘20
Decision: Reject