How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
Authors: Ahmed Alaa
Boris van Breugel
Evgeny Saveliev
Mihaela van der Schaar
Venue: ICLR 2022 Submitted (2022)

Abstract: Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, ($\alpha$-Precision, $\beta$-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of  any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample-level and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional dimension for model performance that quantifies the extent to which a model copies training data---a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.

Under review as a conference paper at ICLR 2022
How FAITHFUL IS YOUR SYNTHETIC DATA? SAMPLE-LEVEL METRICS FOR EVALUATING AND AUDITING GENERATIVE MODELS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the imgge synthesis setup, exhibit limited capacity for diagnos- ing the modes of failure of generative models across broader application domains. In this paper, introduce 3-dimensional metric, (a-Precision, 3-Recall, Au- thenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a aggoosic fashion. Our metric unifies statisti- cal divergence measures with precision-recal analysis, enabling sample-level and distribution-leved diagnoses of model fidelity and diversity. We introduce gener- alization as an additional dimension for model performance that quantifies the ex- tent to which a model copies training data-a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric compo- nents correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by (black-box model, discarding low-quality sam- ples and hence improving the overall model performance a post-hoc manner.
INTRODUCTION
Intuitively, it would seem that evaluating the likelihood function of a generative model is all it takes to assess its performance. As it turns out, the problem of evaluating generative models is far more complicated. This is not only because state-of oh-the-art models, such as Variational Autoencoders (VAE) (Kingma & Welling (2013)) and Generative Adversarial Networks (GANs) (Goodfellow al. (2014)), do not possess tractable likelihood functions, but also because the likelihood score itself is a flawed measure of performance-it scales badly in high dimensions, and it obscures distinct modes of model failure into single uninterpretable score (Theis et al. (2015)). Absent metrics, earlier work focused on crafting domain-specific scores, e.g., Inception score (Salimans et al. (2016)), with an exclusive emphasis on image data (Lucic et al. (2018)).
In this paper, introduce an alternative approach evaluating generative models, where instead of assessing the generative distribution by looking at all synthetic samples collectively to compute likelihood divergence, we classify each sample indivicually as being of high or low quality. In this way, our metric comprises interpretable probabilistic discriminative models ge.g., AUC-ROC) which describe the rate by which model makes errors. When averaged over all samples, our sample-level scores reflect discrepancy between real and gen- crative distributions in way similar to statistical divergence measures (e.g., KL divergence, Fréchet distance (Heusel al. (2017)). maximum mean discrepancy (Sutherland al. (2016)). In this sense, our metric enables diagnosing model performance on both the sample and distribution levels.
renitiiee-rreebliinn those used to evaluate
Our metric represents the performance of generative model point in 3-dimensional space each dimension corresponds to a distinct quality of the model. These qualities are: Fidelity, Diversity and Generalization. Fidelity measures the quality of model's synthetic samples, and Diversity is

---

Under review as a conference paper at ICLR 2022
Support of real distribution Pr
Figure 1: Pictorial depiction for the pro- posed evaluation metrics. The blue and red spheres correspond to the a- and 8-supports of real and generative distributions, respec- tively. Blue and red points correspond to real and synthetic data. Synthetic data falling outside the blue sphere will look unrealistic noisy. (b) Overfitted models can generate ostensibly high-quality data samples that are "unauthentic" because they are copied from the training data. (c) High-quality data sam- ples should reside inside the blue sphere. (d) Outliers do not count in the 3-Recall met- ric. (Here, a=3=0.9, a- a-Precision-8//1 /9, 3- Recall 4/9, and Authenticity 9/10.)
Support of synthetic distribution Ir
Noisy synthetic sample
(a)
Unauthentic
high-quality sample
(b)
(d)
Authentic
high-quality
Real
Synthetic
Outlier
sample
the extent to which these samples cover the full variability of real samples, whereas Generalization quantifies the extent which model overfits (copies) training data.
How do we quantify the 3 dimensions of performance? We build on the precision-recall analysis framework proposed in (Sajjadi et al. (2018)), and introduce the a-Precision and 3-Recall metrics to quantify model Fidelity and Diversity, respectively. Both metrics assume that fraction 1- (or 1- B) of the real (and synthetic) data are "outliers", and a (or 3) are "typical". a-Precision is the fraction of synthetic samples that resemble the "most typical" real samples, whereas 3-Recall the fraction of real samples covered by the most typical 3 synthetic samples. a-Precision and 3-Recall are evaluated for all a, 3 [0, 1], providing entire precision and recall curves instead of single numbers. compute both metrics, we embed the (real and synthetic) data into hyperspheres with most samples concentrated around the centers, i.e., the real and generative distributions (P, and Pg) has spherical-shaped supports. In this transformed feature space, typical samples would be located near the centers of the spheres, whereas outliers would be closer to the boundaries.
To quantify Generalization, we introduce the Authenticity metric, which reflects the likelihood of a synthetic sample being copied from training data. We derive the Authenticity metric from hypoth- esis test for data copying based on the observed proximity of synthetic samples to real ones in the embedded feature space. A pictorial illustration for all metrics shown in Figure
How is our metric different? one think of standard precision and recall metrics as "hard" binary classifiers of real and synthetic samples, our a-Precision and 3-Recall boundary classifiers that do not only compare the supports of IP, and P. but also assesses whether both distributions are calibrated. Precision and recall metrics are special cases of a-Precision and 3-Recall for a 3 = 1. As we show later, our new metric definitions solve many of the drawbacks of standard precision-recee recall analysis, such as lack of robustness to outliers and failure to detect dis- tributional mismatches (Naeem et al. (2020)). They also enable detailed diagnostics of different types model failure, such as mode collapse and mode invention. Moreover, optimal values of our metrics are achieved only when P, and P are identical, thereby eliminating the need to augment the evaluation procedure with additional measures of statistical divergence (c.g., KL divergence).
be thought of as soft-
Previous works relied pre-trained embeddings (using ImageNet feature extractors (Deng et al. (2009))). In this work, we propose feature embeddings that are model- and domain-agnostic. and are tailored to metric definitions and data set hand. Our proposed feature embedding step be completely bespoke to raw data, or augmented with pre-trained trained embeddings. This enables our metric remain operable in application domains where no pre-trained representations exist.
Overfitting is a crucial mode failure of generative models, especially when modeling sensitive data (e.g., clinical data) for which data copying may violate privacy requirements (Yoon al. (2020)), but it has been overlooked in previous works which focused exclusively on quantifying the Fidelity- Diversity characterization (Brock et al. (2018)). As we show in our experiments (Section 5), because our metric accounts for Generalization, it can provide a fuller picture of a generative model's perfor- mance. Precisely, we show that some of the celebrated generative models score highly for Fidelity
2

---

Under review as a conference paper at ICLR 2022
and Diversity simply because they memorize real samples, rendering them inappropriate for privacy- sensitive applications. A comprehensive survey of prior work is provided in the Appendix.
Model auditing novel use case. The sample-level nature of our metrics inspires the case of model auditing, wherein we judge individual synthetic samples by their quality, and reject unauthentic. In Section 5, we show that model audits can
samples that have low Fidelity improve the outputs of a black-box model in a post-hoc fashion without any modifications to the model itself, and demonstrate this use case in synthesizing clinical data for COVID-19 patients.
EVALUATING AND AUDITING GENERATIVE MODELS
2.1 PROBLEM SETUP
We denote real and generated data as X, Irr and X₂ F. respectively, where X,, X, E X, with F, and Pg being the real and generative distributions, and X' being the input space. The generative distribution. P. g, is estimated using generative model (c.g., GAN). The real and synthetic data sets are Dreat= (x,,)== and Daynth = (X where X7 ~ IP, and Xgs ~ Pg.
2.2 WHAT MAKES A GOOD SYNTHETIC DATA SET?
Our goal is to construct metric (Dreat Deynth) for the quality of Deynth in order to (i) evaluate the performance the underlying generative model gg. and (ii) audit the model outputs by discard- ing (individual) "low-quality" samples, thereby improving the overall quality of Dyyuth- In order for to fulfill the evaluation and auditing tasks, must satisfy the following desiderata: (1) it should be able disentangle the different modes of failure of Pd through interpretable measures of performance, and (2) it should be sample-wise computable, i.c., we should be able to tell a given (individual) synthetic sample X, ~ y of low quality.
Having outlined the desiderata for our sought-after evaluation metric, we now propose three qualities of synthetic data that the metric & should be able to quantify. Failure fulfill any of these three qualities correspond to independent modes of failure of the model IPg. These qualities are:
Fidelity-the the generated samples resemble real samples from Pr. A high-fidelity synthetic data set should contain "realistic" samples, e.g. visually-realistic images.
2. Diversity-the generated samples are diverse enough to cover the variability of real data, i.e., model should able to generate a wide variety good samples.
3. Generalization generated samples should be mere copies of the (real) samples in training data, i.c.. models that overfit to Dreat are not truly "generative".
In Section 3, we propose a dine-diimesio evaluation metric & that captures all of the qualitics above. Our proposed metric can be succinctly described as follows:
E A (a-Precision, 3-Recall, Authenticity
Fidelity
Diversity
Generolization:
The a-Precision and 3-Recall metrics are generalizations of the conventional notions of precision and recall used in binary classification analysis (Flach & Kull (2015)). Precision measures the rate by which the model synthesizes "realistic-looking" samples, whereas the recall measures the fraction of real samples that are covered by IPg. The authenticity score measures the fraction of synthetic samples that are invented by the model and not copied from the training data.
2.3 EVALUATION AND AUDITING PIPELINES
Having provided a bird's sird's-eye view of our proposed metric E, we now briefly summarize the steps involved in the evaluation and auditing tasks. Since statistical comparisons of complex data types in the raw input space X' are difficult, the evaluation pipeline starts by embedding X, and X, into a "meaningful" feature space through a representation Φ dubbed the evaluation embedding, and then computing & on the embedded features (see Figure 2(a)). Section 4, we propose a representation learning approach construct embeddings tailored our metric and the data set hand.
3

---

Under review as a conference paper at ICLR 2022
In the (post-hoc) model auditing task, we compute the sample-level metrics for each Xo. in D Paynth. and discard samples with low authenticity and/or precision scores, which results
(a) Evaluation pipeline
D synth
a-Precision
Authenticity
X,
"curated" synthetic data
Dreal
B-Recall
set with an improved overall performance. When granted direct access to the model P.. the auditor serves rejection sam- pler that repeatedly draws samples from Fg. only accepting ones with high preci- sion and authenticity (Figure 2(b)). Model auditing is possible through our metrics they can be used to evaluate the quality of individual synthetic samples; the same task cannot be carried out with statisti- cal divergence measures that compare the overall real and generative distributions.
Evaluation embedding
Sample-level metrics
(b) Auditing pipeline
High-quality sample?
Accept or
Check a-Precision
X
Reject
PP
E
X₂
Authenticity
Generative model
Evaluation pipeline
Sample filter
Figure 2: The evaluation and auditing pipelines.
Q- a-PRECISION, 8-RECALL AND AUTHENTICITY
3.1 DEFINITIONS AND NOTATIONS
Let X, (Xx,) and X,, (X,)) be the embedded real and synthetic features. For simplicity, we will use P., and Pg to refer to the distributions over the raw and embedded features interchangeably. Let S, = supp(PP, and S, = supp(F,g) where supp(P) is the support of Central our proposed metrics is more general notion for the support of distribution P, which we dub the a-support. We define the a-support as the smallest subset of = supp(P) supporting probability mass a, i.e.,
S min V(s), s.f. P(s)
22
where V(s) is the volume (Lebesgue measure) of and [0, 1]. One can think of an a-support as dividing the full support of IP into "normal" samples concentrated in S and "outliers" residing in Sa, where S = S US* The notion of a-support is also known as the minimum volume set, and has been traditionally used in outlier detection models (Polonik (1997); Scott & Nowak (2006)).
Finally, we define the distance between data sample X and the training data Dreat as the distance between and the closest sample Dreats i.c.,
d(X, Dreat) min d(x, X..), 15$50
(3)
where is distance metric defined over the input space X.
3.2
SAMPLE-LEVEL EVALUATION METRICS
3.2.1
G-PRECISION AND 3 REEALL
a-PPrecssion. The conventional Precision metric defined as the probability that generated sample supported by the real distribution, iee. P(X, E S,) (Sajjadi et al. (2018)). We propose a more refined measure of sample fidelity, dubbed the a-Precision (denoted P.), defined follows:
Po A P(X, E 8,),, for E [].
(4)
That is, Pa the probability that a synthetic sample resides in the a-support of the real distribution.
B-Recall. To assess diversity in synthetic data, propose the 3-Recall metric generalization of the conventional Recall metric. Formally, we define the B R-Recall as follows:
R3 P(X, E Sy),, for [0, 1].
(5)
i.e. Rais the fraction of real samples that reside in the B-support of the generative distribution.
4

---

Under review as a conference paper at ICLR 2022
(a) Mode collapse
(b) Mode invention
(c) Density shift
(d) a-Precision and B-Recall curves
P) score 1, suboptimal Pa curve
P) score - 1, poor Pa curve
P) score I, better Pa curve
1.0
(a)
F
P
Ph
PP
P
(c) (b)
0.0
1.0
S
1.0
Rj Rystorect c1 poor Rl curve
Po
Pg
(4)
poor Rá curve
suboptimal R3 curve
Ra
(b)
Pr
P
(a)
37
S
2.
0.0
1.0
Figure 3: Interpretation of the Pa and Ro curves, Real distribution is colored in blue, generative distribution in red. Distributions collapsed into I dimension for simplicity. Here, P, is multimodal distribution of cat images, with one mode representing orange tabby cats and another mode for Calico cats; outliers comprise exotic Caracal cats. Shaded areas represent the probability mass covered by a- and f-ssppoots--thes supports concentrate around the modes, but need not be contiguous for multimodal distributions, i... we have S.: S²,1 U S. and S S
5,11 U USS, (a) Here, the model Py exhibits mode collapse where it over-represents orange tabbies. Such model would achieve a precision score of Pi = but a suboptimal (concave) Pa curve (panel (d)). Because it does not cover all modes, the model will have both suhoptimal R3 score and R3 curve. (b) This model perfectly nails the support of IP,, hence it scores optimal standard metries P1 R1 However, the model invents mode by over-representing outliers, where mostly generates images for the exotic cat breed. Standard metrics imply that model (a) outperforms (b) where in reality (a) is more faithful to the real data. P. and Ra give us fuller picture of the comparative performances of both models. (c) This model realizes both types of cats but estimates slightly shifted support and density; intuitively. this is the best of the three models, but it will appear inferior (b) under Pi and see that model (c) has less deviation from optimal performance (the dashed black lines panel (d)).
R1. By cxamining the Pa-Ra, Rgg curves, we
Interpreting a--Precision and B-Recall. To interpret (4) and (5), we first need revisit the notion of a-support. From (2), we know that an a-support hosts the most densely packed probability mass in distribution, hence Se and Sy always concentrate around the modes of P., and IP, (Figure 3); samples residing outside of S.: and S can be thought of as outliers. In this sense, Pa and Rg do not count outliers when assessing fidelity and diversity. That is, the a- a-Precisson score deems synthetic sample to be of a high fidelity not only if it looks "realistic", but also it looks "typical". Similarly, 3-Recall counts real sample as being covered by Pg only if it is not outlier in pgy By sweeping the values of a and |3 from 0 to 1, we obtain a varying definition of which samples are typical and which are outliers-the gives us entire Pa and Rg curves as illustrated in Figure
Generalizing precision-recal analysis. Unlike standard precision and recall, Pa and R3 take into account only the supports of P, and Fg. but also their densities. Standard precision (and recall) correspond to one point on the Pa (and Rş) curve; they coincide with Pa and Rg evaluated on the full supports (i.e., P1 and R1). By defining our metries with respect to the a- and B-supports, we do not treat all samples equally, but assign higher importance to samples in "denser" regions of S, and Sy. Pa and R, reflect the extent to which PP, and Pg are calibrated i.e., good Pa and Rg are achieved when I., and P., share the same modes and not just common support.
Our proposed Pa and R: metrics address major shortcomings of the commonly used Pi and Ri, among these are: lack of robustness to outliers, failure to detect matching distributions, and inability to diagnose different types of distributional failure (Naeem et al. (2020). Basically, Pg will score perfectly precision and recall =R]= as long as it nails the support of P,,, even if P, and P, place totally different densities on their common support. Figure illustrates how our metries rem- edy these shortcomings. While optimal R, and P. are achieved by arbitrarily mismatched densities, our P. and Rg curves are optimized only when P., and P, are identical as stated by Theorem 1.
Theorem 1. The a-Precision and B-Recall satisfy the condition P./a a Ra/3 1, Va, 3, if and only if the generative and real densities are identical, i.e., P, = Pr-
That is, model is optimal if and only if its Pa and R2 are both straight lines with unity slopes.
5

---

Under review as a conference paper at ICLR 2022
Measuring statistical discrepancy with Pa and Rg. While the Pa and Rg curves provide de- tailed view on a model's fidelity and diversity, is often more convenient to summarize performance in single number. To this end, we define the mean absolute deviation of P. and Rg as:
P = ( Pa --- -a da, ARg = S. RR - -,,33,
(6)
where APa E [0, 1/2] and AR3 E [0, quantify the extent which Pa and Rg deviate from their optimal values. We define the integrated Pa and Rg metrics as IP. = 2A Pa and I RRg-
1 Rg. both take values in [0, 1]. From Theorem 1, IP 2Rg only P. P,
Together, IPa and I RRg serve as measure of the discrepancy between the distributions F, and Pg. eliminating the need to augment our precision-recal analysis with measures of statistical divergence. Moreover, unlike f-divergence measures, the (IPa. IRş) metric disentangles fidelity and diversity into separate components, and does not require that P, and P, share a common support.
3.2.2 AUTHENTICITY
Generalization is independent of precision and recall since model achieve perfect fidelity and diversity without truly generating any samples, simply by resampling training data. Unlike discrim- inative models for which generalization is casily tested via held-out data, evaluating generalization in generative models is not straightforward (Adlam et al. (2019); Meehan et al. (2020)). We propose an authenticity score A E [o. 1] to quantify the rate by which model generates new samples. To pin down a mathematical definition for A, we reformulate P', mixture of densities follows:
Fg = P.F. + (1 A) -g.e.
(7)
where IP, is the generative distribution conditioned on the synthetic samples being non-overfitted, and 6y is noisy distribution over training data. In particular, we define 6g. as 6g.- = 6, (N,,
where 8y a discrete distribution that places an unknown probability mass on each training data point in Dreat, is an arbitrarily small noise variance, and is the convolution operator. Essentially, (7) assumes that the model flips a (biased coin), pulling off a training sample with probability A and adding some noise to it, innovating new sample with probability A. A model with A always innovates, whereas an overfitted model will concentrate Fg around the training data.
4 ESTIMATING THE EVALUATION METRIC
Since the metrics in Section 3 are defined through binary conditions on individual samples, we can obtain an estimate غ (Pa. R2, A) of the metric E, for given a and [3, by assigning binary scores Pa.j. A, {0, }} to each synthetic sample X,.j in D syntth, and R3. E {0,1) 1] to each real sample X in Dreat, then averaging over all samples, i.e., Pa = E, Pa.y R.
Σ, Ra A Σ Ag. To assign binary scores, construct classifiers fr, fn. fa X {0. 1), where Pa., = (X,,), Rp., = (x,,..) and A, fa(8g.)) We explain the operation of each classifier in what follows.
Precision and Recall classifiers. Based on definitions (4) and (5), both classifiers check a sample resides in an a- (or (-) support, i.e., f main difficulty in implementing fr and fn is estimating the supports S. and 8 9 in fact, even we know the exact distributions IP, and P computing their and 8-supports is not straightforward as it involves solving the optimization problem in (2).
1(X, S:) and fn(X.)) = 1(X, 5.). Hence, the
To address this challenge, we pre-process the real and synthetic data in way that renders estimation of a-and B-supports straightforward. The idea is to train the evaluation embedding so as to cast the supports of the real data, S,., into hypersphere with radius and cast the distribution IP, into an isotropic density concentrated around the center c, of the hypersphere. We achieve this by modeling t as one-class neural network trained with the following loss function: L Σ, l, where
4-
max || ((x,)) c, 1 -2).
(8)
The loss is minimized over the radius r and the parameters of <b; the output dimensions of 4, c, and v viewed hyperparameters (see Supplementary material). The loss in (8) is based the seminal

---

Under review as a conference paper at ICLR 2022
work one-class SVMs in (Schölkopf et al. (2001)), which is commonly applied to outlier detection problems, e.g., (Ruff et al. (2018)). In nutshell, the evaluation embedding squeezes real data into the minimum-volume hypersphere centered around Cy, hence the real a-support is estimated as:
5 = B(), Fo = (I - G 1siish,
where (cc, r) is Euclidean ball with center and radius r, and Qa is the a-quantile function. The set of all a-supports of IP, corresponds to the set of all concentric spheres with center er and radii Par Vo E [0, 11. Thus, the precision classifier assigns a score I to a synthetic sample Xg if it resides in the Ball Sp, i.e., fo(X,) 1 X - ++1 s 8. Now define Co = i persphere (ccg, F8). where n3z Q cyll isss I m). We construct fx follows:
m Σj Xg.. and consider hy-
fn(X,.) = 1(x;, 9. E B(X NNDx.
(10)
where X is the synthetic sample in (c, T8) that is closest to Xy,1 and (XD,(X,,)) is the 9.5" distance between X and its k-nearest neighbor in Dreat- Similar to the estimator in (Naeem et al. (2020)), (10) is a nonparametric estimate of S that checks if each real sample is locally covered by a synthetic sample in (cg. Pg). A discussion on how to select the an alternative method for estimating S using one-class representations is provided in the Appendix.
k. as well as
Authenticity classifier. We derive the classifier SA from hypothesis test that tests if a sample Xy.i non-memorized. Let Hi A, 1 be the hypothesis that Xg. is authentic, with the null hypothesis Ho Aj 0. To test the hypothesis, we use the Akelihood-ratio statistic (Van Trees (2004)):
^(X,,) PP I.j A, = 1)/P(X 9. = -) (x9.)/6
(11)
which follows from (7). Since both likelihood functions in (11) are unknown, we need to test the hypothesis Hi : A, = using an alternative sufficient statistic with a known probability distribution.
Let dz. = d( xx.i Dreat) be the distance between synthetic sample and the training data set, and let it be the training sample in Dreat closest to Xaj. i.c., dg., d(X,.,, Xx.,). ). Let dr. be the distance between X and Dreat/ (x,,,) }, i.e., the training data with sample i" removed. Now consider the statistic aj = 1 [dg.s S dr.a., }. which indicates if synthetic sample j is closer to training data than any other training sample. The likelihood ratio for observations {az}; under hypotheses Ho and Hi is
Maa)) = P(a, A, = 1)/P(a, A = 0) = a5² P(dg. S dr,,. A, = =1).
(12)
Here, we used the fact that if sample j is memorized copy of i", and if the noise variance e in (7) is arbitrarily small, then ay = 1 almost surely and P(ay A 0) ss 1. If is authentic, then Xg, lies in the convex hull of the training data, and hence P(a; A, 0)
and A for large real data set. Thus, fA issues a label A₁ = 1 if aj = 0, and A, = 0 otherwise. Intuitively, fA deems sample j unauthentic if it is closer i* than any other real sample in the training data.
EXPERIMENTS AND USE CASES
5.1 EVALUATING & AUDITING GENERATIVE MODELS FOR SYNTHESIZING COVID-19 DATA
In this experiment, we use our metric to assess the ability of different generative models to synthesize COVID-19 patient data that can be used for predictive modeling. Using SIVEP- Gripe (SIVEP- Gripe (2020)), database of 99,557 COVID patients in Brazil, including sensitive data such as ethnicity. We use generative models to synthesize replicas of this data and fit predictive models to the replicas.
Models and baselines. We create 4 synthetic data sets using GAN, VAE, Wasserstein GANs with a gradient penalty (WGAN-GP) (Gulrajani et al. (2017)), and ADS-GAN, which is specifically de- signed to prevent patient identifiablity in generated data (Yoon et al. (2020)). To evaluate these synthetic data sets, we use Fréchet Inception Distance (FID) (Heusel et al. (2017)). Precision/Recall (P//R1) (Sajjadi al. (2018)), Density/Coverage (DIC) (Naeem et al. (2020)), Parzen window like- lihood (PW) (Bengio et al. (2013)) and Wasserstein distance (W) as baselines. On each synthetic data, we fit predictive Logistic regression model to predict patient-level COVID-19 mortality.
7

---

Under review as a conference paper at ICLR 2022
Ranking generative and predictive models
Hyper- papee-paramete optimization
(c) Synthetic data improvement auditing
Gmandrmutt ranling:
X a a WGAN-GPAA A. WA GAN
scale of ADS-GAN dentifability loss
ADS-GAN Ausited ADS-EAN
P.
Metric
Generative models
AUC-ROC
ranking
F10
0.79 : 0.02
Improved authenticily
14
PW
0.79:
94
II
P
0.05 X
IPa Pix
R
2
0.55:0.03
Improved
precision
11
18s
0.79 20.02
Figure 4: Predictive modeling with with synthetic data. (a) Here, we rank the 4 generative models (ADS-GAN: x. WGAN-GP: e, VAE: A, GAN: m) with respect to each evaluation metric (leftmost is hest). For each metric, we train a predictive model on the synthetic data set with highest score, and test its AUC on real data. Ground-truth ranking of synthetic data is the ranking of the AUC of predictive models trained on them. (b) Slyper-paramete tuning for ADS-GAN, (Dashed lines are linear regression lines.) (c) Post-hoc auditing of ADS-GAN.
Predictive modeling with synthetic data. In the context of predictive modeling, a generative model assessed with respect its usefulness in training predictive models that generalize well on real data. Hence, the "ground-trute
ranking of the 4 generative models corresponds to the ranking of the AUC-ROC scores achieved by predictive models fit to their respective synthetic data sets and tested on real data (Figure 4(a)). The data synthesized by ADS-GAN (x) displayed the best performance, followed by WGAN-GP (+), VAE (4), and GAN (m). To assess the accuracy of baseline evaluation metrics, we test if each metric can recover the ground-truth ranking of the 4 generative models (Figure 4(a)). Our integrated precision and recall metrics IPa and RRg both assign the highest scores to ADS-GAN; IPa exactly nails the right ranking of generative models. On the other hand, competing metrics such as P1. C and D, over-estimate the quality of VAE and WGAN-GP if we these metrics to decide which generative model to use, will end up with predictive models that perform poorly, i.e. AUC-ROC of the predictive model fitted to synthetic data with best P1 is 0.55, compared an AUC-ROC of 0.79 for our IP, score.
These results highlight the importance of accounting for the densities Py and Pr, and not just their supports, when evaluating a generative model. This is because a shifted IP, would result in "covari- ate shift" in synthetic data, leading to poor generalization for predictive models fitted to it, even when real and synthetic support coincide. As we can see in Figure 4(a), metrics that compare distributions (our metrics, PW and FID), are able to accurately rank the 4 generative models.
Hlpee-parameter tuning & the privacy-utility tradeoff. Another use case for our metric is hyper- parameter optimization for generative models. Here, we focus on the best-performing model in our previous experiment: ADS-GAN. This model has a hyper-parameter importance of the privacy-preservan (Yoon et al. (2020)): smaller values make the model more prone to overfitting, and hence privacy leakage. Figure 4(b) shows how our precision and authenticity metrics change with the different values of X: the curve provides an interpretable tradeoff between privacy and utility (c.g.. for À = 2, an A score of 0.4 means that 60% of patients may have personal information exposed). Increasing improves privacy at the expense of precision. By visualizing this tradeoff using our metric, data holders can understand the risks of different modeling choices involved in data synthesis.
E R that determines the loss function used to regularize the training of ADS-GAN
Improving synthetic data via model auditing. Our metrics are not only useful for tuning, but can also be used to improve the quality of synthetic data generated by an already-traines model using (post-hoc) auditing. Because our metrics are defined on the sample level, we can dis- card unauthentic or imprecise samples. This does not only lead nearly optimal precision and au- thenticity for the curated data (Figure 4(c)), but also improves the AUC-ROC of the predictive model fitted to audited data (from 0.76 0.78 for the audited ADS-GAN synthetic data, p < 0.005), since auditing eliminates noisy data points that would otherwise undermine generalization performance.
5.2 DIAGNOSING GENERATIVE DISTRIBUTIONS OF MNIST
In this experiment, we test the ability of our metrics to detect common modes of failure in generative modeling- in particular, we emulate mode dropping scenario, where the generative model fails to
8

---

Under review as a conference paper at ICLR 2022
(a) Diagnosing mode dropping in MNIST data
(b) Hide-and-seek competition
Wasserstein distance
FID
Predisoon Recall
Density Coverage
a-Precsiin & f-Recall
Hight--urhhenicc models
R,
2Rg 25%
A
D
95
model
F'rrpp
Parep
Fitees
Farop
IRa
Parop
0
O
Pdrop >> 0
IP,
0
Submission
2
0
2
7
9 9
D
5
6
8
0
I
2
15
Figure 5: (a) Diagnosing mode collapse in MNIST data. (b) Results for the hide-and-seek competition.
recognize the distinct modes in multimodal distribution Pr. and instead recovers a single mode in P.. To construct this scenario, we fit a conditional GAN (CGAN) model (Wang et al. (2018)) on the MNIST data set, and generate 1,000 samples for each of the digits 0-9. (We can think of each digit as a distinct mode P.,.) To apply mode dropping, we first sample 1,000 instances of each digit from the CGAN, and then delete individual samples of digits to with probability Parop, and replace the deleted samples with new samples of the digit 0 to complete a data set of 10,000 instances. The parameter Parop [0, 1] determines the severity of mode dropping: for Parop O, the data set has all digits being equally represented with 1. 1,000 samples, and for Parop = 1, the data set has 10,000 samples of the digit only as depicted pictorially in Figure 5()) (bottom panel).
We show how the different evaluation metrics respond to varying P Parop from 0 to I in Figure 5(a) (top). Because mode dropping pushes the generative distribution away from the real one, statistical distance metrics such as W and FID increase as Parop approaches 1. However, these metrics only reflect a discrepancy between P, and Pgg. and do not disentangle the Fidelity and Diversity compo- nents of this discrepancy. On the other hand, standard precision and recall metric are completely insensitive to mode dropping except for the extreme case when Parop = This is because both metrics only check supports of P,, and P. so they cannot recognize mode dropping as long as there a non-zen non-zero probability that the model will generates digits 1-9. On the contrary, mode dropping reflects in our metrics, which manifest in declining RRg as Parop increases. Since mode dropping affects coverage of digits and not the quality of images, only affects RR but not IPa.
5.3 REVISITING THE HIDE-AND-S CHALLENGE FOR SYNTHESIZING
DATA
Finally, we use our metric re-evaluate the generative models submitted to the NeurIPS 2020 Hide-and-Seck competition (Jordon et al. (2020)). In this competition, participants were required to synthesize intensive care time-series data based on real data from the AmsterdamUMCCd database. A total of 16 submissions were judged based on the accuracy of predictive models fit to the synthetic data (an approach similar to the one in Section 5. The submissions followed various modeling choices, including recurrent GANs, autoencoders, differential privacy GANs, etc. Details of all submissions are available online. Surprisingly, the winning submission was a very simplistic model that adds Gaussian noise to the real data to create samples.
To evaluate our metrics on time-series data, we trained a Seq-2-Seq embedding that is augmented with our One-class representations to transform time-series into fixed feature vectors. (The archi- tecture for this embedding is provided in the Supplementary material.) In Figure 5(b), we evaluate all submissions with respect to precision, recall and authenticity. As mission comes out as one of the least authentic models, despite performing competitively in terms of precision and recall. This highlights the detrimental impact of using naive metrics for evaluating generative models baadels-based on the competition results, clinical institutions seeking to create synthetic data sets may be led believe that Submission 1 in Figure 5(b) is the right model to use. However, our metrics-whicr give a fuller picture of the true quality of all model creates unauthentic samples that are mere noisy copies of real data, which would pose risk to patient privacy. We hope that our metrics and our pre-trained Seq-2-Seq embeddings can help clinical institutions evaluate the quality of their synthetic time-series data the future.
see, the winning sub-
peassiinnsshwwe that such

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ben Adlam, Charles Weill, and Amol Kapoor. Investigating under and overfitting in wasserstein generative adversarial networks. arXiv preprint arXiv: 1et::9910113331 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 1xXiv:1701.07875 2017.
Wasserstein gan. arXiv preprint
Pedro Baqui, Ioana Bica, Valerio Marra, Ari Ercole, and Mihaela van Der Schaar. Ethnic and regional variations in hospital mortality from covid-19 in brazil: cross-sectional observational study. The Lancet Global Health, 808xc1018--1126, 2020.
Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep represen- tations. In International conference on machine learning, pp. 552-560. PMLR, 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rXiv: 1109, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Peter Flach and Meelis Kull. Precision-recall-gann curves: neural information processing systems, 28:838 846, 2015.
analysis done right. Advances in
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27:2672 26800, 2014.
Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In Advances in neural information processing systems, pp. 1205-1213. Citeseer, 2012.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. Proceedings of the 3 International Conference on Neural Information Processing Systems, pp. 5769-5779, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to nash equilibrium. 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800 2002.
Aapo Hyvärinen, Jarmo Hurri, and Patrik O Hoyer. Estimation of non-normalized statistical models. In Natural Image Statistics, pp. 419-426. Springer, 2009.
James Jordon, Daniel Jarrett, Jinsung Yoon, Tavian Barnes, Paul Elbers, Patrick Thoral, Ari Ercole, Cheng Zhang, Danielle Belgrave, and Mihaela van der Schaar. Hide-and-seek privacy challenge. arXiv preprint arXiv:2007. 12087, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 1312v: 2013.
arXiv preprint
Tuomas Kynkäänniemi. Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In NeurIPS, 2019.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nnist// 1998.
10

---

Under review as a conference paper at ICLR 2022
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. Advances in neural information processing systems, 31:700- 2018.
Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. A non-parametric test to detect data- copying in generative models. arXiv preprint arXiv:2004.056775 05675, 2020.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reli- able fidelity and diversity metrics for generative models. International Conference Machine Learning (ICML), 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS- W, 2017.
Wolfgang Polonik. Minimum volume sets and generalized quantile processes. Stochastic processes and their applications, 69(1):1-24, 1997.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pp. 4393-4402. PMLR, 2018.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, Assessing generative models via precision and recall. Advances in Neural Information Processing Systems, 2018.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. arXiv preprint arviv: 1606004436 2016.
Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of a high-dimensional distribution. Neural computation, 13(7):1443-1411 2001.
Clayton D Scott and Robert D Nowak. Learning minimum volume sets. The Journal of Machine Learning Research, 7:665-704, 2006.
Viipe-Gripe.
http://plataformassuddd soriddegoo.b//eeeo www.br//ooooaiissdadds-aaarrrrre SIVEP- Gripe public dataset, (accessed May 10, 2020; in Portuguese), 2020.
In Ministry of Health.
Jascha Sohl-Dickstein, Peter B Battaglino, and Michael R DeWeese. New method for parameter estimation in probabilistic models: minimum probability flow. Physical review letters, 107()2):
220601, 2011.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video rep- resentations using Istms. In International conference machine learning, pp. 843-852, PMLR, 2015.
Dougal J Sutherland, Hsiao- Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. arXiv preprint arXiv: /6/ 1,04488, 2016.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. arXiv preprint vrXiv:1511.011
2015.
Harry I. Van Trees. Detection, estimation, and modulation theory, part I: detection, estimation, and linear modulation theory. John Wiley & Sons, 2004.
Fing-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8798 8807, 2018.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint arXiv: /802.06739, 2018.
Jinsung Yoon, Lydia N Drumright, and Mihaela Van Der Schaar. Anonymization through data syn- thesis using generative adversarial networks (ads-gan). IEEE Journal of Biomedical and Health Informatics, 2020.
11

---

Under review as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
APPENDIX A: LITERATURE REVIEW
In this Section, we provide comprehensive survey of prior work, along with detailed discussion on how our metric relates to existing ones. We classify existing metrics for evaluating generative models into two main classes:
1. Statistical divergence metrics
2. Precision and recall metrics
Divergence metries are single-valued measures of the distance between the real and generative dis- tributions, whereas precision-recat metrics classify real and generated samples as to whether they covered by generative and real distributions, respectively. In what follows, we list examples of these two types of metrics, highlighting their limitations.
Statistical divergence metries.
The most straightforward approach for evaluating a generative distribution is to compute the model rog-likelihood-ae for density estimation tasks, this has been the de-facto standard for training and evaluating generative models. However, the likelihood function is a modee-dependent criteria: this problematic because the likelihood of many tate-offtheeart models is inaccessible. For instance, GANs are implicit likelihood models and hence provide no explicit expression for its achieved log- likelihood Goodfellow et al. (2014). Other models, energy-based models has normalization con- stant in the likelihood expression that is generally difficult to compute as they require solving in- tractable complex integrals Kingma& Welling (2013).
Statistical divergence measures are alternative (model-independent) metrics that are related to log- likelihood, and are commonly used for training and evaluating generative models. Examples include lower bounds on the log-likelihood Kingma & Welling (2013), contrastive divergence and noise con- trastive estimation Hinton (2002); Gutmann & Hyvärinen (2010), probability flow Sohl-Dickstein et al. (201 1), score matching Hyvärinen et al. (2009), maximum mean discrepancy (MMD) Gretton et al. (2012), and the Tensen-Shannon divergence (JSD).
In general, statistical divergence measures suffer from the following limitations. The first limitation that likelihood-based measures can be inadequate in nigh-dimensional feature spaces. As has been shown in (Theis et al., 2015), one can construct scenarios with poor likelihood and great samples through a simple lookup table model, and vice versa, we can think of scenarios with great likelihood and poor samples. This is because, if the model samples white noise 99% of the time, and samples high-quality outputs 1% of the time, the log-likelihodl will be hardly distinguishable from a model that samples high-quality outputs 100% of the time if the data dimension large. Our metrics solve this problem by measuring the rate of error on a sample-level rather than evaluating the overall distribution of samples.
Moreover, statistical divergence measures collapse the different modes of failure of the generative distribution into single number. This hinders our ability diagnose the different modes of gener- ative model failures such as mode dropping, mode collapse, poor coverage, etc.
Precision and recall metries.
Precision and recall metrics for evaluating generative models were originally proposed in Sajjadi al. (2018). Our metrics differ from these metrics in various ways. First, unlike standard metrics, a-Precision and 3-Recall take into account not only the supports of IP, and P but also the actual probability densities of both distributions. Standard precision (and recall) correspond to one point on the Pa (and R3)) curve; they are equal to Pa and R3 evaluated on the full support (i.e., Pi and R1). By defining our metrics with respect to the a- and 8-supports, we do not treat all samples equally, but rather assign higher importance to samples that land in "denser" regions of S, and S,. Hence, P. and R3 reflect the extent to which P,, and P., are calibrated ie., good Pa and R3 curves are achieved when P, and P., share the same modes and not just common support. While optimal R and P1 can be achieved by arbitrarily mismatched P, and P,, our P, and Rg curves are optimized only when P. and y are identical stated by Theorem 1.
12

---

Under review as a conference paper at ICLR 2022
The new Pa and R, metrics address the major shortcomings of precision and recall. Among these shortcomings are: lack of robustness to outliers, failure to detect matching distributions, and inabil- ity to diagnose different types of distributional failure (such as mode collapse, mode invention, or density shifts) Naeem et al. (2020). Basically, a model Pg will score perfectly on precision and recall (R1=P==1) long it nails the support of P,, even if P. and P,, place totally different densities on their common support.
In addition the above, our metrics estimate the supports of real and generative distributions using neural networks rather than nearest neighbor estimates as in Naeem et al. (2020). This prevents our estimates from overestimating the supports of real and generative distributions, thereby overestimat- ing the coverage or quality of the generated samples.
APPENDIX B: PROOF OF THEOREM 1
To prove the statement of the Theorem, we need to prove the two following statements:
(1) Pr Pa/a- R3/B B (2) Pa/a= R3/B 1, Va, B + F = P,
To prove (1), we start by noting that since have P. Pr, then S Sa Va E [0, 1]. Thus, have
P S; = P(X, E Sg) = a.
(13)
for a E [0, 1]. and similarly, we have
R3 = P(, E St = P(X, E S. = B, for all B E [0, 1), which concludes condition (1).
(14)
Now we consider condition (2). We first note that S.; C Sa' for all d' > a. If P. for all then we have
/ dp, a, Va [0.1].
P(G, E S.)
(15)
Now assume that a' + Aa, then we have
d.p
dP, Aa.
(16)
Sw sw' 1Sy
Syl /Sp
Thus, the probability masses P and P, are equal for all infinitesimally small region Sp+ /S; (for Aa - 0) of the a-support of IP,,, hence P = Pr for all subsets of S.. By applying the similar argument the recall metric, we also have P, P. for all subsets of Sy. and hence P. P..
APPENDIX C: ALTERNATIVE APPROACH FOR ESTIMATING THE SUPPORT OF SYNTHETIC DATA & CODE SNIPPETS
Instead of using a À-NN approach to estimate the generative support S6, one could use a separate one-class representation & for each new synthetic sample being evaluated. We provide code snip- pets and comparisons between the two approaches in the an anonymized Colab notebook. While the two approaches perform rather similarly. opt to adopt the k-NN based approach to avoid potential biases induced by using a separate representation for each generative model when using metric for model comparisons.
APPENDIX D: EXPERIMENTAL DETAILS
.1 DATA
In this research the argue for the versatility of our metrics, hence we have included results for tabular (static), time-series and image data (see Table 1). For the tabular data Baqui al. (2020)'s
13

---

Under review as a conference paper at ICLR 2022
Table 1: Datasets used, with n and d the number samples and features, respectively.
Name
Type
7
d
Embedding
demb
SVEE-GGIPE
Tabular
6882
AmsterdamUMCdh
Time-series
7695
70
Seq-2-Seq
280
MNIST
Image 10000
784
Inception 2048
preprocessed version of the SIVEP-GRIPE dataset of Brazilian ICU Covid-19 19 patient data. For the image experiments, we use the 10,000 samples in the default MNIST test set LeCun (1998). For proper evaluation of the authenticity metric, the same original data is used for training of generative models and evaluation of all metrics.
For the time-series experiments, AmsterdamUMMcdb is used in manner exactly analogous to the NeurIPS 2020 Hide-and-Seck Privacy Challenge Jordon et al. (2020), which describes it as follows: "AmsterdamUUMcc was developed and released by Amsterdam UMC in the Netherlands and the European Society of Intensive Care Medicine (ESICM). It is the first freely accessible comprehen- sive and high resolution European intensive care database. It is also first to have addressed com- pliance with General Data Protection Regulation [...] Amsterdam contains approximately billion clinical data points related to 23,106 106 admissions 20, 109 unique patients between 2003 and 2016. The released data points include patient monitor and life support device data, laboratory measurements, clinical observations and scores, medical procedures and tasks, medication, fluid bal- ance, diagnosis groups and clinical patient outcomes.". Notably, only the longitudinal features from this datahase are kept, with static ones discarded. The same subset as was used in the competition for "hider" synthetic data generation is used; this consists of 7695 examples with 70 features (and time column), sequence length is limited to 100 (the original data contains sequences of up to length 135,337). The features are normalised to [0, 1] and imputed as follows: (1) back-fill, (2) forward-fill (3) feature median imputation. This preprocessing is chosen to match the competition Jordon et al. (2020). The competition "hider" submissions were trained on this dataset and the synthetic data generated.
For metric consistency and the avoidance of tedious architecture optimization for each data modality, we follow previous works (e.g. Heusel et al. (2017); Sajjadi et al. (2018); Kynkäääniemi et al. (2019); Naeem et al. (2020)) and embed image and time series data into a static embedding. This is required, since the original space is non-euclidean and will result in failure of most metrics. The static embedding is used for computing baseline metrics, and is used as input for the One-Class embedder.
For finding static representations of MNIST, images are upscaled and embedded using Inception pre-trained on ImageNET without top layer. This is the same embedder used for computing Frchet Inception Distance Heusel et al. (2017). Very similar results were obtained using instead a VGG- 16 embedder Brock al. (2018); Kynkäänniemi et al. (2019). Preliminary experimentation with random VGG-16 models Naeem et al. (2020) did not yield stable results for neither baselines nor our methods.
.2 TIME SERIES EMBEDDING
The time series embeddings used throughout this work based on Unsupervised Learning of Video Representations using LSTMs Srivastava et al. (2015), specifically the "LSTM Autoencoder Mode". sequence-tee seaeennceet---eeee sequence LSTM network is trained, with the target sequence set as the input sequence (reversed for ease of optimization), see Figure 1. The encoder hidden and cell states (h and vectors) the end of a sequence are used as the learned representation and are passed to the decoder during training. At inference, these concatenated to obtain one fixed-length vector per example.
The specifics of the LSTM autoencoder used here are as follows. Two LSTM layers are used in each encoder and decoder. The size of h, vectors is 70 (280 after concatenation). The model was implemented in Py Torch Paszke et al. (2017), utilising sequence packing for computational efficiency. All autoencoders were trained convergence the original data; the synthetic time series data was passed through this at inference. The time column (when present in data) was discarded.
14

---

Under review as a conference paper at ICLR 2022
Leamed Representation
Wa
Figure 1: Architecture of LSTM Autoencoder, sourced from Srivastava et al. (2015).
Table Metries on tabular data for different generative models. Row "audited" contains results for data generated by ADS-GAN, but in which samples are rejected if they do not meet the precision or authenticity threshold.
Model
W
FD
Parzen
Pi
R1
D
C
A
IPa
1 IRs
VAE 2.2626
2.9697
-11.3653
1.0000
0.1022
1.3999
0.0926
0.5147
0.4774
0.0596
GAN
1,5790
1.8065
-10.7824
0.7059
0.0875
0.4316
0.0939
0.6077
0.9802
0.0648
WGAN-GP
-0.0194
0.0856
-8.3650
0.9439
0.7299
1.0709
0.8945
0.4712
0.9398
0.4468
ADS-GAN
0.3578
0.2134
-8.7952
0.8083
0.6133
0.4711
0.5357
0.5905
0.7744
0.2914
DPGAN
1.1216
0.9389
-8.8394
0.9923
0.1822
1.4885
0.5065
0.3793
0.9591
0.1863
audited
-0.0470
0.0600
-8.5408
0.8986
0.8737
0.7560
0.8050
1.0000
0.9994
0.1961
.3
FULL RESULTS
Table contains metrics computed different generated versions of the SIVEP -VEE-GGRPE tabular dataset. Included metrics are Wasserstein distance, Frchet Distance (FD), Parzen window likeli- hood estimate, precision Pl. recall R2 density (D), coverage C and the proposed metrics, specifi- cally integrated a-precision IP... integrated 3-recall I RRg and authenticity A For the tabular data, data generated using a VAE, GAN, Wasserstein GAN with gradient penalisation (WGAN-GP) Ar- jovsky et al. (2,17), ADS-GAN Yoon et al. (2020), Differentially Private GAN (DP-GAN) Xie et al. (2018) and an ADS-GAN GAN generated dataset in which samples are audited on precision and authen- ticity. Similarly, Table contains metric results for MNIST, generated by VAE, Deep convolution GAN (DCGAN), WGAN-GP and ADS-GAN. Table contains results for MIMIC generation using different methods from the Hide-and-Seed Privacy Competition Jordon et al. (2020). The submis- sion that won the competition is the penultimate model, Hamada. The last row shows results for an audited version of the Hamada dataset, in which keep generating data using the Hamada model and discard samples that do not meet the precision or authenticity threshold.
Note that here, the Frehet Distance equals the Frehet Inception Distance, because the metrics are computed on Inception embeddings of the image data.
Table 3: Metrics on MNIST data for different generative models.
Model
W
FD
Parzen
P
R:
D
C
A
IP.
IR
VAE
606.5
112934
349913
0.2160
0.0140
0,0885
0.0810
0.8167
0.4280
0.1452
DCGAN
-98.5
2578
-180132
0.8947
0.8785
0.8589
0.9071
0.6059
0.9889
0.4815
WGAN-GP
-64.8
4910
-185745
0.8931
0.8504
0.8084
0.8509
0.6146
0.9894
0.4199
ADS-GAN
-114.1
574
-28657
1.0000
0.9998
1.1231
1,0000
0.5268
0.9900
0.5549
15

---

Under review as a conference paper at ICLR 2022
Table 4: Metrics on AmsterdamUMCdd data for different generative models.
Model
W
FD
Parzen
Pt
R1
D
C
A
IPo
RR3
(CodaLab username)
Add noise [baseline]
40.6
497
-228
0.0013
0.0173
0.0003
0.0010
0.8450
0.5187
0.0148
Time-GAN [baseline]
249.0
30478
-1404
0.0000
0.6642
0.0000
0.0000
0.9991
0.5000
0,0005
akashdeepsingh
13.3
44
-34
0.7643
0.3566
0.4263
0,0741
0.3958
0.6074
0,1408
sacedsa
93.4
1509
-2597
0,0000
0.0000
0.0000
0.0000
0.9925
0,5018
0.0025
wangzq312
62.3
645
-119
0.4430
0.8630
0.0920
0.0061
0.7665
0.6116
0.0185
esetraynor
64.6
1604
-4710
0.0000
0.0000
0.0000
0.0000
1.0000
0.5339
0.0000
jilljenn
118.0
17235
0.0000
0.0000
0,0000
0.0000
1.0000
0,4999
0.0000
SatoshiHasegawa
2.4
-156
0.9333
0.3264
0.5626
0.0988
0.3172
0.8050
0.1717
flynngo
126.3
8786
-120
0.0359
0.7663
0.0087
0.0022
0.8448
0.6497
0.0271
swwaanchhiken-wrrpp
118.7
5344
-892
0.0000
0.0535
0.0000
0.0000
0,9854
0.5030
0.0053
Atrin
175.3
15159
-1933
0.0000
0.7306
0.0000
0.0000
0.9975
0.5008
0.0019
wangz10
113.8
4624
-221
0.0039
0.2993
0.0010
0,0019
0.8509
0.5141
0,0113
yingruiz
146.7
8973
-562
0.0069
0.0626
0.0014
0.0001
0.8995
0.5129
0.0062
yingjialin
138,9
8919
-570
0.0000
0.0613
0.0000
0.0000
0.8864
0,5011
0.0030
lumip
25.8
271
-78
0.1796
0.4749
0.0727
0.0256
0.6335
0.5938
0.0801
hamada
8.3
73
-192
0.8933
0.4010
0,5906
0.0398
0.3572
0,5482
0.0850
hamada [audited]
-204
0,9006
0.0665
0.5931
0.0444
0.9976
0.0334
.4
HYPERPARAMETER OPTIMIZATION
.4.1 BASELINES
For computing the density and coverage metrics, we set a threshold of 0.95 on the minimum expected coverage. as recommended in the original work (Eq. 9 Naeem et al. (2020)). For all datasets, this is achieved for & 5. For consistency these comparisons, we use for the precision and recall metrics too.
.4.2 ONECLASS EMBEDDINGS
We use Deep SVDD Ruff et al. (2018) to embed static data into One-Class representations. To mitigate hypersphere collapse (Propostions 2 and 3 of Ruff et al. (2018)), we do not include bias term and use ReLU activation for the One-Class embedder. Original data is split into training (80%) and validation (20%) set, and One-Class design is fine-tuned to minimise validation loss. We use the SoftBoundary objective (Eq. 3 Ruff et al. (2018)) with v = 0.01 and center c = 1 for tabular and time-series data and = 10 1 for image data. Let n4 be the number of hidden layers with each d, nodes, and let dg be the dimension of the representation layer. For tabular data, we use n4 = 3, dn = 32 and d- 25; for time-series data, 71, 2, dn 128 and d₂ 32: and for MNIST n& 3, d, = 128 and dg = 32. Models are implemented in PoTorc Paszke et al. (2017) and the Adam W optimizer used with weight decay 1
For the 8-recall metric, estimating the support of synthetic data involves tuning the k parameter of the K-NN estimator. The k parameter can be tuned by fitting the NN estimator on a portion of the data for every given k. and then testing the recall on held out (real) sample. The selected i for each a is the smallest that covers held out samples. Similar to Naeem al. (2020), we found that through this procedure, k = 5 seems to come up the optimal & for most experiments.
.5 TOY EXPERIMENTS
We include two toy experiments that highlight the advantage of the proposed metrics compared to previous works. We focus our comparison on the improved precision and recall Kynkäänniemi et al. (2019) and density and coverage Nacem et al. (2020) metrics.
.5.1 ROBUSTNESS TO OUTLIERS
Nacem et al. (2020) showed that the precision and recall metrics as proposed by Sajjadi et al. (2018); Kynkäänniemi et al. (2019) are not robust to outliers. We replicate toy experiments to show the proposed Preecission and 3-Recall do not suffer the same fate.
16

---

Under review as a conference paper at ICLR 2022
Let X, R denote original and synthetic samples respectively, with original X ~ (() and
~ N(,,1) I). We compute all metrics for μl E [-1,1]. In this setting we conduct three experiments:
1. No outliers
2. One outlier in the real data X
3. One outlier the synthetic data at
We set d 64 and both original and synthetic data sample 10000 points. Subsequent metric scores are shown in Figure
0.0
No outller
No outler
0.7
Outler
Outher
Outlier
0.6
0.5
0.4
6
0.3
0.2
0.1
8.0
-0.75
-0.50
-0.25
0.25
0.75
1.00
-1.00
-0.75
-0.50
-0.25
0.00
0.50
1.00
(a) Precision
(b) Recall
0.8
0.7
0.35
0.5
0.25
0.4
0.20
0.3
0.15
0.2
0.30
0.1
Octlier real Octlier faka
Dutlier in
Outieer in fake
0.0
-0.75
0.5
0.00
(o) IPa
(d) IRo
Figure 2: Toy experiment I: outlier robustnes
As can be seen, the precision and recall metrics are not robust to outliers, as just a single outlier has dramatic effects. The IP. and RRg are not affected, as the outlier does not belong the a-support (or 3 suppoott) unless (or (3) is large.
.5.2 MODE RESOLUTION
The precision and recall metrics only take into account the support of original and synthetic data, but not the actual densities. The density and coverage metric take this into account, but here we show these are not able to capture this well enough to distinguish similar distributions.
In this experiment we look at mode resolution: how well is the metric able to distinguish a single mode from two modes? Let the original distribution be a mixture of two gaussians that are separated by distance z and have 1,
X
2 N 1) + N(+
2
and let the synthetic data be given by
Y N 1 1²)
17

---

Under review as a conference paper at ICLR 2022
This situation would arise if synthetic data generator fails to distinguish the two nodes, and instead tries to capture the two close-by by modes of the original distribution using a single mode. We compute metries for fu [0, 5].
0.0
0.8
0.6
0.2
02-
0.0
(a) Precision
(b) Recall
0.6
0.6
0.4
0.2
02
(c) Density
(d) Coverage
0.8
0.6
0.0
(c) IP.
(f) IR3
Figure 3: Toy experiment mode resolution
As can be seen, neither P&R D&C notice that the synthetic data only consists of single mode, whereas the original data consisted of two. The a-precision metric is able to capture this metric: for small the a-support of the original distribution centred around the two separated, and does not contain the space that separates the modes (i.e. the mode of the synthetic data).
18


Reviews:
[{"review_id": "oHKYgyZi5B2", "reviewer": "Reviewer_hx4h", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The paper addresses an important topic that is time critical and relevant to the generative modeling community and the broader sphere of AI. It is well written and motivated and a pleasure to read overall. The authors clearly point out the issues that are lacking with existing metrics and motivate the need to define the new versions of precision / recall metrics. Moreover, the metrics enable a new use case of model auditing which can be very relevant is safety critical applications.\n\nWeaknesses: \n1. The paper promises a lot in terms of the defined metrics but the evaluations are bit underwhelming. The experiment with the predictive model depends on the downstream classifier as well - the performance of the generative model is tied to the performance of the classifier - hence the hyper-parameter settings and training settings for the classifier should be explained clearly. \n2. Another aspect that is missing from the evaluation is the application of model auditing on real life critical data. The idea of model auditing becomes useful mainly in cases where we need to filter at the sample level. When a generative model is trained on medical data (imaging) how do the metrics help in model auditing?\n3. An interesting question to probe is how to compare different generative models that have comparative expertise? For instance, one model could be an expert in color and another could be in shape and so on - can we use the defined metrics to identify such behaviors?\n4. Please provide error bars on the predictive modeling experiments - as in, account for the randomness in the generative model training process.", "summary_of_the_paper": "This work addressed an interesting problem and a very relevant one - how to audit generative models? The most popular metric to evaluate generative models is FID but it is extremely opaque and population based. This paper proposes new fidelity metrics that overcome both these challenges. While the definition of these metrics are well grounded, their measurement process itself - which is done via trained NN classifiers - is a bit unconvincing. The authors demonstrate that the proposed metric can recover the real ranking of generative models on a synthetic data generation task and can discover issues like mode collapse.", "summary_of_the_review": "Overall, the paper does a good job in addressing a relevant problem in a well motivated manner. Even though there are some open concerns regarding the evaluation, the fact that the current approach is one of the first to facilitate model auditing for generative models makes this a good contribution. Authors, please address my concerns in the weaknesses listed above.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W87Lrv5bSPn", "reviewer": "Reviewer_qovd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Well written, easy to follow.\n2. Timely topic.\n3. Discussion of NeurIPS challenge is important.\n\nWeaknesses:\n\n1. I did not learn something substantial from reading the paper.\n2. The claims are too strong.\n\nDetailed comments:\n\n1. The generalized definitions of precision and recall are effective markers of understanding synthetic data quality. However, empirically they do not outperform other distribution measuring mechanisms (such as FID, PW, W) and I suspect methods like KL divergence or MMD, or TV distance will also perform comparably. \n2. To my understanding, approaches like FID, KL Divergence, MMD, TV distance are also “model agnostic”; I humbly request the authors to rethink the positioning of their work along this particular claim.\n3. The appeal in this work lies in the fact that the proposed measures can be calculated on a per-sample basis, unlike prior works. \n4. Empirically, the authors consider few issues which may result in poor fidelity/diversity (mode collapse being one of them). If the models did not generalize during training, how do these methods capture them?\n5. Could the authors clarify how the proposed method provides interpretability? The authors observe correlations between induced failures and their scores, but they can’t really attribute the score to a particular failure event?\n6. The proposed measure for authenticity relies on the ability to detect overfitting. However, recent work by Feldman et al. argues that overfitting (or memorization, in some cases) is essential for generalization. Can the authors comment on the same?\n", "summary_of_the_paper": "The authors propose a mechanism to calculate the \"goodness\" of a generative model on a per-sample level; this method is model agnostic. This can be used to audit the model post-hoc. ", "summary_of_the_review": "Key requirements: I would urge the authors to empirically validate if overfitting has occurred to validate their claims (maybe through MI attacks).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T3D-mCoOpzo", "reviewer": "Reviewer_AgxQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Clear writing and demonstrations.\n+ Meaningful research topic.\n+ Technically reproducible.\n\nWeaknesses:\n- The technical novelty is marginally revised from Sajjadi et al. 2018.\n- The experiments are incomplete and therefore unconvincing.\n  - The testing datasets are a bit toy-like. Like Sajjadi et al. 2018, please also experiment on CelebA [1] or FFHQ [2]. For mode collapse evaluation, please consider using attribute classifiers pre-trained on CelebA, as well as IvOM [3].\n  - The testing generative models are out of date. Results on old-fashioned architectures are not conclusive for cutting-edge research works. Imagine our community works on a new GAN paradigm and targets to outperforming StyleGAN3 [1], are we convinced to use the proposed metric which has only been validated on toy data/models? For GAN, please consider validating on StyleGAN3 [4]. For VAE, please consider validating on VQ-VAE-2 [5].\n\n[1] Liu, Ziwei, et al. \"Deep learning face attributes in the wild.\" ICCV 2015.\n\n[2] Karras, Tero, Samuli Laine, and Timo Aila. \"A style-based generator architecture for generative adversarial networks.\" CVPR 2019.\n\n[3] Metz, Luke, et al. \"Unrolled generative adversarial networks.\" ICLR 2017.\n\n[4] Karras, Tero, et al. \"Alias-free generative adversarial networks.\" NeurIPS 2021.\n\n[5] Razavi, Ali, Aaron van den Oord, and Oriol Vinyals. \"Generating diverse high-fidelity images with vq-vae-2.\" NeurIPS 2019.", "summary_of_the_paper": "This paper targets to a 3-dim metric: alpha-Precision, beta-Recall, and Authenticity, that quantifies the fidelity, diversity, and generalization performance of a generative model. The proposed metric serves as soft-boundary classifiers between real and generated spherical-shaped supports, and improves the robustness against outliers and generation failure cases. This sample-wise metric can audit models by judging individual synthetic samples by their quality. ", "summary_of_the_review": "See the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6HV7oueO223", "reviewer": "Reviewer_caMn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper contributes useful thought and discussion for how best to argue the holistic performance of a generative model, contrasting with FID and other domain-specific approaches.  There are novel perspectives in this paper that are worth sharing with the community, particularly those contributing to discussion around generalization vs. memorization (copying) where it is shown that prior metric ignoring this aspect otherwise indicate performant models (c.f. time-series NeurIPS challenge data).  The paper could be improved by drawing deeper relationship with prior thought in this area e.g. Adlam et al. (2019), Meehan et al. (2020) who are cited in Sec 3.2.2 but not discused.  Indeed the brief literature review is relegated to the paper Appendix and was not helpful in arguing for the novel contribution of the paper in context of such work.  The paper also lacks any reflection or conclusion on the limitations of the proposed metrics.  It was unclear to me how the proposed authenticity classifier would work for multi-modal distribution given the assumptions of noise within a hypersphere.  It did not seem like it would be practical for realistic, complex data problems.  A use case on high-dimensional data e.g. images beyond a toy MNIST example would have better demonstrated the practical utility of this work for data where the need of such performance metrics is clearer.", "summary_of_the_paper": "The paper presents a methodology assessing the performance of a generative model in a domain agnostic fashion.  A 3-dimensional metric space is proposed: fidelity (output quality), diversity (coverage of expected variability of output) and generalization (to what degree model avoids memorizing training data i.e. is truly generative) is proposed.  Particularly the latter element is novel and the manner in which it is may be evaluated by developing an `authenticity’ measure for the task. Three illustrative use cases in image (mnist) and medical patient data (COVID-19, `Hide and Seek’ seq2seq data) domains are provided.", "summary_of_the_review": "I am on the borderline tending toward accepting the paper given the interesting discussion around authenticity and memorization, but I am not fully convinced this is a practically useful paper.  \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new set of metrics for evaluation of generative models based on the well-established precision-recall framework, and an additional dimension quantifying the degree of memorization. The authors evaluated the proposed approach in several settings and compared it to a subset of the classic evaluation measures in this space. The reviewers agreed that this is an important and challenging problem relevant to the generative modeling community at large. The paper is well-written and the proposed method and motivation are clearly explained. 

The initial reviews were borderline, and after the discussion phase we have 2 borderline accepts, one strong accept, and one strong reject. After reading the manuscript, the rebuttal, and the discussion, I feel that the work should not be accepted on the grounds of insufficient empirical validation. Establishing a new evaluation metric is a very challenging task -- one needs to demonstrate the pitfalls of existing metrics, as well as how the new metric is capturing the missing dimensions in a thorough empirical validation. While the former was somewhat shown in this work (and in many other works), the latter was not fully demonstrated. The primary reason is the use of a non-standard benchmark to evaluate the utility of the proposed metrics. I agree that covering a broader set of tasks and models makes sense in general, but it shouldn’t be done at the cost of existing, well-understood benchmarks. I expected to see a thorough comparison with [1], one of the most practical metrics used today which can be easily extended to all settings considered in this work (notwithstanding the drawbacks outlined in [2]). What are the additional insights? What is [1] failing to capture in practical instances? Does the rank correlation change with respect to modern models across classic datasets (beyond MNIST and CIFAR10)? This would remove confounding variables and significantly strengthen the paper.

My final assessment is that this work is borderline, but below the acceptance bar for ICLR. I strongly suggest the authors to showcase the additional improvements over methods such as [1] in practical and well-understood settings commonly used to benchmark generative models (e.g. on images). The experiments suggested by the reviewers are a step in the right direction, but not sufficient.

[1] Improved Precision and Recall Metric for Assessing Generative Models. Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, Timo Aila. NeurIPS ’19

[2] Evaluating generative models using divergence frontiers. 
Josip Djolonga, Mario Lučić, Marco Cuturi, Olivier Frederic Bachem, Olivier Bousquet, Sylvain Gelly. AISTATS ‘20
Decision: Reject