AUDITING GENERATIVE MODELS
Year: 2021

Abstract:
Devising domainand model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional metric, (α-Precision, β-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample-level and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional dimension for model performance that quantifies the extent to which a model copies training data—a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.

text: Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional metric, (α-Precision, β-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample-level and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional dimension for model performance that quantifies the extent to which a model copies training data—a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.

heading: 1 INTRODUCTION
text: Intuitively, it would seem that evaluating the likelihood function of a generative model is all it takes to assess its performance. As it turns out, the problem of evaluating generative models is far more complicated. This is not only because state-of-the-art models, such as Variational Autoencoders (VAE) (Kingma & Welling (2013)) and Generative Adversarial Networks (GANs) (Goodfellow et al. (2014)), do not possess tractable likelihood functions, but also because the likelihood score itself is a flawed measure of performance—it scales badly in high dimensions, and it obscures distinct modes of model failure into a single uninterpretable score (Theis et al. (2015)). Absent domain-agnostic metrics, earlier work focused on crafting domain-specific scores, e.g., Inception score (Salimans et al. (2016)), with an exclusive emphasis on image data (Lucic et al. (2018)).
In this paper, we introduce an alternative approach to evaluating generative models, where instead of assessing the generative distribution by looking at all synthetic samples collectively to compute likelihood or divergence, we classify each sample individually as being of high or low quality. In this way, our metric comprises interpretable probabilistic quantities—resembling those used to evaluate discriminative models (e.g., AUC-ROC)—which describe the rate by which a model makes errors. When averaged over all samples, our sample-level scores reflect discrepancy between real and generative distributions in a way similar to statistical divergence measures (e.g., KL divergence, Fréchet distance (Heusel et al. (2017)), or maximum mean discrepancy (Sutherland et al. (2016)). In this sense, our metric enables diagnosing model performance on both the sample and distribution levels.
Our metric represents the performance of a generative model as a point in a 3-dimensional space— each dimension corresponds to a distinct quality of the model. These qualities are: Fidelity, Diversity and Generalization. Fidelity measures the quality of a model’s synthetic samples, and Diversity is
Figure 1: Pictorial depiction for the proposed evaluation metrics. The blue and red spheres correspond to the α- and β-supports of real and generative distributions, respectively. Blue and red points correspond to real and synthetic data. (a) Synthetic data falling outside the blue sphere will look unrealistic or noisy. (b) Overfitted models can generate ostensibly high-quality data samples that are “unauthentic” because they are copied from the training data. (c) High-quality data samples should reside inside the blue sphere. (d) Outliers do not count in the β-Recall metric. (Here, α=β=0.9, α-Precision=8/9, βRecall = 4/9, and Authenticity = 9/10.)
Support of real distribution Support of synthetic
distribution
Real Synthetic
Noisy synthetic sample
Unauthentic high-quality sample
Authentic high-quality
sample Outlier
(a)
(b)
(c)
(d)
the extent to which these samples cover the full variability of real samples, whereas Generalization quantifies the extent to which a model overfits (copies) training data.
How do we quantify the 3 dimensions of performance? We build on the precision-recall analysis framework proposed in (Sajjadi et al. (2018)), and introduce the α-Precision and β-Recall metrics to quantify model Fidelity and Diversity, respectively. Both metrics assume that a fraction 1 − α (or 1 − β) of the real (and synthetic) data are “outliers”, and α (or β) are “typical”. α-Precision is the fraction of synthetic samples that resemble the “most typical” α real samples, whereas β-Recall is the fraction of real samples covered by the most typical β synthetic samples. α-Precision and β-Recall are evaluated for all α, β ∈ [0, 1], providing entire precision and recall curves instead of single numbers. To compute both metrics, we embed the (real and synthetic) data into hyperspheres with most samples concentrated around the centers, i.e., the real and generative distributions (Pr and Pg) has spherical-shaped supports. In this transformed feature space, typical samples would be located near the centers of the spheres, whereas outliers would be closer to the boundaries.
To quantify Generalization, we introduce the Authenticity metric, which reflects the likelihood of a synthetic sample being copied from training data. We derive the Authenticity metric from a hypothesis test for data copying based on the observed proximity of synthetic samples to real ones in the embedded feature space. A pictorial illustration for all metrics is shown in Figure 1.
How is our metric different? If one think of standard precision and recall metrics as “hard” binary classifiers of real and synthetic samples, our α-Precision and β-Recall can be thought of as softboundary classifiers that do not only compare the supports of Pr and Pg , but also assesses whether both distributions are calibrated. Precision and recall metrics are special cases of α-Precision and β-Recall for α = β = 1. As we show later, our new metric definitions solve many of the drawbacks of standard precision-recall analysis, such as lack of robustness to outliers and failure to detect distributional mismatches (Naeem et al. (2020)). They also enable detailed diagnostics of different types of model failure, such as mode collapse and mode invention. Moreover, optimal values of our metrics are achieved only when Pr and Pg are identical, thereby eliminating the need to augment the evaluation procedure with additional measures of statistical divergence (e.g., KL divergence).
Previous works relied on pre-trained embeddings (using ImageNet feature extractors (Deng et al. (2009))). In this work, we propose feature embeddings that are model- and domain-agnostic, and are tailored to our metric definitions and data set at hand. Our proposed feature embedding step can be completely bespoke to raw data, or augmented with pre-trained embeddings. This enables our metric to remain operable in application domains where no pre-trained representations exist.
Overfitting is a crucial mode of failure of generative models, especially when modeling sensitive data (e.g., clinical data) for which data copying may violate privacy requirements (Yoon et al. (2020)), but it has been overlooked in previous works which focused exclusively on quantifying the FidelityDiversity characterization (Brock et al. (2018)). As we show in our experiments (Section 5), because our metric accounts for Generalization, it can provide a fuller picture of a generative model’s performance. Precisely, we show that some of the celebrated generative models score highly for Fidelity
and Diversity simply because they memorize real samples, rendering them inappropriate for privacysensitive applications. A comprehensive survey of prior work is provided in the Appendix.
Model auditing as a novel use case. The sample-level nature of our metrics inspires the new use case of model auditing, wherein we judge individual synthetic samples by their quality, and reject samples that have low Fidelity or are unauthentic. In Section 5, we show that model audits can improve the outputs of a black-box model in a post-hoc fashion without any modifications to the model itself, and demonstrate this use case in synthesizing clinical data for COVID-19 patients.

heading: 2 EVALUATING AND AUDITING GENERATIVE MODELS

heading: 2.1 PROBLEM SETUP
text: We denote real and generated data asXr ∼ Pr andXg ∼ Pg , respectively, whereXr, Xg ∈ X , with Pr and Pg being the real and generative distributions, and X being the input space. The generative distribution, Pg , is estimated using a generative model (e.g., GAN). The real and synthetic data sets are Dreal = {Xr,i}ni=1 and Dsynth = {Xg,j}mj=1, where Xr,i ∼ Pr and Xg,j ∼ Pg .

heading: 2.2 WHAT MAKES A GOOD SYNTHETIC DATA SET?
text: Our goal is to construct a metric E(Dreal,Dsynth) for the quality of Dsynth in order to (i) evaluate the performance of the underlying generative model Pg , and (ii) audit the model outputs by discarding (individual) “low-quality” samples, thereby improving the overall quality ofDsynth. In order for E to fulfill the evaluation and auditing tasks, it must satisfy the following desiderata: (1) it should be able to disentangle the different modes of failure of Pg through interpretable measures of performance, and (2) it should be sample-wise computable, i.e., we should be able to tell if a given (individual) synthetic sample Xg ∼ Pg is of a low quality. Having outlined the desiderata for our sought-after evaluation metric, we now propose three qualities of synthetic data that the metric E should be able to quantify. Failure to fulfill any of these three qualities correspond to independent modes of failure of the model Pg . These qualities are:
1. Fidelity—the generated samples resemble real samples from Pr. A high-fidelity synthetic data set should contain “realistic” samples, e.g. visually-realistic images.
2. Diversity—the generated samples are diverse enough to cover the variability of real data, i.e., a model should be able to generate a wide variety of good samples.
3. Generalization—the generated samples should not be mere copies of the (real) samples in training data, i.e., models that overfit to Dreal are not truly “generative”.
In Section 3, we propose a three-dimensional evaluation metric E that captures all of the qualities above. Our proposed metric can be succinctly described as follows:
E , (α-Precision︸ ︷︷ ︸ Fidelity , β-Recall︸ ︷︷ ︸ Diversity , Authenticity︸ ︷︷ ︸ Generalization ). (1)
The α-Precision and β-Recall metrics are generalizations of the conventional notions of precision and recall used in binary classification analysis (Flach & Kull (2015)). Precision measures the rate by which the model synthesizes “realistic-looking” samples, whereas the recall measures the fraction of real samples that are covered by Pg . The authenticity score measures the fraction of synthetic samples that are invented by the model and not copied from the training data.

heading: 2.3 EVALUATION AND AUDITING PIPELINES
text: Having provided a bird’s-eye view of our proposed metric E , we now briefly summarize the steps involved in the evaluation and auditing tasks. Since statistical comparisons of complex data types in the raw input space X are difficult, the evaluation pipeline starts by embedding Xr and Xg into a “meaningful” feature space through a representation Φ, dubbed the evaluation embedding, and then computing E on the embedded features (see Figure 2(a)). In Section 4, we propose a representation learning approach to construct embeddings tailored to our metric and the data set at hand.
In the (post-hoc) model auditing task, we compute the sample-level metrics for each Xg,j in Dsynth, and discard samples with low authenticity and/or precision scores, which results in a “curated” synthetic data set with an improved overall performance. When granted direct access to the model Pg , the auditor serves as a rejection sampler that repeatedly draws samples from Pg , only accepting ones with high precision and authenticity (Figure 2(b)). Model auditing is possible through our metrics as they can be used to evaluate the quality of individual synthetic samples; the same task cannot be carried out with statistical divergence measures that compare the overall real and generative distributions.
3 α-PRECISION, β-RECALL AND AUTHENTICITY

heading: 3.1 DEFINITIONS AND NOTATIONS
text: Let X̃r = Φ(Xr) and X̃g = Φ(Xg) be the embedded real and synthetic features. For simplicity, we will use Pr and Pg to refer to the distributions over the raw and embedded features interchangeably. Let Sr = supp(Pr) and Sg = supp(Pg), where supp(P) is the support of P. Central to our proposed metrics is a more general notion for the support of a distribution P, which we dub the α-support. We define the α-support as the smallest subset of S = supp(P) supporting a probability mass α, i.e.,
Sα , min s⊆S V (s), s.t. P(s) = α, (2)
where V (s) is the volume (Lebesgue measure) of s, and α ∈ [0, 1]. One can think of an α-support as dividing the full support of P into “normal” samples concentrated in Sα, and “outliers” residing in S̄α, where S = Sα∪S̄α. The notion of α-support is also known as the minimum volume set, and has been traditionally used in outlier detection models (Polonik (1997); Scott & Nowak (2006)).
Finally, we define the distance between a data sample X and the training data Dreal as the distance between X and the closest sample in Dreal, i.e.,
d(X,Dreal) = min 1≤i≤n d(X,Xr,i), (3)
where d is a distance metric defined over the input space X .

heading: 3.2 SAMPLE-LEVEL EVALUATION METRICS
text: 3.2.1 α-PRECISION AND β-RECALL
α-Precision. The conventional Precision metric is defined as the probability that a generated sample is supported by the real distribution, i.e. P(X̃g ∈ Sr) (Sajjadi et al. (2018)). We propose a more refined measure of sample fidelity, dubbed the α-Precision (denoted as Pα), defined as follows:
Pα , P(X̃g ∈ Sαr ), for α ∈ [0, 1]. (4)
That is, Pα is the probability that a synthetic sample resides in the α-support of the real distribution.
β-Recall. To assess diversity in synthetic data, we propose the β-Recall metric as a generalization of the conventional Recall metric. Formally, we define the β-Recall as follows:
Rβ , P(X̃r ∈ Sβg ), for β ∈ [0, 1], (5)
i.e., Rβ is the fraction of real samples that reside in the β-support of the generative distribution.
Interpreting α-Precision and β-Recall. To interpret (4) and (5), we first need to revisit the notion of α-support. From (2), we know that an α-support hosts the most densely packed probability mass α in a distribution, hence Sαr and Sβg always concentrate around the modes of Pr and Pg (Figure 3); samples residing outside of Sαr and Sβg can be thought of as outliers. In this sense, Pα andRβ do not count outliers when assessing fidelity and diversity. That is, the α-Precision score deems a synthetic sample to be of a high fidelity not only if it looks “realistic”, but also if it looks “typical”. Similarly, β-Recall counts a real sample as being covered by Pg only if it is not an outlier in Pg . By sweeping the values of α and β from 0 to 1, we obtain a varying definition of which samples are typical and which are outliers—this gives us entire Pα and Rβ curves as illustrated in Figure 3.
Generalizing precision-recall analysis. Unlike standard precision and recall, Pα and Rβ take into account not only the supports of Pr and Pg , but also their densities. Standard precision (and recall) correspond to one point on the Pα (and Rβ) curve; they coincide with Pα and Rβ evaluated on the full supports (i.e., P1 and R1). By defining our metrics with respect to the α- and β-supports, we do not treat all samples equally, but assign higher importance to samples in “denser” regions of Sr and Sg . Pα and Rβ reflect the extent to which Pr and Pg are calibrated— i.e., good Pα and Rβ are achieved when Pr and Pg share the same modes and not just a common support.
Our proposed Pα and Rβ metrics address major shortcomings of the commonly used P1 and R1, among these are: lack of robustness to outliers, failure to detect matching distributions, and inability to diagnose different types of distributional failure (Naeem et al. (2020)). Basically, Pg will score perfectly on precision and recall (R1=P1=1) as long as it nails the support of Pr, even if Pr and Pg place totally different densities on their common support. Figure 3 illustrates how our metrics remedy these shortcomings. While optimal R1 and P1 are achieved by arbitrarily mismatched densities, our Pα and Rβ curves are optimized only when Pr and Pg are identical as stated by Theorem 1.
Theorem 1. The α-Precision and β-Recall satisfy the condition Pα/α = Rβ/β = 1, ∀α, β, if and only if the generative and real densities are identical, i.e., Pg = Pr.
That is, a model is optimal if and only if its Pα and Rβ are both straight lines with unity slopes.
Measuring statistical discrepancy with Pα and Rβ. While the Pα and Rβ curves provide a detailed view on a model’s fidelity and diversity, it is often more convenient to summarize performance in a single number. To this end, we define the mean absolute deviation of Pα and Rβ as:
∆Pα = ∫ 1 0 |Pα − α| dα, ∆Rβ = ∫ 1 0 |Rβ − β| dβ, (6)
where ∆Pα ∈ [0, 1/2] and ∆Rβ ∈ [0, 1/2] quantify the extent to which Pα and Rβ deviate from their optimal values. We define the integrated Pα and Rβ metrics as IPα = 1− 2∆Pα and IRβ = 1− 2∆Rβ , both take values in [0, 1]. From Theorem 1, IPα = IRβ = 1 only if Pg = Pr. Together, IPα and IRβ serve as a measure of the discrepancy between the distributions Pr and Pg , eliminating the need to augment our precision-recall analysis with measures of statistical divergence. Moreover, unlike f -divergence measures, the (IPα, IRβ) metric disentangles fidelity and diversity into separate components, and does not require that Pr and Pg share a common support.

heading: 3.2.2 AUTHENTICITY
text: Generalization is independent of precision and recall since a model can achieve perfect fidelity and diversity without truly generating any samples, simply by resampling training data. Unlike discriminative models for which generalization is easily tested via held-out data, evaluating generalization in generative models is not straightforward (Adlam et al. (2019); Meehan et al. (2020)). We propose an authenticity score A ∈ [0, 1] to quantify the rate by which a model generates new samples. To pin down a mathematical definition for A, we reformulate Pg as a mixture of densities as follows:
Pg = A · P′g + (1−A) · δg, , (7)
where P′g is the generative distribution conditioned on the synthetic samples being non-overfitted, and δg, is a noisy distribution over training data. In particular, we define δg, as δg, = δg∗N (0, 2), where δg is a discrete distribution that places an unknown probability mass on each training data point in Dreal, is an arbitrarily small noise variance, and ∗ is the convolution operator. Essentially, (7) assumes that the model flips a (biased coin), pulling off a training sample with probability 1−A and adding some noise to it, or innovating a new sample with probability A. A model with A = 1 always innovates, whereas an overfitted model will concentrate Pg around the training data.

heading: 4 ESTIMATING THE EVALUATION METRIC
text: Since the metrics in Section 3 are defined through binary conditions on individual samples, we can obtain an estimate Ê = (P̂α, R̂β , Â) of the metric E, for a given α and β, by assigning binary scores P̂α,j , Âj ∈ {0, 1} to each synthetic sample X̃g,j in Dsynth, and R̂β,i ∈ {0, 1} to each real sample X̃r,i in Dreal, then averaging over all samples, i.e., P̂α = 1m ∑ j P̂α,j , R̂β = 1 n ∑ i R̂β,i, Â = 1 m ∑ j Âj . To assign binary scores, we construct 3 classifiers fP , fR, fA : X̃ → {0, 1}, where P̂α,j = fP (X̂g,j), R̂β,i = fR(X̂r,i) and Âj = fA(X̂g,j). We explain the operation of each classifier in what follows.
Precision and Recall classifiers. Based on definitions (4) and (5), both classifiers check if a sample resides in an α- (or β-) support, i.e., fP (X̃g) = 1{X̃g ∈ Ŝαr } and fR(X̃r) = 1{X̃r ∈ Ŝβg }. Hence, the main difficulty in implementing fP and fR is estimating the supports Ŝαr and Ŝβg —in fact, even if we know the exact distributions Pr and Pg , computing their α- and β-supports is not straightforward as it involves solving the optimization problem in (2).
To address this challenge, we pre-process the real and synthetic data in a way that renders estimation of α-and β-supports straightforward. The idea is to train the evaluation embedding Φ so as to cast the supports of the real data, Sr, into a hypersphere with radius r, and cast the distribution Pr into an isotropic density concentrated around the center cr of the hypersphere. We achieve this by modeling Φ as a one-class neural network trained with the following loss function: L = ∑ i `i, where
`i = r 2 +
1 ν max{0, ‖Φ(Xr,i)− cr ‖2 − r2}. (8)
The loss is minimized over the radius r and the parameters of Φ; the output dimensions of Φ, cr and ν are viewed as hyperparameters (see Supplementary material). The loss in (8) is based on the seminal
work on one-class SVMs in (Schölkopf et al. (2001)), which is commonly applied to outlier detection problems, e.g., (Ruff et al. (2018)). In a nutshell, the evaluation embedding squeezes real data into the minimum-volume hypersphere centered around cr, hence the real α-support is estimated as:
Ŝαr = B(cr, r̂α), r̂α = Q̂α{‖X̃r,i − cr‖ : 1 ≤ i ≤ n}, (9)
whereB(c, r) is a Euclidean ball with center c and radius r, and Q̂α is the α-quantile function. The set of all α-supports of Pr corresponds to the set of all concentric spheres with center cr and radii r̂α, ∀α ∈ [0, 1]. Thus, the precision classifier assigns a score 1 to a synthetic sample X̃g if it resides in the Ball Ŝαr , i.e., fp(X̃g) = 1{‖X̃g − cr‖ ≤ r̂α}. Now define cg = 1m ∑ j X̃g,j , and consider a hypersphereB(cg, r̂β), where r̂β = Q̂β{‖X̃g,j − cg‖ : 1 ≤ j ≤ m}. We construct fR as follows:
fR(X̃r,i) = 1{X̃βg,j∗ ∈ B(X̃r,i,NNDk(X̃r,i))}, (10)
where X̃βg,j∗ is the synthetic sample in B(cg, r̂β) that is closest to X̃r,i, and NNDk(X̃r,i) is the distance between X̃r,i and its k-nearest neighbor in Dreal. Similar to the estimator in (Naeem et al. (2020)), (10) is a nonparametric estimate of Sβg that checks if each real sample i is locally covered by a synthetic sample inB(cg, r̂β). A discussion on how to select the hyper-parameter k, as well as an alternative method for estimating Sβg using one-class representations is provided in the Appendix.
Authenticity classifier. We derive the classifier fA from a hypothesis test that tests if a sample X̃g,j is non-memorized. LetH1 : Aj = 1 be the hypothesis that X̃g,j is authentic, with the null hypothesis H0 : Aj = 0. To test the hypothesis, we use the likelihood-ratio statistic (Van Trees (2004)):
Λ(X̃g,j) = P(X̃g,j |Aj = 1)/P(X̃g,j |Aj = 0) = P′g(X̃g,j)/δg, (X̃g,j), (11)
which follows from (7). Since both likelihood functions in (11) are unknown, we need to test the hypothesisH1 : Aj = 1 using an alternative sufficient statistic with a known probability distribution.
Let dg,j = d(X̃g,j ,Dreal) be the distance between synthetic sample j and the training data set, and let i∗ be the training sample inDreal closest toXg,j , i.e., dg,j = d(X̃g,j , X̃r,i∗). Let dr,i∗ be the distance between X̃r,i∗ and Dreal/{X̃r,i∗}, i.e., the training data with sample i∗ removed. Now consider the statistic aj = 1{dg,j ≤ dr,i∗}, which indicates if synthetic sample j is closer to training data than any other training sample. The likelihood ratio for observations {aj}j under hypothesesH0 andH1 is
Λ(aj) = P(aj |Aj = 1)/P(aj |Aj = 0) ≈ a−1j · P(dg,j ≤ dr,i∗ |Aj = 1). (12)
Here, we used the fact that if sample j is a memorized copy of i∗, and if the noise variance in (7) is arbitrarily small, then aj = 1 almost surely and P(aj |Aj = 0) ≈ 1. If j is authentic, then X̃g,j lies in the convex hull of the training data, and hence P(aj |Aj = 0)→ 0 and Λ→∞ for a large real data set. Thus, fA issues a label Aj = 1 if aj = 0, and Aj = 0 otherwise. Intuitively, fA deems sample j unauthentic if it is closer to i∗ than any other real sample in the training data.

heading: 5 EXPERIMENTS AND USE CASES

heading: 5.1 EVALUATING & AUDITING GENERATIVE MODELS FOR SYNTHESIZING COVID-19 DATA
text: In this experiment, we use our metric to assess the ability of different generative models to synthesize COVID-19 patient data that can be used for predictive modeling. Using SIVEP-Gripe (SIVEP-Gripe (2020)), a database of 99,557 COVID patients in Brazil, including sensitive data such as ethnicity. We use generative models to synthesize replicas of this data and fit predictive models to the replicas.
Models and baselines. We create 4 synthetic data sets using GAN, VAE, Wasserstein GANs with a gradient penalty (WGAN-GP) (Gulrajani et al. (2017)), and ADS-GAN, which is specifically designed to prevent patient identifiablity in generated data (Yoon et al. (2020)). To evaluate these synthetic data sets, we use Fréchet Inception Distance (FID) (Heusel et al. (2017)), Precision/Recall (P1/R1) (Sajjadi et al. (2018)), Density/Coverage (D/C) (Naeem et al. (2020)), Parzen window likelihood (PW ) (Bengio et al. (2013)) and Wasserstein distance (W ) as baselines. On each synthetic data, we fit a predictive Logistic regression model to predict patient-level COVID-19 mortality.
Predictive modeling with synthetic data. In the context of predictive modeling, a generative model is assessed with respect to its usefulness in training predictive models that generalize well on real data. Hence, the “ground-truth” ranking of the 4 generative models corresponds to the ranking of the AUC-ROC scores achieved by predictive models fit to their respective synthetic data sets and tested on real data (Figure 4(a)). The data synthesized by ADS-GAN (×) displayed the best performance, followed by WGAN-GP (•), VAE (N), and GAN ( ). To assess the accuracy of baseline evaluation metrics, we test if each metric can recover the ground-truth ranking of the 4 generative models (Figure 4(a)). Our integrated precision and recall metrics IPα and IRβ both assign the highest scores to ADS-GAN; IPα exactly nails the right ranking of generative models. On the other hand, competing metrics such as P1, C and D, over-estimate the quality of VAE and WGAN-GP—if we use these metrics to decide which generative model to use, we will end up with predictive models that perform poorly, i.e. AUC-ROC of the predictive model fitted to synthetic data with best P1 is 0.55, compared to an AUC-ROC of 0.79 for our IPα score.
These results highlight the importance of accounting for the densities Pg and Pr, and not just their supports, when evaluating a generative model. This is because a shifted Pg would result in a “covariate shift” in synthetic data, leading to poor generalization for predictive models fitted to it, even when real and synthetic support coincide. As we can see in Figure 4(a), metrics that compare distributions (our metrics, PW and FID), are able to accurately rank the 4 generative models.
Hyper-parameter tuning & the privacy-utility tradeoff. Another use case for our metric is hyperparameter optimization for generative models. Here, we focus on the best-performing model in our previous experiment: ADS-GAN. This model has a hyper-parameter λ ∈ R that determines the importance of the privacy-preservation loss function used to regularize the training of ADS-GAN (Yoon et al. (2020)): smaller values of λmake the model more prone to overfitting, and hence privacy leakage. Figure 4(b) shows how our precision and authenticity metrics change with the different values of λ: the curve provides an interpretable tradeoff between privacy and utility (e.g., for λ = 2, an A score of 0.4 means that 60% of patients may have personal information exposed). Increasing λ improves privacy at the expense of precision. By visualizing this tradeoff using our metric, data holders can understand the risks of different modeling choices involved in data synthesis.
Improving synthetic data via model auditing. Our metrics are not only useful for hyper-parameter tuning, but can also be used to improve the quality of synthetic data generated by an already-trained model using (post-hoc) auditing. Because our metrics are defined on the sample level, we can discard unauthentic or imprecise samples. This does not only lead to nearly optimal precision and authenticity for the curated data (Figure 4(c)), but also improves the AUC-ROC of the predictive model fitted to audited data (from 0.76 to 0.78 for the audited ADS-GAN synthetic data, p < 0.005), since auditing eliminates noisy data points that would otherwise undermine generalization performance.

heading: 5.2 DIAGNOSING GENERATIVE DISTRIBUTIONS OF MNIST
text: In this experiment, we test the ability of our metrics to detect common modes of failure in generative modeling—in particular, we emulate a mode dropping scenario, where the generative model fails to
recognize the distinct modes in a multimodal distribution Pr, and instead recovers a single mode in Pg . To construct this scenario, we fit a conditional GAN (CGAN) model (Wang et al. (2018)) on the MNIST data set, and generate 1,000 samples for each of the digits 0-9. (We can think of each digit as a distinct mode in Pr.) To apply mode dropping, we first sample 1,000 instances of each digit from the CGAN, and then delete individual samples of digits 1 to 9 with a probability Pdrop, and replace the deleted samples with new samples of the digit 0 to complete a data set of 10,000 instances. The parameter Pdrop ∈ [0, 1] determines the severity of mode dropping: for Pdrop = 0, the data set has all digits being equally represented with 1,000 samples, and for Pdrop = 1, the data set has 10,000 samples of the digit 0 only as depicted pictorially in Figure 5(a) (bottom panel).
We show how the different evaluation metrics respond to varying Pdrop from 0 to 1 in Figure 5(a) (top). Because mode dropping pushes the generative distribution away from the real one, statistical distance metrics such as W and FID increase as Pdrop approaches 1. However, these metrics only reflect a discrepancy between Pr and Pg , and do not disentangle the Fidelity and Diversity components of this discrepancy. On the other hand, standard precision and recall metric are completely insensitive to mode dropping except for the extreme case when Pdrop = 1. This is because both metrics only check supports of Pr and Pg , so they cannot recognize mode dropping as long as there is a non-zero probability that the model will generates digits 1-9. On the contrary, mode dropping reflects in our metrics, which manifest in a declining IRβ as Pdrop increases. Since mode dropping affects coverage of digits and not the quality of images, it only affects IRβ but not IPα.

heading: 5.3 REVISITING THE HIDE-AND-SEEK CHALLENGE FOR SYNTHESIZING TIME-SERIES DATA
text: Finally, we use our metric to re-evaluate the generative models submitted to the NeurIPS 2020 Hide-and-Seek competition (Jordon et al. (2020)). In this competition, participants were required to synthesize intensive care time-series data based on real data from the AmsterdamUMCdb database. A total of 16 submissions were judged based on the accuracy of predictive models fit to the synthetic data (an approach similar to the one in Section 5.1). The submissions followed various modeling choices, including recurrent GANs, autoencoders, differential privacy GANs, etc. Details of all submissions are available online. Surprisingly, the winning submission was a very simplistic model that adds Gaussian noise to the real data to create new samples.
To evaluate our metrics on time-series data, we trained a Seq-2-Seq embedding that is augmented with our One-class representations to transform time-series into fixed feature vectors. (The architecture for this embedding is provided in the Supplementary material.) In Figure 5(b), we evaluate all submissions with respect to precision, recall and authenticity. As we can see, the winning submission comes out as one of the least authentic models, despite performing competitively in terms of precision and recall. This highlights the detrimental impact of using naı̈ve metrics for evaluating generative models—based on the competition results, clinical institutions seeking to create synthetic data sets may be led to believe that Submission 1 in Figure 5(b) is the right model to use. However, our metrics—which give a fuller picture of the true quality of all submissions—shows that such model creates unauthentic samples that are mere noisy copies of real data, which would pose risk to patient privacy. We hope that our metrics and our pre-trained Seq-2-Seq embeddings can help clinical institutions evaluate the quality of their synthetic time-series data in the future.

heading: APPENDIX A: LITERATURE REVIEW
text: In this Section, we provide a comprehensive survey of prior work, along with a detailed discussion on how our metric relates to existing ones. We classify existing metrics for evaluating generative models into two main classes:
1. Statistical divergence metrics
2. Precision and recall metrics
Divergence metrics are single-valued measures of the distance between the real and generative distributions, whereas precision-recall metrics classify real and generated samples as to whether they are covered by generative and real distributions, respectively. In what follows, we list examples of these two types of metrics, highlighting their limitations.
Statistical divergence metrics.
The most straightforward approach for evaluating a generative distribution is to compute the model log-likelihood—for density estimation tasks, this has been the de-facto standard for training and evaluating generative models. However, the likelihood function is a model-dependent criteria: this is problematic because the likelihood of many state-of-the-art models is inaccessible. For instance, GANs are implicit likelihood models and hence provide no explicit expression for its achieved loglikelihood Goodfellow et al. (2014). Other models, energy-based models has a normalization constant in the likelihood expression that is generally difficult to compute as they require solving intractable complex integrals Kingma & Welling (2013).
Statistical divergence measures are alternative (model-independent) metrics that are related to loglikelihood, and are commonly used for training and evaluating generative models. Examples include lower bounds on the log-likelihood Kingma & Welling (2013), contrastive divergence and noise contrastive estimation Hinton (2002); Gutmann & Hyvärinen (2010), probability flow Sohl-Dickstein et al. (2011), score matching Hyvärinen et al. (2009), maximum mean discrepancy (MMD) Gretton et al. (2012), and the Jensen-Shannon divergence (JSD).
In general, statistical divergence measures suffer from the following limitations. The first limitation is that likelihood-based measures can be inadequate in high-dimensional feature spaces. As has been shown in (Theis et al., 2015), one can construct scenarios with poor likelihood and great samples through a simple lookup table model, and vice versa, we can think of scenarios with great likelihood and poor samples. This is because, if the model samples white noise 99% of the time, and samples high-quality outputs 1% of the time, the log-likelihood will be hardly distinguishable from a model that samples high-quality outputs 100% of the time if the data dimension is large. Our metrics solve this problem by measuring the rate of error on a sample-level rather than evaluating the overall distribution of samples.
Moreover, statistical divergence measures collapse the different modes of failure of the generative distribution into a single number. This hinders our ability to diagnose the different modes of generative model failures such as mode dropping, mode collapse, poor coverage, etc.
Precision and recall metrics.
Precision and recall metrics for evaluating generative models were originally proposed in Sajjadi et al. (2018). Our metrics differ from these metrics in various ways. First, unlike standard metrics, α-Precision and β-Recall take into account not only the supports of Pr and Pg , but also the actual probability densities of both distributions. Standard precision (and recall) correspond to one point on the Pα (and Rβ) curve; they are equal to Pα and Rβ evaluated on the full support (i.e., P1 and R1). By defining our metrics with respect to the α- and β-supports, we do not treat all samples equally, but rather assign higher importance to samples that land in “denser” regions of Sr and Sg . Hence, Pα and Rβ reflect the extent to which Pr and Pg are calibrated—i.e., good Pα and Rβ curves are achieved when Pr and Pg share the same modes and not just a common support. While optimal R1 and P1 can be achieved by arbitrarily mismatched Pr and Pg , our Pα and Rβ curves are optimized only when Pr and Pg are identical as stated by Theorem 1.
The new Pα and Rβ metrics address the major shortcomings of precision and recall. Among these shortcomings are: lack of robustness to outliers, failure to detect matching distributions, and inability to diagnose different types of distributional failure (such as mode collapse, mode invention, or density shifts) Naeem et al. (2020). Basically, a model Pg will score perfectly on precision and recall (R1=P1=1) as long as it nails the support of Pr, even if Pr and Pg place totally different densities on their common support.
In addition to the above, our metrics estimate the supports of real and generative distributions using neural networks rather than nearest neighbor estimates as in Naeem et al. (2020). This prevents our estimates from overestimating the supports of real and generative distributions, thereby overestimating the coverage or quality of the generated samples.

heading: APPENDIX B: PROOF OF THEOREM 1
text: To prove the statement of the Theorem, we need to prove the two following statements:
(1) Pg = Pr → Pα/α = Rβ/β = 1, ∀α, β (2) Pα/α = Rβ/β = 1, ∀α, β → Pg = Pr To prove (1), we start by noting that since we have Pg = Pr, then Sgα = Srα, ∀α ∈ [0, 1]. Thus, we have
Pα = P(X̃g ∈ Sαr ) = P(X̃g ∈ Sαg ) = α, (13)
for all α ∈ [0, 1], and similarly, we have
Rβ = P(X̃r ∈ Sβg ) = P(X̃r ∈ Sβr ) = β, (14)
for all β ∈ [0, 1], which concludes condition (1).
Now we consider condition (2). We first note that Sαr ⊆ Sα ′ r for all α ′ > α. If Pα = α for all α, then we have
P(X̃g ∈ Sαr ) = ∫ Sαr dPg = α, ∀α ∈ [0, 1]. (15)
Now assume that α′ = α+ ∆α, then we have∫ Sα′r /Sαr dPg = ∫ Sα′r /Sαr dPr = ∆α. (16)
Thus, the probability masses of Pg and Pr are equal for all infinitesimally small region Sα+∆αr /Sαr (for ∆α→ 0) of the α-support of Pr, hence Pg = Pr for all subsets of S1r . By applying the similar argument to the recall metric, we also have Pg = Pr for all subsets of S1g , and hence Pg = Pr.

heading: APPENDIX C: ALTERNATIVE APPROACH FOR ESTIMATING THE SUPPORT OF SYNTHETIC DATA & CODE SNIPPETS
text: Instead of using a k-NN approach to estimate the generative support Sβg , one could use a separate one-class representation Φg for each new synthetic sample being evaluated. We provide code snippets and comparisons between the two approaches in the an anonymized Colab notebook. While the two approaches perform rather similarly, we opt to adopt the k-NN based approach to avoid potential biases induced by using a separate representation for each generative model when using our metric for model comparisons.

heading: APPENDIX D: EXPERIMENTAL DETAILS
text: .1 DATA
In this research the argue for the versatility of our metrics, hence we have included results for tabular (static), time-series and image data (see Table 1). For the tabular data we use Baqui et al. (2020)’s
preprocessed version of the SIVEP-GRIPE dataset of Brazilian ICU Covid-19 patient data. For the image experiments, we use the 10,000 samples in the default MNIST test set LeCun (1998). For proper evaluation of the authenticity metric, the same original data is used for training of generative models and evaluation of all metrics.
For the time-series experiments, AmsterdamUMCdb is used in a manner exactly analogous to the NeurIPS 2020 Hide-and-Seek Privacy Challenge Jordon et al. (2020), which describes it as follows: “AmsterdamUMCdb was developed and released by Amsterdam UMC in the Netherlands and the European Society of Intensive Care Medicine (ESICM). It is the first freely accessible comprehensive and high resolution European intensive care database. It is also first to have addressed compliance with General Data Protection Regulation [...] AmsterdamUMCdb contains approximately 1 billion clinical data points related to 23,106 admissions of 20,109 unique patients between 2003 and 2016. The released data points include patient monitor and life support device data, laboratory measurements, clinical observations and scores, medical procedures and tasks, medication, fluid balance, diagnosis groups and clinical patient outcomes.”. Notably, only the longitudinal features from this database are kept, with static ones discarded. The same subset as was used in the competition for “hider” synthetic data generation is used; this consists of 7695 examples with 70 features (and a time column), sequence length is limited to 100 (the original data contains sequences of up to length 135,337). The features are normalised to [0, 1] and imputed as follows: (1) back-fill, (2) forward-fill, (3) feature median imputation. This preprocessing is chosen to match the competition Jordon et al. (2020). The competition “hider” submissions were trained on this dataset and the synthetic data generated.
For metric consistency and the avoidance of tedious architecture optimization for each data modality, we follow previous works (e.g. Heusel et al. (2017); Sajjadi et al. (2018); Kynkäänniemi et al. (2019); Naeem et al. (2020)) and embed image and time series data into a static embedding. This is required, since the original space is non-euclidean and will result in failure of most metrics. The static embedding is used for computing baseline metrics, and is used as input for the One-Class embedder.
For finding static representations of MNIST, images are upscaled and embedded using InceptionV3 pre-trained on ImageNET without top layer. This is the same embedder used for computing Frchet Inception Distance Heusel et al. (2017). Very similar results were obtained using instead a VGG16 embedder Brock et al. (2018); Kynkäänniemi et al. (2019). Preliminary experimentation with random VGG-16 models Naeem et al. (2020) did not yield stable results for neither baselines nor our methods.
.2 TIME SERIES EMBEDDING
The time series embeddings used throughout this work are based on Unsupervised Learning of Video Representations using LSTMs Srivastava et al. (2015), specifically the “LSTM Autoencoder Mode”. A sequence-to-sequence LSTM network is trained, with the target sequence set as the input sequence (reversed for ease of optimization), see Figure 1. The encoder hidden and cell states (h and c vectors) at the end of a sequence are used as the learned representation and are passed to the decoder during training. At inference, these are concatenated to obtain one fixed-length vector per example.
The specifics of the LSTM autoencoder used here are as follows. Two LSTM layers are used in each encoder and decoder. The size of h, c vectors is 70 (280 after concatenation). The model was implemented in PyTorch Paszke et al. (2017), utilising sequence packing for computational efficiency. All autoencoders were trained to convergence on the original data; the synthetic time series data was passed through this at inference. The time column (when present in data) was discarded.
.3 FULL RESULTS
Table 2 contains metrics computed on different generated versions of the SIVEP-GRIPE tabular dataset. Included metrics are Wasserstein distance, Frchet Distance (FD), Parzen window likelihood estimate, precision P1, recall R1, density (D), coverage C and the proposed metrics, specifically integrated α-precision IPα, integrated β-recall IRβ and authenticity A . For the tabular data, data is generated using a VAE, GAN, Wasserstein GAN with gradient penalisation (WGAN-GP) Arjovsky et al. (2017), ADS-GAN Yoon et al. (2020), Differentially Private GAN (DP-GAN) Xie et al. (2018) and an ADS-GAN generated dataset in which samples are audited on precision and authenticity. Similarly, Table 3 contains metric results1 for MNIST, generated by a VAE, Deep convolution GAN (DCGAN), WGAN-GP and ADS-GAN. Table 4 contains results for MIMIC generation using different methods from the Hide-and-Seek Privacy Competition Jordon et al. (2020). The submission that won the competition is the penultimate model, Hamada. The last row shows results for an audited version of the Hamada dataset, in which we keep generating data using the Hamada model and discard samples that do not meet the precision or authenticity threshold.
1Note that here, the Frchet Distance equals the Frchet Inception Distance, because the metrics are computed on InceptionV3 embeddings of the image data.
.4 HYPERPARAMETER OPTIMIZATION
.4.1 BASELINES
For computing the density and coverage metrics, we set a threshold of 0.95 on the minimum expected coverage, as recommended in the original work (Eq. 9 Naeem et al. (2020)). For all datasets, this is achieved for k = 5. For consistency in these comparisons, we use k = 5 for the precision and recall metrics too.
.4.2 ONECLASS EMBEDDINGS
We use Deep SVDD Ruff et al. (2018) to embed static data into One-Class representations. To mitigate hypersphere collapse (Propostions 2 and 3 of Ruff et al. (2018)), we do not include a bias term and use ReLU activation for the One-Class embedder. Original data is split into training (80%) and validation (20%) set, and One-Class design is fine-tuned to minimise validation loss. We use the SoftBoundary objective (Eq. 3 Ruff et al. (2018)) with ν = 0.01 and center c = 1 for tabular and time-series data and c = 10 ·1 for image data. Let nh be the number of hidden layers with each dh nodes, and let dz be the dimension of the representation layer. For tabular data, we use nh = 3, dh = 32 and dz = 25; for time-series data, nh = 2, dh = 128 and dz = 32; and for MNIST nh = 3, dh = 128 and dz = 32. Models are implemented in PyTorch Paszke et al. (2017) and the AdamW optimizer is used with weight decay 10−2.
For the β-recall metric, estimating the support of synthetic data involves tuning the k parameter of the k-NN estimator. The k parameter can be tuned by fitting the NN estimator on a portion of the data for every given k, and then testing the recall on a held out (real) sample. The selected k for each α is the smallest k that covers α held out samples. Similar to Naeem et al. (2020), we found that through this procedure, k = 5 seems to come up as the optimal k for most experiments.
.5 TOY EXPERIMENTS
We include two toy experiments that highlight the advantage of the proposed metrics compared to previous works. We focus our comparison on the improved precision and recall Kynkäänniemi et al. (2019) and density and coverage Naeem et al. (2020) metrics.
.5.1 ROBUSTNESS TO OUTLIERS
Naeem et al. (2020) showed that the precision and recall metrics as proposed by Sajjadi et al. (2018); Kynkäänniemi et al. (2019) are not robust to outliers. We replicate toy experiments to show the proposed α-Precision and β-Recall do not suffer the same fate.
Let X,Y ∈ Rd denote original and synthetic samples respectively, with original X ∼ N(0, I) and Y ∼ N(µ, I). We compute all metrics for µ ∈ [−1, 1]. In this setting we conduct three experiments:
1. No outliers
2. One outlier in the real data at X = 1
3. One outlier in the synthetic data at Y = 1
We set d = 64 and both original and synthetic data we sample 10000 points. Subsequent metric scores are shown in Figure
As can be seen, the precision and recall metrics are not robust to outliers, as just a single outlier has dramatic effects. The IPα and IRβ are not affected, as the outlier does not belong to the α-support (or β-support) unless α (or β) is large.
.5.2 MODE RESOLUTION
The precision and recall metrics only take into account the support of original and synthetic data, but not the actual densities. The density and coverage metric do take this into account, but here we show these are not able to capture this well enough to distinguish similar distributions.
In this experiment we look at mode resolution: how well is the metric able to distinguish a single mode from two modes? Let the original distribution be a mixture of two gaussians that are separated by distance µ and have σ = 1,
X ∼ 1 2 N(−µ 2 , 1) + 1 2 N(+ µ 2 , 1)
and let the synthetic data be given by
Y ∼ N(0, 1 + µ2).
This situation would arise if a synthetic data generator fails to distinguish the two nodes, and instead tries to capture the two close-by modes of the original distribution using a single mode. We compute metrics for µ ∈ [0, 5].
(a) Precision (b) Recall
(c) Density (d) Coverage
(e) IPα (f) IRβ
Figure 3: Toy experiment II: mode resolution As can be seen, neither P&R nor D&C notice that the synthetic data only consists of a single mode, whereas the original data consisted of two. The α-precision metric is able to capture this metric: for small α the α-support of the original distribution is centred around the two separated, and does not contain the space that separates the modes (i.e. the mode of the synthetic data).

Review summaries:
This paper targets to a 3-dim metric: alpha-Precision, beta-Recall, and Authenticity, that quantifies the fidelity, diversity, and generalization performance of a generative model. The proposed metric serves as soft-boundary classifiers between real and generated spherical-shaped supports, and improves the robustness against outliers and generation failure cases. This sample-wise metric can audit models by judging individual synthetic samples by their quality. 

This work addressed an interesting problem and a very relevant one - how to audit generative models? The most popular metric to evaluate generative models is FID but it is extremely opaque and population based. This paper proposes new fidelity metrics that overcome both these challenges. While the definition of these metrics are well grounded, their measurement process itself - which is done via trained NN classifiers - is a bit unconvincing. The authors demonstrate that the proposed metric can recover the real ranking of generative models on a synthetic data generation task and can discover issues like mode collapse.

The paper presents a methodology assessing the performance of a generative model in a domain agnostic fashion.  A 3-dimensional metric space is proposed: fidelity (output quality), diversity (coverage of expected variability of output) and generalization (to what degree model avoids memorizing training data i.e. is truly generative) is proposed.  Particularly the latter element is novel and the manner in which it is may be evaluated by developing an `authenticity’ measure for the task. Three illustrative use cases in image (mnist) and medical patient data (COVID-19, `Hide and Seek’ seq2seq data) domains are provided.