How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem . Most existing metrics , which were tailored solely to the image synthesis setup , exhibit a limited capacity for diagnosing the modes of failure of generative models across broader application domains . In this paper , we introduce a 3-dimensional metric , ( α-Precision , β-Recall , Authenticity ) , that characterizes the fidelity , diversity and generalization performance of any generative model in a domain-agnostic fashion . Our metric unifies statistical divergence measures with precision-recall analysis , enabling sample-level and distribution-level diagnoses of model fidelity and diversity . We introduce generalization as an additional dimension for model performance that quantifies the extent to which a model copies training data—a crucial performance indicator when modeling sensitive data with requirements on privacy . The three metric components correspond to ( interpretable ) probabilistic quantities , and are estimated via sample-level binary classification . The sample-level nature of our metric inspires a novel use case which we call model auditing , wherein we judge the quality of individual samples generated by a ( black-box ) model , discarding low-quality samples and hence improving the overall model performance in a post-hoc manner . 1 INTRODUCTION . Intuitively , it would seem that evaluating the likelihood function of a generative model is all it takes to assess its performance . As it turns out , the problem of evaluating generative models is far more complicated . This is not only because state-of-the-art models , such as Variational Autoencoders ( VAE ) ( Kingma & Welling ( 2013 ) ) and Generative Adversarial Networks ( GANs ) ( Goodfellow et al . ( 2014 ) ) , do not possess tractable likelihood functions , but also because the likelihood score itself is a flawed measure of performance—it scales badly in high dimensions , and it obscures distinct modes of model failure into a single uninterpretable score ( Theis et al . ( 2015 ) ) . Absent domain-agnostic metrics , earlier work focused on crafting domain-specific scores , e.g. , Inception score ( Salimans et al . ( 2016 ) ) , with an exclusive emphasis on image data ( Lucic et al . ( 2018 ) ) . In this paper , we introduce an alternative approach to evaluating generative models , where instead of assessing the generative distribution by looking at all synthetic samples collectively to compute likelihood or divergence , we classify each sample individually as being of high or low quality . In this way , our metric comprises interpretable probabilistic quantities—resembling those used to evaluate discriminative models ( e.g. , AUC-ROC ) —which describe the rate by which a model makes errors . When averaged over all samples , our sample-level scores reflect discrepancy between real and generative distributions in a way similar to statistical divergence measures ( e.g. , KL divergence , Fréchet distance ( Heusel et al . ( 2017 ) ) , or maximum mean discrepancy ( Sutherland et al . ( 2016 ) ) . In this sense , our metric enables diagnosing model performance on both the sample and distribution levels . Our metric represents the performance of a generative model as a point in a 3-dimensional space— each dimension corresponds to a distinct quality of the model . These qualities are : Fidelity , Diversity and Generalization . Fidelity measures the quality of a model ’ s synthetic samples , and Diversity is Figure 1 : Pictorial depiction for the proposed evaluation metrics . The blue and red spheres correspond to the α- and β-supports of real and generative distributions , respectively . Blue and red points correspond to real and synthetic data . ( a ) Synthetic data falling outside the blue sphere will look unrealistic or noisy . ( b ) Overfitted models can generate ostensibly high-quality data samples that are “ unauthentic ” because they are copied from the training data . ( c ) High-quality data samples should reside inside the blue sphere . ( d ) Outliers do not count in the β-Recall metric . ( Here , α=β=0.9 , α-Precision=8/9 , βRecall = 4/9 , and Authenticity = 9/10 . ) Support of real distribution Support of synthetic distribution Real Synthetic Noisy synthetic sample Unauthentic high-quality sample Authentic high-quality sample Outlier ( a ) ( b ) ( c ) ( d ) the extent to which these samples cover the full variability of real samples , whereas Generalization quantifies the extent to which a model overfits ( copies ) training data . How do we quantify the 3 dimensions of performance ? We build on the precision-recall analysis framework proposed in ( Sajjadi et al . ( 2018 ) ) , and introduce the α-Precision and β-Recall metrics to quantify model Fidelity and Diversity , respectively . Both metrics assume that a fraction 1 − α ( or 1 − β ) of the real ( and synthetic ) data are “ outliers ” , and α ( or β ) are “ typical ” . α-Precision is the fraction of synthetic samples that resemble the “ most typical ” α real samples , whereas β-Recall is the fraction of real samples covered by the most typical β synthetic samples . α-Precision and β-Recall are evaluated for all α , β ∈ [ 0 , 1 ] , providing entire precision and recall curves instead of single numbers . To compute both metrics , we embed the ( real and synthetic ) data into hyperspheres with most samples concentrated around the centers , i.e. , the real and generative distributions ( Pr and Pg ) has spherical-shaped supports . In this transformed feature space , typical samples would be located near the centers of the spheres , whereas outliers would be closer to the boundaries . To quantify Generalization , we introduce the Authenticity metric , which reflects the likelihood of a synthetic sample being copied from training data . We derive the Authenticity metric from a hypothesis test for data copying based on the observed proximity of synthetic samples to real ones in the embedded feature space . A pictorial illustration for all metrics is shown in Figure 1 . How is our metric different ? If one think of standard precision and recall metrics as “ hard ” binary classifiers of real and synthetic samples , our α-Precision and β-Recall can be thought of as softboundary classifiers that do not only compare the supports of Pr and Pg , but also assesses whether both distributions are calibrated . Precision and recall metrics are special cases of α-Precision and β-Recall for α = β = 1 . As we show later , our new metric definitions solve many of the drawbacks of standard precision-recall analysis , such as lack of robustness to outliers and failure to detect distributional mismatches ( Naeem et al . ( 2020 ) ) . They also enable detailed diagnostics of different types of model failure , such as mode collapse and mode invention . Moreover , optimal values of our metrics are achieved only when Pr and Pg are identical , thereby eliminating the need to augment the evaluation procedure with additional measures of statistical divergence ( e.g. , KL divergence ) . Previous works relied on pre-trained embeddings ( using ImageNet feature extractors ( Deng et al . ( 2009 ) ) ) . In this work , we propose feature embeddings that are model- and domain-agnostic , and are tailored to our metric definitions and data set at hand . Our proposed feature embedding step can be completely bespoke to raw data , or augmented with pre-trained embeddings . This enables our metric to remain operable in application domains where no pre-trained representations exist . Overfitting is a crucial mode of failure of generative models , especially when modeling sensitive data ( e.g. , clinical data ) for which data copying may violate privacy requirements ( Yoon et al . ( 2020 ) ) , but it has been overlooked in previous works which focused exclusively on quantifying the FidelityDiversity characterization ( Brock et al . ( 2018 ) ) . As we show in our experiments ( Section 5 ) , because our metric accounts for Generalization , it can provide a fuller picture of a generative model ’ s performance . Precisely , we show that some of the celebrated generative models score highly for Fidelity and Diversity simply because they memorize real samples , rendering them inappropriate for privacysensitive applications . A comprehensive survey of prior work is provided in the Appendix . Model auditing as a novel use case . The sample-level nature of our metrics inspires the new use case of model auditing , wherein we judge individual synthetic samples by their quality , and reject samples that have low Fidelity or are unauthentic . In Section 5 , we show that model audits can improve the outputs of a black-box model in a post-hoc fashion without any modifications to the model itself , and demonstrate this use case in synthesizing clinical data for COVID-19 patients . 2 EVALUATING AND AUDITING GENERATIVE MODELS . 2.1 PROBLEM SETUP . We denote real and generated data asXr ∼ Pr andXg ∼ Pg , respectively , whereXr , Xg ∈ X , with Pr and Pg being the real and generative distributions , and X being the input space . The generative distribution , Pg , is estimated using a generative model ( e.g. , GAN ) . The real and synthetic data sets are Dreal = { Xr , i } ni=1 and Dsynth = { Xg , j } mj=1 , where Xr , i ∼ Pr and Xg , j ∼ Pg . 2.2 WHAT MAKES A GOOD SYNTHETIC DATA SET ? . Our goal is to construct a metric E ( Dreal , Dsynth ) for the quality of Dsynth in order to ( i ) evaluate the performance of the underlying generative model Pg , and ( ii ) audit the model outputs by discarding ( individual ) “ low-quality ” samples , thereby improving the overall quality ofDsynth . In order for E to fulfill the evaluation and auditing tasks , it must satisfy the following desiderata : ( 1 ) it should be able to disentangle the different modes of failure of Pg through interpretable measures of performance , and ( 2 ) it should be sample-wise computable , i.e. , we should be able to tell if a given ( individual ) synthetic sample Xg ∼ Pg is of a low quality . Having outlined the desiderata for our sought-after evaluation metric , we now propose three qualities of synthetic data that the metric E should be able to quantify . Failure to fulfill any of these three qualities correspond to independent modes of failure of the model Pg . These qualities are : 1 . Fidelity—the generated samples resemble real samples from Pr . A high-fidelity synthetic data set should contain “ realistic ” samples , e.g . visually-realistic images . 2 . Diversity—the generated samples are diverse enough to cover the variability of real data , i.e. , a model should be able to generate a wide variety of good samples . 3 . Generalization—the generated samples should not be mere copies of the ( real ) samples in training data , i.e. , models that overfit to Dreal are not truly “ generative ” . In Section 3 , we propose a three-dimensional evaluation metric E that captures all of the qualities above . Our proposed metric can be succinctly described as follows : E , ( α-Precision︸ ︷︷ ︸ Fidelity , β-Recall︸ ︷︷ ︸ Diversity , Authenticity︸ ︷︷ ︸ Generalization ) . ( 1 ) The α-Precision and β-Recall metrics are generalizations of the conventional notions of precision and recall used in binary classification analysis ( Flach & Kull ( 2015 ) ) . Precision measures the rate by which the model synthesizes “ realistic-looking ” samples , whereas the recall measures the fraction of real samples that are covered by Pg . The authenticity score measures the fraction of synthetic samples that are invented by the model and not copied from the training data .

Summary: This paper targets to a 3-dim metric: alpha-Precision, beta-Recall, and Authenticity, that quantifies the fidelity, diversity, and generalization performance of a generative model. The proposed metric serves as soft-boundary classifiers between real and generated spherical-shaped supports, and improves the robustness against outliers and generation failure cases. This sample-wise metric can audit models by judging individual synthetic samples by their quality.