How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem . Most existing metrics , which were tailored solely to the image synthesis setup , exhibit a limited capacity for diagnosing the modes of failure of generative models across broader application domains . In this paper , we introduce a 3-dimensional metric , ( α-Precision , β-Recall , Authenticity ) , that characterizes the fidelity , diversity and generalization performance of any generative model in a domain-agnostic fashion . Our metric unifies statistical divergence measures with precision-recall analysis , enabling sample-level and distribution-level diagnoses of model fidelity and diversity . We introduce generalization as an additional dimension for model performance that quantifies the extent to which a model copies training data—a crucial performance indicator when modeling sensitive data with requirements on privacy . The three metric components correspond to ( interpretable ) probabilistic quantities , and are estimated via sample-level binary classification . The sample-level nature of our metric inspires a novel use case which we call model auditing , wherein we judge the quality of individual samples generated by a ( black-box ) model , discarding low-quality samples and hence improving the overall model performance in a post-hoc manner . 1 INTRODUCTION . Intuitively , it would seem that evaluating the likelihood function of a generative model is all it takes to assess its performance . As it turns out , the problem of evaluating generative models is far more complicated . This is not only because state-of-the-art models , such as Variational Autoencoders ( VAE ) ( Kingma & Welling ( 2013 ) ) and Generative Adversarial Networks ( GANs ) ( Goodfellow et al . ( 2014 ) ) , do not possess tractable likelihood functions , but also because the likelihood score itself is a flawed measure of performance—it scales badly in high dimensions , and it obscures distinct modes of model failure into a single uninterpretable score ( Theis et al . ( 2015 ) ) . Absent domain-agnostic metrics , earlier work focused on crafting domain-specific scores , e.g. , Inception score ( Salimans et al . ( 2016 ) ) , with an exclusive emphasis on image data ( Lucic et al . ( 2018 ) ) . In this paper , we introduce an alternative approach to evaluating generative models , where instead of assessing the generative distribution by looking at all synthetic samples collectively to compute likelihood or divergence , we classify each sample individually as being of high or low quality . In this way , our metric comprises interpretable probabilistic quantities—resembling those used to evaluate discriminative models ( e.g. , AUC-ROC ) —which describe the rate by which a model makes errors . When averaged over all samples , our sample-level scores reflect discrepancy between real and generative distributions in a way similar to statistical divergence measures ( e.g. , KL divergence , Fréchet distance ( Heusel et al . ( 2017 ) ) , or maximum mean discrepancy ( Sutherland et al . ( 2016 ) ) . In this sense , our metric enables diagnosing model performance on both the sample and distribution levels . Our metric represents the performance of a generative model as a point in a 3-dimensional space— each dimension corresponds to a distinct quality of the model . These qualities are : Fidelity , Diversity and Generalization . Fidelity measures the quality of a model ’ s synthetic samples , and Diversity is Figure 1 : Pictorial depiction for the proposed evaluation metrics . The blue and red spheres correspond to the α- and β-supports of real and generative distributions , respectively . Blue and red points correspond to real and synthetic data . ( a ) Synthetic data falling outside the blue sphere will look unrealistic or noisy . ( b ) Overfitted models can generate ostensibly high-quality data samples that are “ unauthentic ” because they are copied from the training data . ( c ) High-quality data samples should reside inside the blue sphere . ( d ) Outliers do not count in the β-Recall metric . ( Here , α=β=0.9 , α-Precision=8/9 , βRecall = 4/9 , and Authenticity = 9/10 . ) Support of real distribution Support of synthetic distribution Real Synthetic Noisy synthetic sample Unauthentic high-quality sample Authentic high-quality sample Outlier ( a ) ( b ) ( c ) ( d ) the extent to which these samples cover the full variability of real samples , whereas Generalization quantifies the extent to which a model overfits ( copies ) training data . How do we quantify the 3 dimensions of performance ? We build on the precision-recall analysis framework proposed in ( Sajjadi et al . ( 2018 ) ) , and introduce the α-Precision and β-Recall metrics to quantify model Fidelity and Diversity , respectively . Both metrics assume that a fraction 1 − α ( or 1 − β ) of the real ( and synthetic ) data are “ outliers ” , and α ( or β ) are “ typical ” . α-Precision is the fraction of synthetic samples that resemble the “ most typical ” α real samples , whereas β-Recall is the fraction of real samples covered by the most typical β synthetic samples . α-Precision and β-Recall are evaluated for all α , β ∈ [ 0 , 1 ] , providing entire precision and recall curves instead of single numbers . To compute both metrics , we embed the ( real and synthetic ) data into hyperspheres with most samples concentrated around the centers , i.e. , the real and generative distributions ( Pr and Pg ) has spherical-shaped supports . In this transformed feature space , typical samples would be located near the centers of the spheres , whereas outliers would be closer to the boundaries . To quantify Generalization , we introduce the Authenticity metric , which reflects the likelihood of a synthetic sample being copied from training data . We derive the Authenticity metric from a hypothesis test for data copying based on the observed proximity of synthetic samples to real ones in the embedded feature space . A pictorial illustration for all metrics is shown in Figure 1 . How is our metric different ? If one think of standard precision and recall metrics as “ hard ” binary classifiers of real and synthetic samples , our α-Precision and β-Recall can be thought of as softboundary classifiers that do not only compare the supports of Pr and Pg , but also assesses whether both distributions are calibrated . Precision and recall metrics are special cases of α-Precision and β-Recall for α = β = 1 . As we show later , our new metric definitions solve many of the drawbacks of standard precision-recall analysis , such as lack of robustness to outliers and failure to detect distributional mismatches ( Naeem et al . ( 2020 ) ) . They also enable detailed diagnostics of different types of model failure , such as mode collapse and mode invention . Moreover , optimal values of our metrics are achieved only when Pr and Pg are identical , thereby eliminating the need to augment the evaluation procedure with additional measures of statistical divergence ( e.g. , KL divergence ) . Previous works relied on pre-trained embeddings ( using ImageNet feature extractors ( Deng et al . ( 2009 ) ) ) . In this work , we propose feature embeddings that are model- and domain-agnostic , and are tailored to our metric definitions and data set at hand . Our proposed feature embedding step can be completely bespoke to raw data , or augmented with pre-trained embeddings . This enables our metric to remain operable in application domains where no pre-trained representations exist . Overfitting is a crucial mode of failure of generative models , especially when modeling sensitive data ( e.g. , clinical data ) for which data copying may violate privacy requirements ( Yoon et al . ( 2020 ) ) , but it has been overlooked in previous works which focused exclusively on quantifying the FidelityDiversity characterization ( Brock et al . ( 2018 ) ) . As we show in our experiments ( Section 5 ) , because our metric accounts for Generalization , it can provide a fuller picture of a generative model ’ s performance . Precisely , we show that some of the celebrated generative models score highly for Fidelity and Diversity simply because they memorize real samples , rendering them inappropriate for privacysensitive applications . A comprehensive survey of prior work is provided in the Appendix . Model auditing as a novel use case . The sample-level nature of our metrics inspires the new use case of model auditing , wherein we judge individual synthetic samples by their quality , and reject samples that have low Fidelity or are unauthentic . In Section 5 , we show that model audits can improve the outputs of a black-box model in a post-hoc fashion without any modifications to the model itself , and demonstrate this use case in synthesizing clinical data for COVID-19 patients . 2 EVALUATING AND AUDITING GENERATIVE MODELS . 2.1 PROBLEM SETUP . We denote real and generated data asXr ∼ Pr andXg ∼ Pg , respectively , whereXr , Xg ∈ X , with Pr and Pg being the real and generative distributions , and X being the input space . The generative distribution , Pg , is estimated using a generative model ( e.g. , GAN ) . The real and synthetic data sets are Dreal = { Xr , i } ni=1 and Dsynth = { Xg , j } mj=1 , where Xr , i ∼ Pr and Xg , j ∼ Pg . 2.2 WHAT MAKES A GOOD SYNTHETIC DATA SET ? . Our goal is to construct a metric E ( Dreal , Dsynth ) for the quality of Dsynth in order to ( i ) evaluate the performance of the underlying generative model Pg , and ( ii ) audit the model outputs by discarding ( individual ) “ low-quality ” samples , thereby improving the overall quality ofDsynth . In order for E to fulfill the evaluation and auditing tasks , it must satisfy the following desiderata : ( 1 ) it should be able to disentangle the different modes of failure of Pg through interpretable measures of performance , and ( 2 ) it should be sample-wise computable , i.e. , we should be able to tell if a given ( individual ) synthetic sample Xg ∼ Pg is of a low quality . Having outlined the desiderata for our sought-after evaluation metric , we now propose three qualities of synthetic data that the metric E should be able to quantify . Failure to fulfill any of these three qualities correspond to independent modes of failure of the model Pg . These qualities are : 1 . Fidelity—the generated samples resemble real samples from Pr . A high-fidelity synthetic data set should contain “ realistic ” samples , e.g . visually-realistic images . 2 . Diversity—the generated samples are diverse enough to cover the variability of real data , i.e. , a model should be able to generate a wide variety of good samples . 3 . Generalization—the generated samples should not be mere copies of the ( real ) samples in training data , i.e. , models that overfit to Dreal are not truly “ generative ” . In Section 3 , we propose a three-dimensional evaluation metric E that captures all of the qualities above . Our proposed metric can be succinctly described as follows : E , ( α-Precision︸ ︷︷ ︸ Fidelity , β-Recall︸ ︷︷ ︸ Diversity , Authenticity︸ ︷︷ ︸ Generalization ) . ( 1 ) The α-Precision and β-Recall metrics are generalizations of the conventional notions of precision and recall used in binary classification analysis ( Flach & Kull ( 2015 ) ) . Precision measures the rate by which the model synthesizes “ realistic-looking ” samples , whereas the recall measures the fraction of real samples that are covered by Pg . The authenticity score measures the fraction of synthetic samples that are invented by the model and not copied from the training data .

Summary: This work addressed an interesting problem and a very relevant one - how to audit generative models? The most popular metric to evaluate generative models is FID but it is extremely opaque and population based. This paper proposes new fidelity metrics that overcome both these challenges. While the definition of these metrics are well grounded, their measurement process itself - which is done via trained NN classifiers - is a bit unconvincing. The authors demonstrate that the proposed metric can recover the real ranking of generative models on a synthetic data generation task and can discover issues like mode collapse.