Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals

1 Introduction . Deep generative models have recently taken a giant leap forward in their ability to model complex , high-dimensional distributions . Recent advances are able to produce incredibly detailed and realistic images [ 34 , 51 , 32 ] , strikingly consistent and coherent text [ 50 , 66 , 6 ] , and music of near-human quality [ 16 ] . The advances in these models , particularly in the image domain , have been spurred by the development of quantitative evaluation tools which enable a large-scale comparison of models , as well as diagnosing of where and why a generative model fails [ 55 , 38 , 27 , 54 , 31 ] . Divergence frontiers were recently proposed by Djolonga et al . [ 18 ] to quantify the trade-off between quality and diversity in generative modeling with modern deep neural networks [ 54 , 37 , 59 , 44 , 49 ] . In particular , a good generative model must not only produce high-quality samples that are likely under the target distribution but also cover the target distribution with diverse samples . While this framework is mathematically elegant and empirically successful [ 37 , 49 ] , the statistical properties of divergence frontiers are not well understood . Estimating divergence frontiers from data for large generative models involves two approximations : ( a ) joint quantization of the model distribution and the target distribution into discrete distributions with quantization level k , and ( b ) statistical estimation of the divergence frontiers based on the empirical estimators of the quantized distributions . Djolonga et al . [ 18 ] argue that the quantization often introduces a positive bias , making the distributions appear closer than they really are ; while a small sample size can result in a pessimistic estimate of the divergence frontiers . The latter effect is due to the missing mass of the samples , causing the two distributions to appear farther than they really are because the samples do not cover some parts of the distributions . The first consideration favors a large k , while the second favors a small k. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we are interested in answering the following questions : ( a ) Given two distributions , how many samples are needed to achieve a desired estimation accuracy , or in other words , what is the sample complexity of the estimation procedure ; ( b ) Given a sample size budget , how to choose the quantization level to balance the errors induced by the two approximations ; ( c ) Can we have estimators better than the naïve empirical estimator . Outline . We review the definitions of divergence frontiers and propose a novel statistical summary in Sec . 2 . We establish non-asymptotic bounds for the estimation of divergence frontiers in Sec . 3 , and discuss the choice of the quantization level by balancing the errors induced by the two approximations . We show how smoothed distribution estimators , such as the add-constant estimator and the GoodTuring estimator , improve the estimation accuracy in Sec . 4 . Finally , we demonstrate in Sec . 5 , through simulations on synthetic data as well as generative adversarial networks on images and transformer-based language models on text , that our bounds exhibit the correct dependence of the estimation error on the sample size n and the support size k. Related work . The most widely used metrics for generative models include Inception Score [ 55 ] , Fréchet Inception Distance [ 27 ] , and Kernel Inception Distance [ 4 ] . The former two are extended to conditional generative models in [ 2 ] . They summarize the performance by a single value and thus can not distinguish different failure cases , i.e. , low quality and low diversity . Motivated by this limitation , Sajjadi et al . [ 54 ] propose a metric to evaluate the quality of generative models using two separate components : precision and recall . This formulation is extended in [ 59 ] to arbitrary probability measures using a density ratio estimator , while alternative definitions based on nonparametric representations of the manifolds of the data were proposed in [ 37 ] . These notions are generalized by the divergence frontier framework of Djolonga et al . [ 18 ] . Pillutla et al . [ 49 ] propose MAUVE , an area-under-the-curve summary based on divergence frontiers for neural text generation . They find that MAUVE correlates well with human judgements on how close the machine generated text and the human text are . Another line of related work is the estimation of functionals of discrete distributions ; see [ 62 ] for an overview . In particular , estimation of KL divergences has been studied by [ 8 , 67 , 7 , 24 ] in both fixed and large alphabet regimes . These results focus on the expected L1 and L2 risks and require additional assumptions on the two distributions such as boundedness of density ratio which is not needed in our results . Recently , Sreekumar et al . [ 60 ] investigated a modern way to estimate f -divergences using neural networks . On the practical side , there is a new line of successful work that uses deep neural networks to find data-dependent quantizations for the purpose of estimating information theoretic quantities from samples [ 53 , 23 ] . Notation . Let P ( X ) be the space of probability distributions on some measurable space X . For any P , Q 2 P ( X ) , let KL ( PkQ ) be the Kullback-Leibler ( KL ) divergence between P and Q . For 2 ( 0 , 1 ) , we define the interpolated KL divergence as KL ( PkQ ) : = KL ( Pk P + ( 1 ) Q ) . For a partition S : = { S1 , . . . , Sk } of X , we define PS the quantized version of P so that PS 2 P ( S ) with PS ( Si ) = P ( Si ) for any i 2 [ k ] : = { 1 , . . . , k } . 2 Divergence frontiers . Divergence frontiers compare two distributions P and Q using a frontier of statistical divergences . Each point on the frontier compares the individual distributions against a mixture of the two . By sweeping through mixtures , the curve interpolates between measurements of two types of costs . Fig . 1 illustrates divergence frontiers , which we formally introduce below . Evaluating generative models via divergence frontiers . Consider a generative model Q 2 P ( X ) which attempts to model the target distribution P 2 P ( X ) . It has been argued in [ 54 , 37 ] that one must consider two types of costs to evaluate Q with respect to P : ( a ) a type I cost ( loss in precision ) , which is the mass of Q that has low or zero probability mass under P , and ( b ) a type II cost ( loss in recall ) , which is the mass of P that Q does not adequately capture . Suppose P and Q are uniform distributions on their supports , and R is uniform on the union of their supports . Then , the type I cost is the mass of Supp ( Q ) \ Supp ( P ) , or equivalently , the mass of Supp ( R ) \ Supp ( P ) . We measure this using the surrogate KL ( QkR ) , which is large if there exists a such that Q ( a ) is large but R ( a ) is small . Likewise , the type II cost is measured by KL ( PkR ) . When P and Q are not constrained to be uniform , it is not clear what the measure R should be . Djolonga et al . [ 18 ] propose to vary R over all possible probability measures and consider the Pareto frontier of the multi-objective optimization minR KL ( PkR ) , KL ( QkR ) . This leads to a curve called the divergence frontier , and is reminiscent of the precision-recall curve in binary classification . See [ 15 , 13 , 14 , 19 ] and references therein on trade-off curves in machine learning . Formally , it can be shown that the divergence frontier F ( P , Q ) of probability measures P and Q is carved out by mixtures R = P + ( 1 ) Q for 2 ( 0 , 1 ) ( cf . Fig . 1 ) . It admits the closed-form F ( P , Q ) = n KL ( PkR ) , KL ( QkR ) : 2 ( 0 , 1 ) o . Practical computation of divergence frontiers . In practical applications , P is a complex , highdimensional distribution which could either be discrete , as in natural language processing , or continuous , as in computer vision . Likewise , Q is often a deep generative model such as GPT-3 for text and GANs for images . It is infeasible to compute the divergence frontier F ( P , Q ) directly because we only have samples from P and the integrals or sums over Q are intractable . Therefore , the recipe used by practitioners [ 54 , 18 , 49 ] has been to ( a ) jointly quantize P and Q over a partition S = { St } kt=1 of X to obtain discrete distributions PS = ( P ( St ) ) kt=1 and QS = ( Q ( St ) ) kt=1 , ( b ) estimate the quantized distributions from samples to get P̂S and Q̂S , and ( c ) compute F ( P̂S , Q̂S ) . In practice , the best quantization schemes are data-dependent transformations such as k-means clustering or lattice-type quantization of dense representations of images or text [ 53 ] . Statistical summary of divergence frontiers . In the minimax theory of hypothesis testing , where the goal is also to study two types of errors ( yet different from the ones considered here ) , it is common to theoretically analyze their linear combination ; see , e.g. , [ 30 , Sec . 1.2 ] and [ 9 , Thm . 7 ] . In the same spirit , we consider a linear combination of the two costs , quantified by the KL divergences , L ( P , Q ) : = KL ( PkR ) + ( 1 ) KL ( QkR ) . ( 1 ) Note that R is exactly the minimizer of the linearized objective KL ( PkR ) + ( 1 ) KL ( QkR ) according to [ 18 , Props . 1 and 2 ] . L is also known as the -skew Jensen-Shannon Divergence [ 46 ] . The linearized cost L depends on the choice of . To remove this dependency , we define a novel integral summary , called the frontier integral FI ( P , Q ) of two distributions P and Q as FI ( P , Q ) : = 2 Z 1 0 L ( P , Q ) d . ( 2 ) We can interpret the frontier integral as the average linearized cost over 2 ( 0 , 1 ) . While the length of the divergence frontier can be unbounded as shown in Appx . H , the frontier integral is always bounded in [ 0 , 1 ] . Moreover , it is a symmetric divergence with FI ( P , Q ) = 0 iff P = Q ( Appx . B ) . In practice , it can be estimated using the same recipe as the divergence frontier . Error decomposition . In Sec . 3 , we decompose the error in estimating the frontier integral into two components : the statistical error of estimating the quantized distribution and the quantization error . Our goal is to derive the rate of convergence for the overall estimation error . To control the statistical error , we use a different treatment for the masses that appear in the sample and the ones that never appear ( i.e. , the missing mass ) . We obtain a high probability bound as well as a bound for its expectation , leading to upper bounds for its sample complexity and rate of convergence . These results carry over to the divergence frontiers as well . As for the quantization error , we construct a distributiondependent quantization scheme whose error is at most O ( k 1 ) , where k is the quantization level . A combination of these two bounds sheds light on the optimal choice of the quantization level . In Sec . 5 , we verify empirically the tightness of the rates on synthetic and real data .

Summary: This works studies an evaluation metric for comparing two distributions with possibly non-overlap support. The metric called "divergence frontier" was proposed in Djolonga et al. This work proved non-asymptotic bounds for quantized estimator of the metric, suggesting a good quantization level (number of bins k ~ n^{1/3}. It also suggests using smoothed empirical distribution estimators (add-constant/Good-Turing) to improve the estimation accuracy. The experiments are difficult to understand due to low clarity.