H-divergence: A Decision-Theoretic Probability Discrepancy Measure 

1 INTRODUCTION . Quantifying the difference between two probability distributions is a fundamental problem in machine learning . Modelers choose different types of discrepancies , or probability divergences , to encode their prior knowledge , i.e . which aspects should be considered to evaluate the difference , and how they should be weighted . The divergences used in machine learning typically fall into two categories , integral probability metrics ( IPMs , Müller ( 1997 ) ) , and f -divergences ( Csiszár , 1964 ) . IPMs , such as the Wasserstein distance , maximum mean discrepancy ( MMD ) , are based on the idea that if two distributions are identical , any function should have the same expectation under both distributions . IPM is defined as the maximum difference in expectation for a set of functions . IPMs are used to define training objectives for generative models ( Arjovsky et al. , 2017 ) , perform independence tests ( Doran et al. , 2014 ) , robust optimization ( Esfahani & Kuhn , 2018 ) among many other applications . On the other hand , f -divergences , such as the KL divergence and the Jensen Shannon divergence , and are based on the idea that if two distributions are identical , they assign the same likelihood to every point , so the ratio of the likelihood always equals one . One can define a distance based on the how the likelihood ratio differs from one . KL divergence underlies some of the most commonly used training objectives for both supervised and unsupervised machine learning algorithms , such as minimizing the cross entropy loss . We propose a third category of divergences called H-divergences that overlaps with but does not equate the set of integral probability metrics or the set f -divergences . Our distance is based on a generalization ( DeGroot et al. , 1962 ) of Shannon entropy and the quadratic entropy ( Burbea & Rao , 1982 ) . Instead of measuring the best average code length of any encoding scheme ( Shannon entropy ) , the generalized entropy can choose any loss function ( rather than code length ) and set of actions ( rather than encoding schemes ) , and is defined as the best expected loss among the set of actions . In particular , given two distribution p and q , we compare the generalized entropy of the mixture distribution ( p + q ) /2 and the generalized entropy of p and q individually . Intuitively , if p and q are different , it is more difficult to minimize expected loss under the mixture distribution ( p+ q ) /2 , and hence the mixture distribution should have higher generalized entropy ; if p and q are identical , then the mixture distribution is identical to p or q , and hence should have the same generalized entropy . We define the divergence based on the difference between entropy of the mixture distribution and the entropy of individual distributions . Our distance strictly generalizes the maximum mean discrepancy and the Jensen Shannon divergence . We illustrate this via the Venn diagram in Figure 1 . This generalization allows us to choose special losses and actions spaces to leverage inductive biases and machine learning models from different problem domains . For example , if we choose the generalized entropy as the maximum log likelihood of deep generative models , we are able to recover a distance that works well for distributions over high dimensional images . To demonstrate the empirical utility of our proposed divergence , we use it for the task of two sample test , where the goal is to identify whether two sets of samples come from the same distribution or not . A test based on a probability discrepancy declares two sets of samples different if their discrepancy exceed some threshold . We use H-divergences based on generalized entropy defined by the log likelihood of off-the-shelf generative models . Compared to state-of-the-art tests based on e.g . MMD with deep kernels ( Liu et al. , 2020 ) , tests based on the H-divergence achieve better test power on a large set of benchmark datasets . As another application , we use H-divergence for sample quality evaluation , where the goal is to compare a set of samples ( e.g . generated images from a GAN ) with ground truth samples ( e.g . real images ) . We show that H-divergences generally monotonically increase with the amount of corruption added to the samples ( which should lead to worse sample quality ) , even in certain situations where the FID score ( Heusel et al. , 2017 ) is not monotonically increasing . Finally we show that H-Divergence can be used to understand whether distribution change affect decision making . As an illustrative example , we study whether climate change affect decision making in agriculture and energy production . Traditional divergences ( such as KL ) let policy makers measure if the climate has changed ; H-Divergence can provide additional information on whether the change is relevant to decision making for different social and economic activities . 2 BACKGROUND . 2.1 PROBABILITY DISTANCES . Let X denote a finite set or a finite dimensional vector space , and P ( X ) denote the set of probability distributions on X that have a density . We consider the problem of defining a probability divergence between any two distributions in P ( X ) , where a probability divergence is any function D : P ( X ) × P ( X ) → R that satisfies D ( p‖q ) ≥ 0 , D ( p‖p ) = 0 , ∀p , q ∈ P ( X ) ( Note that in general a divergence does not require D ( p‖q ) > 0 ∀p 6= q ) . Integral Probability Metrics Let F denote some set of functions X → R. The integral probability metrics is defined as IPMF ( p‖q ) = sup f∈F |Ep [ f ( X ) ] − Eq [ f ( X ) ] | Several important divergences belong to integral probability metrics . Examples include the Wasserstein distance , where F is the set of 1-Lipschitz functions ; the total variation distance , where F is the set of functions X → [ −1 , 1 ] . The maximum mean discrepancy ( MMD ) ( Rao , 1982 ; Burbea & Rao , 1984 ; Gretton et al. , 2012 ) chooses a kernel function k : X × X → R+ and is defined by MMD ( p‖q ) = Ep , pk ( X , Y ) + Eq , qk ( X , Y ) − 2Ep , qk ( X , Y ) MMD is an IPM where F is the unit norm functions in the RKHS associated with the kernel k. f -Divergences Choose any convex continuous function f : R+ → R such that f ( 1 ) = 0 , the f -Divergence is defined as ( assuming densities exist ) Df ( p‖q ) = Eq [ f ( p ( X ) /q ( X ) ) ] . Examples of f -Divergences include the KL divergence , where f : t 7→ t log t and the Jensen Shannon divergence , where f : t 7→ ( t+ 1 ) log ( 2 t+1 ) + t log t. Scoring Rule Distance ( Grünwald et al. , 2004 ; Gneiting & Raftery , 2007 ) Another large class of probability distances are defined by proper scoring rules . A function S : P ( X ) × P ( X ) → R is called a proper scoring rule if ∀p , q ∈ P ( X ) we have S ( p , q ) ≥ S ( p , p ) . Intuitively it is any function that is small when two distributions are identical and large when two distributions differ . Given a scoring rule S we can define a distance by DS ( p‖q ) = S ( p , q ) − S ( p , p ) . 2.2 H-ENTROPY . For any action space A and any loss function ` : X ×A → R , the H-entropy ( DeGroot et al. , 1962 ; DeGroot , 2005 ; Grünwald et al. , 2004 ) is defined as H ` ( p ) = infa∈A Ep [ ` ( X , a ) ] . In words , H-entropy is the Bayes optimal loss of a decision maker who must select some action a not for a particular x , but for an expectation over p ( x ) . H-entropy generalizes several important notions of uncertainty . Examples include : Shannon Entropy , where A as the set of probabilities P ( X ) , and ` ( x , a ) = − log a ( x ) ; Variance where A = X , and ` ( x , a ) = ‖x − a‖22 ; Predictive V-entropy , where A ⊂ P ( X ) is some subset of distributions , and ` ( x , a ) = − log a ( x ) ( Xu et al. , 2020 ) . The most important property that we will use is that the H entropy is concave . Lemma 1 . ( DeGroot et al. , 1962 ) For any choice of ` : X ×A → R , H ` is a concave function . This Lemma can be proved by observing that inf is a concave function , i.e. , it is always better to pick an optimal action for p and q separately rather than a single one for both . H ` ( αp+ ( 1− α ) q ) = inf a ( αEp [ ` ( X , a ) ] + ( 1− α ) Eq [ ` ( X , a ) ] ) ≥ α inf a Ep [ ` ( X , a ) ] + ( 1− α ) inf a Eq [ ` ( X , a ) ] = αH ` ( p ) + ( 1− α ) H ` ( q ) This Lemma reflects why H ` can be thought of as a measurement of entropy or uncertainty . If the distribution is more uncertain ( e.g . mixture of p and q rather than p and q separately ) then the optimal action always suffers a higher loss . 3 DEFINITION AND THEORETICAL PROPERTIES . 3.1 H-JENSEN SHANNON DIVERGENCE . As a warm up , we first present a special case of our definition . Definition 1 ( H-Jensen Shannon divergence ) . DJS ` ( p , q ) = H ` ( p+ q 2 ) − 1 2 ( H ` ( p ) +H ` ( q ) ) ( 1 ) The above is a divergence between p and q because H-entropy is concave , so DJS ` is always nonnegative . In particular , if we choose H ` as the Shannon entropy , Definition 1 recovers the usual Jensen Shannon divergence . Other special choices of entropy can also recover definitions in ( Burbea & Rao , 1982 ) . In addition , we can define a divergence for any convex combination αp + ( 1 − α ) q where α ∈ ( 0 , 1 ) but for this paper we only consider α = 1/2 . 3.2 GENERAL H-DIVERGENCE . In addition to the H-Jensen Shannon divergence , there are other functions based on the H-entropy that satisfy the requirements of a divergence . For example , the following quantity DMin ` = H ` ( p+ q 2 ) −min ( H ` ( p ) , H ` ( q ) ) ( 2 ) is also a valid divergence ( this will be proved later as a special case of Lemma 2 ) . We can define a general set of divergences that includes the above two divergences with the following definition : Definition 2 ( H-divergence ) . For two distributions p , q on X , choose any continuous function φ : R2 → R such that φ ( θ , λ ) > 0 whenever θ + λ > 0 and φ ( θ , λ ) = 0 whenever θ + λ = 0 , define Dφ ` ( p‖q ) = φ ( H ` ( p+ q 2 ) −H ` ( p ) , H ` ( p+ q 2 ) −H ` ( q ) ) Intuitively H ` ( p+q 2 ) − H ` ( p ) and H ` ( p+q 2 ) − H ` ( q ) measure how much more difficult it is to minimize loss on the mixture distribution ( p+ q ) /2 than on p and q respectively . φ is a general class of function that could convert these differences into a divergence , while satisfying the desirable properties in the next section . The H-divergence generalizes all the previous definitions , as shown by the following proposition . Therefore any property of H-divergence is inherited by e.g . H-Jensen Shannon divergence . Proposition 1 . Choose φ ( θ , λ ) = θ+λ2 thenD φ ` ( p , q ) is the H-Jensen Shannon divergence in Eq. ( 1 ) . Choose φ ( θ , λ ) = max ( θ , λ ) then Dφ ` ( p , q ) is the H-Min divergence in Eq . ( 2 ) .

Summary: This paper proposes a H divergence that is a generalization of many popular f divergences and IPMs. The paper gives an empirical estimator with convergence rates for this divergence, where the rates are very fast when the two distributions are equal. The paper shows how the empirical estimator has practical use for two sample tests and measuring the corruption of a sample. The proposed H divergence is "useful" when the two distributions are close to each other, but as the authors acknowledge in the future work, it is an open question whether it could be "useful" in other cases.