H-divergence: A Decision-Theoretic Probability Discrepancy Measure 

1 INTRODUCTION . Quantifying the difference between two probability distributions is a fundamental problem in machine learning . Modelers choose different types of discrepancies , or probability divergences , to encode their prior knowledge , i.e . which aspects should be considered to evaluate the difference , and how they should be weighted . The divergences used in machine learning typically fall into two categories , integral probability metrics ( IPMs , Müller ( 1997 ) ) , and f -divergences ( Csiszár , 1964 ) . IPMs , such as the Wasserstein distance , maximum mean discrepancy ( MMD ) , are based on the idea that if two distributions are identical , any function should have the same expectation under both distributions . IPM is defined as the maximum difference in expectation for a set of functions . IPMs are used to define training objectives for generative models ( Arjovsky et al. , 2017 ) , perform independence tests ( Doran et al. , 2014 ) , robust optimization ( Esfahani & Kuhn , 2018 ) among many other applications . On the other hand , f -divergences , such as the KL divergence and the Jensen Shannon divergence , and are based on the idea that if two distributions are identical , they assign the same likelihood to every point , so the ratio of the likelihood always equals one . One can define a distance based on the how the likelihood ratio differs from one . KL divergence underlies some of the most commonly used training objectives for both supervised and unsupervised machine learning algorithms , such as minimizing the cross entropy loss . We propose a third category of divergences called H-divergences that overlaps with but does not equate the set of integral probability metrics or the set f -divergences . Our distance is based on a generalization ( DeGroot et al. , 1962 ) of Shannon entropy and the quadratic entropy ( Burbea & Rao , 1982 ) . Instead of measuring the best average code length of any encoding scheme ( Shannon entropy ) , the generalized entropy can choose any loss function ( rather than code length ) and set of actions ( rather than encoding schemes ) , and is defined as the best expected loss among the set of actions . In particular , given two distribution p and q , we compare the generalized entropy of the mixture distribution ( p + q ) /2 and the generalized entropy of p and q individually . Intuitively , if p and q are different , it is more difficult to minimize expected loss under the mixture distribution ( p+ q ) /2 , and hence the mixture distribution should have higher generalized entropy ; if p and q are identical , then the mixture distribution is identical to p or q , and hence should have the same generalized entropy . We define the divergence based on the difference between entropy of the mixture distribution and the entropy of individual distributions . Our distance strictly generalizes the maximum mean discrepancy and the Jensen Shannon divergence . We illustrate this via the Venn diagram in Figure 1 . This generalization allows us to choose special losses and actions spaces to leverage inductive biases and machine learning models from different problem domains . For example , if we choose the generalized entropy as the maximum log likelihood of deep generative models , we are able to recover a distance that works well for distributions over high dimensional images . To demonstrate the empirical utility of our proposed divergence , we use it for the task of two sample test , where the goal is to identify whether two sets of samples come from the same distribution or not . A test based on a probability discrepancy declares two sets of samples different if their discrepancy exceed some threshold . We use H-divergences based on generalized entropy defined by the log likelihood of off-the-shelf generative models . Compared to state-of-the-art tests based on e.g . MMD with deep kernels ( Liu et al. , 2020 ) , tests based on the H-divergence achieve better test power on a large set of benchmark datasets . As another application , we use H-divergence for sample quality evaluation , where the goal is to compare a set of samples ( e.g . generated images from a GAN ) with ground truth samples ( e.g . real images ) . We show that H-divergences generally monotonically increase with the amount of corruption added to the samples ( which should lead to worse sample quality ) , even in certain situations where the FID score ( Heusel et al. , 2017 ) is not monotonically increasing . Finally we show that H-Divergence can be used to understand whether distribution change affect decision making . As an illustrative example , we study whether climate change affect decision making in agriculture and energy production . Traditional divergences ( such as KL ) let policy makers measure if the climate has changed ; H-Divergence can provide additional information on whether the change is relevant to decision making for different social and economic activities . 2 BACKGROUND . 2.1 PROBABILITY DISTANCES . Let X denote a finite set or a finite dimensional vector space , and P ( X ) denote the set of probability distributions on X that have a density . We consider the problem of defining a probability divergence between any two distributions in P ( X ) , where a probability divergence is any function D : P ( X ) × P ( X ) → R that satisfies D ( p‖q ) ≥ 0 , D ( p‖p ) = 0 , ∀p , q ∈ P ( X ) ( Note that in general a divergence does not require D ( p‖q ) > 0 ∀p 6= q ) . Integral Probability Metrics Let F denote some set of functions X → R. The integral probability metrics is defined as IPMF ( p‖q ) = sup f∈F |Ep [ f ( X ) ] − Eq [ f ( X ) ] | Several important divergences belong to integral probability metrics . Examples include the Wasserstein distance , where F is the set of 1-Lipschitz functions ; the total variation distance , where F is the set of functions X → [ −1 , 1 ] . The maximum mean discrepancy ( MMD ) ( Rao , 1982 ; Burbea & Rao , 1984 ; Gretton et al. , 2012 ) chooses a kernel function k : X × X → R+ and is defined by MMD ( p‖q ) = Ep , pk ( X , Y ) + Eq , qk ( X , Y ) − 2Ep , qk ( X , Y ) MMD is an IPM where F is the unit norm functions in the RKHS associated with the kernel k. f -Divergences Choose any convex continuous function f : R+ → R such that f ( 1 ) = 0 , the f -Divergence is defined as ( assuming densities exist ) Df ( p‖q ) = Eq [ f ( p ( X ) /q ( X ) ) ] . Examples of f -Divergences include the KL divergence , where f : t 7→ t log t and the Jensen Shannon divergence , where f : t 7→ ( t+ 1 ) log ( 2 t+1 ) + t log t. Scoring Rule Distance ( Grünwald et al. , 2004 ; Gneiting & Raftery , 2007 ) Another large class of probability distances are defined by proper scoring rules . A function S : P ( X ) × P ( X ) → R is called a proper scoring rule if ∀p , q ∈ P ( X ) we have S ( p , q ) ≥ S ( p , p ) . Intuitively it is any function that is small when two distributions are identical and large when two distributions differ . Given a scoring rule S we can define a distance by DS ( p‖q ) = S ( p , q ) − S ( p , p ) . 2.2 H-ENTROPY . For any action space A and any loss function ` : X ×A → R , the H-entropy ( DeGroot et al. , 1962 ; DeGroot , 2005 ; Grünwald et al. , 2004 ) is defined as H ` ( p ) = infa∈A Ep [ ` ( X , a ) ] . In words , H-entropy is the Bayes optimal loss of a decision maker who must select some action a not for a particular x , but for an expectation over p ( x ) . H-entropy generalizes several important notions of uncertainty . Examples include : Shannon Entropy , where A as the set of probabilities P ( X ) , and ` ( x , a ) = − log a ( x ) ; Variance where A = X , and ` ( x , a ) = ‖x − a‖22 ; Predictive V-entropy , where A ⊂ P ( X ) is some subset of distributions , and ` ( x , a ) = − log a ( x ) ( Xu et al. , 2020 ) . The most important property that we will use is that the H entropy is concave . Lemma 1 . ( DeGroot et al. , 1962 ) For any choice of ` : X ×A → R , H ` is a concave function . This Lemma can be proved by observing that inf is a concave function , i.e. , it is always better to pick an optimal action for p and q separately rather than a single one for both . H ` ( αp+ ( 1− α ) q ) = inf a ( αEp [ ` ( X , a ) ] + ( 1− α ) Eq [ ` ( X , a ) ] ) ≥ α inf a Ep [ ` ( X , a ) ] + ( 1− α ) inf a Eq [ ` ( X , a ) ] = αH ` ( p ) + ( 1− α ) H ` ( q ) This Lemma reflects why H ` can be thought of as a measurement of entropy or uncertainty . If the distribution is more uncertain ( e.g . mixture of p and q rather than p and q separately ) then the optimal action always suffers a higher loss . 3 DEFINITION AND THEORETICAL PROPERTIES . 3.1 H-JENSEN SHANNON DIVERGENCE . As a warm up , we first present a special case of our definition . Definition 1 ( H-Jensen Shannon divergence ) . DJS ` ( p , q ) = H ` ( p+ q 2 ) − 1 2 ( H ` ( p ) +H ` ( q ) ) ( 1 ) The above is a divergence between p and q because H-entropy is concave , so DJS ` is always nonnegative . In particular , if we choose H ` as the Shannon entropy , Definition 1 recovers the usual Jensen Shannon divergence . Other special choices of entropy can also recover definitions in ( Burbea & Rao , 1982 ) . In addition , we can define a divergence for any convex combination αp + ( 1 − α ) q where α ∈ ( 0 , 1 ) but for this paper we only consider α = 1/2 . 3.2 GENERAL H-DIVERGENCE . In addition to the H-Jensen Shannon divergence , there are other functions based on the H-entropy that satisfy the requirements of a divergence . For example , the following quantity DMin ` = H ` ( p+ q 2 ) −min ( H ` ( p ) , H ` ( q ) ) ( 2 ) is also a valid divergence ( this will be proved later as a special case of Lemma 2 ) . We can define a general set of divergences that includes the above two divergences with the following definition : Definition 2 ( H-divergence ) . For two distributions p , q on X , choose any continuous function φ : R2 → R such that φ ( θ , λ ) > 0 whenever θ + λ > 0 and φ ( θ , λ ) = 0 whenever θ + λ = 0 , define Dφ ` ( p‖q ) = φ ( H ` ( p+ q 2 ) −H ` ( p ) , H ` ( p+ q 2 ) −H ` ( q ) ) Intuitively H ` ( p+q 2 ) − H ` ( p ) and H ` ( p+q 2 ) − H ` ( q ) measure how much more difficult it is to minimize loss on the mixture distribution ( p+ q ) /2 than on p and q respectively . φ is a general class of function that could convert these differences into a divergence , while satisfying the desirable properties in the next section . The H-divergence generalizes all the previous definitions , as shown by the following proposition . Therefore any property of H-divergence is inherited by e.g . H-Jensen Shannon divergence . Proposition 1 . Choose φ ( θ , λ ) = θ+λ2 thenD φ ` ( p , q ) is the H-Jensen Shannon divergence in Eq. ( 1 ) . Choose φ ( θ , λ ) = max ( θ , λ ) then Dφ ` ( p , q ) is the H-Min divergence in Eq . ( 2 ) .

Summary: The distance or divergence between two probability distributions is essential for machine learning. This paper introduces a new class of divergence functions based on optimal decision loss function. They first introduce a class of entropy functional, namely the loss function depending on the action and state. This type of function extends the classical entropy function, including negative Boltzman-Shannon entropy. Using it, they further construct a divergence based on the mixture of probability densities.  Several propositions and numerical experiments demonstrate the effectiveness of proposed divergence functions.