Comparing Distributions by Measuring Differences that Affect Decision Making

1 Introduction . Quantifying the difference between two probability distributions is a fundamental problem in machine learning . Modelers choose different types of discrepancies ( or probability divergences ) to encode their prior knowledge about which aspects are relevant to evaluate the difference . Integral probability metrics ( IPMs , Müller ( 1997 ) ) and f -divergences ( Csiszár , 1964 ) are widely used discrepancies in machine learning . IPMs , such as the Wasserstein distance , maximum mean discrepancy ( MMD ) ( Rao , 1982 ; Burbea & Rao , 1984 ; Gretton et al. , 2012 ) , are based on the idea that if two distributions are identical , any function should have the same expectation under both distributions . IPMs are used to define training objectives for generative models ( Arjovsky et al. , 2017 ) , perform independence tests ( Doran et al. , 2014 ) , robust optimization ( Esfahani & Kuhn , 2018 ) among many other applications . f -divergences , such as the KL divergence and the Jensen Shannon divergence , are based on the idea that if two distributions are identical , they assign the same likelihood to every point . One can then define a discrepancy based on how different the likelihood ratio is from one . KL divergence underlies some of the most commonly used training objectives for both supervised and unsupervised machine learning algorithms , such as cross entropy loss . We propose a third category of divergences called H-divergences that overlaps with but also extends the set of integral probability metrics or the set f -divergences . Intuitively , H-divergence compares two distributions in terms of the optimal loss for a certain decision task . This optimal loss corresponds to a generalized notion of entropy ( DeGroot et al. , 1962 ) . Instead of measuring the best average code length of any encoding scheme ( Shannon entropy ) , the generalized entropy uses arbitrary loss function ( rather than code length ) and set of actions ( rather than encoding schemes ) , and is defined as the best expected loss among the set of actions . In particular , given two distribution p and q , we compare the generalized entropy of the mixture distribution ( p + q ) /2 and the generalized entropy of p and q individually . Intuitively , if p and q are different , it is more difficult to minimize expected loss under the mixture distribution ( p+ q ) /2 , and hence the mixture distribution should have higher generalized entropy ; if p and q are identical , then the mixture distribution is identical to p or q , and hence should have the same generalized entropy . Our divergence strictly generalizes the maximum mean discrepancy family and the Jensen Shannon divergence , which can be obtained with specific choices of the loss function . We illustrate this via the Venn diagram in Figure 1 . Our formulation allows us to choose alternative losses to leverage inductive biases and machine learning models from different problem domains . For example , if we choose the generalized entropy as the maximum log likelihood of deep generative models , we are able to leverage recent progress in modeling high dimensional images . We demonstrate the effectiveness of H-divergence in two sample tests , i.e . to decide whether two sets of samples come from the same distribution or not . A test based on a probability discrepancy declares two sets of samples different if their discrepancy exceeds some threshold . We use H-divergences based on generalized entropy defined by the log likelihood of offthe-shelf generative models . Compared to state-of-the-art tests based on MMD with deep kernels ( Liu et al. , 2020 ) , tests based on the H-divergence achieve better test power ( given identical type I error ) on a large set of benchmarks . More importantly , scientists and policy makers are often interested not only in if two distributions are different , but how two distributions are different and whether the differences affect decision making . Typical divergence measures ( such as KL ) or two sample tests only quantify if two distributions are different , while we show that H-divergence is a useful tool for quantifying how distributions are different with three application examples : studying the effect of climate change , feature selection , and sample quality evaluation . In each of these examples , we compare different aspects of the distributions by choosing specific decision loss functions . For example , climate change ( Figure 3 ) might impact agriculture in a region but not energy production , or vice versa . By choosing suitable loss functions ( related to agriculture , energy , etc ) we can quantify and test if the change in climate distribution impact different economic activities . 2 Background . 2.1 Probability Divergences . Let X denote a finite set or a finite dimensional vector space , and P ( X ) denote the set of probability distributions on X that have a density . We consider the problem of defining a probability divergence between any two distributions in P ( X ) , where a probability divergence is any function D : P ( X ) × P ( X ) → R that satisfies D ( p‖q ) ≥ 0 , D ( p‖p ) = 0 , ∀p , q ∈ P ( X ) . We call the divergence D “ strict ” if D ( p‖q ) > 0 ∀p 6= q , and “ non-strict “ otherwise . In this paper we consider both types of divergences . Integral Probability Metrics Let F denote a set of functions X → R. An integral probability metric is defined as IPMF ( p‖q ) = supf∈F |Ep [ f ( X ) ] −Eq [ f ( X ) ] | . Several important divergences belong to integral probability metrics . Examples include the Wasserstein distance , where F is the set of 1-Lipschitz functions ; the total variation distance , where F is the set of functions X → [ −1 , 1 ] . The maximum mean discrepancy ( MMD ) ( Rao , 1982 ; Burbea & Rao , 1984 ; Gretton et al. , 2012 ) chooses a kernel function k : X × X → R+ and is defined by MMD2 ( p‖q ) = Ep , pk ( X , Y ) + Eq , qk ( X , Y ) − 2Ep , qk ( X , Y ) MMD is an IPM where F is the unit norm functions in the reproducing kernel Hilbert space ( RKHS ) associated with the kernel k. f-Divergences Given any convex continuous function f : R+ → R such that f ( 1 ) = 0 , the f -Divergence is defined as ( assuming densities exist ) Df ( p‖q ) = Eq [ f ( p ( X ) /q ( X ) ) ] . Examples include the KL divergence , where f : t 7→ t log t and the Jensen Shannon divergence , where f : t 7→ ( t+ 1 ) log ( 2 t+1 ) + t log t . 2.2 H-Entropy . For any action space A and any loss function ` : X ×A → R , the H-entropy ( DeGroot et al. , 1962 ; DeGroot , 2005 ; Grünwald et al. , 2004 ) is defined as H ` ( p ) = inf a∈A Ep [ ` ( X , a ) ] In words , H-entropy is the Bayes optimal loss of a decision maker who must select some action a not for a particular x , but in expectation for a random x drawn from p ( x ) . Hentropy generalizes several important notions of uncertainty . Examples include : Shannon Entropy , where A as the set of probabilities P ( X ) , and ` ( x , a ) = − log a ( x ) ; Variance where A = X , and ` ( x , a ) = ‖x − a‖22 ; Predictive V-entropy , where A ⊂ P ( X ) is some subset of distributions , and ` ( x , a ) = − log a ( x ) ( Xu et al. , 2020 ) . A key property we will use is that H-entropy is concave ( DeGroot et al. , 1962 ) . Lemma 1 . For any choice of ` : X ×A → R , H ` is a concave function . This Lemma can be proved by observing that inf is a concave function : it is always better to pick an optimal action for p and q separately rather than a single one for both . H ` ( αp+ ( 1− α ) q ) = inf a ( αEp [ ` ( X , a ) ] + ( 1− α ) Eq [ ` ( X , a ) ] ) ≥ α inf a Ep [ ` ( X , a ) ] + ( 1− α ) inf a Eq [ ` ( X , a ) ] = αH ` ( p ) + ( 1− α ) H ` ( q ) This Lemma reflects why H ` can be thought of as a measurement of entropy or uncertainty . If the distribution is more uncertain ( e.g . a mixture of p and q , rather than p or q separately ) then decisions made under higher uncertainty will suffer a higher loss . 3 Definition and Theoretical Properties . 3.1 H-Jensen Shannon Divergence . As a warm up , we present a special case of our divergence . Definition 1 ( H-Jensen Shannon divergence ) . DJS ` ( p , q ) = H ` ( p+ q 2 ) − 1 2 ( H ` ( p ) +H ` ( q ) ) ( 1 ) DJS ` is always non-negative because H-entropy is concave ( Lemma 1 ) , and clearly D JS ` ( p , q ) = 0 whenever p = q . Therefore , DJS ` is a valid probability divergence . In particular , if we choose H ` as the Shannon entropy , Definition 1 recovers the Jensen Shannon divergence . Other special loss function choices can recover definitions in ( Burbea & Rao , 1982 ) . 3.2 General H-divergence . In addition to the H-Jensen Shannon divergence , there are other functions based on the H-entropy that satisfy the requirements of a divergence . For example , DMin ` = H ` ( p+ q 2 ) −min ( H ` ( p ) , H ` ( q ) ) ( 2 ) is also a valid divergence ( this will be proved later as a special case of Lemma 2 ) . We can define a general set of divergences that includes the above two divergences with the following definition : Definition 2 ( H-divergence ) . For two distributions p , q on X , given any continuous function φ : R2 → R such that φ ( θ , λ ) > 0 whenever θ + λ > 0 and φ ( 0 , 0 ) = 0 , define Dφ ` ( p‖q ) = φ ( H ` ( p+ q 2 ) −H ` ( p ) , H ` ( p+ q 2 ) −H ` ( q ) ) Intuitively H ` ( p+q 2 ) −H ` ( p ) and H ` ( p+q 2 ) −H ` ( q ) measure how much more difficult it is to minimize loss on the mixture distribution ( p+ q ) /2 than on p and q respectively . φ is a general class of functions that map these differences into a scalar divergence , while satisfying some desirable properties described in the next section . The following proposition shows that the H-divergence generalizes the previous definitions ( 1 ) and ( 2 ) . Therefore , any property of H-divergence is inherited by e.g . the H-Jensen Shannon divergence . Proposition 1 . If φ ( θ , λ ) = θ+λ2 then D φ ` ( p , q ) is the H-Jensen Shannon divergence in Eq. ( 1 ) . If φ ( θ , λ ) = max ( θ , λ ) then Dφ ` ( p , q ) is the H-Min divergence in Eq . ( 2 ) .

Summary: This paper proposes a new category of divergences, called H-divergence, for measuring the discrepancy  between two probability distributions. H-divergence is based on the optimal loss with respect to a chosen decision task and is therefore decision dependent. Further, it generalizes a few well known divergences such as the Jensen-Shannon divergence and the maximum mean discrepancy family. The paper proves a few properties of H-divergence including a convergence result when H-divergence is estimated using a finite set of samples. Further, three examples are used to illustrate the applications of H-divergence  including two sample tests, assessing climate change, and feature selection, which demonstrate the advantage of H-divergence compared with other commonly used discrepancies.