Comparing Distributions by Measuring Differences that Affect Decision Making

1 Introduction . Quantifying the difference between two probability distributions is a fundamental problem in machine learning . Modelers choose different types of discrepancies ( or probability divergences ) to encode their prior knowledge about which aspects are relevant to evaluate the difference . Integral probability metrics ( IPMs , Müller ( 1997 ) ) and f -divergences ( Csiszár , 1964 ) are widely used discrepancies in machine learning . IPMs , such as the Wasserstein distance , maximum mean discrepancy ( MMD ) ( Rao , 1982 ; Burbea & Rao , 1984 ; Gretton et al. , 2012 ) , are based on the idea that if two distributions are identical , any function should have the same expectation under both distributions . IPMs are used to define training objectives for generative models ( Arjovsky et al. , 2017 ) , perform independence tests ( Doran et al. , 2014 ) , robust optimization ( Esfahani & Kuhn , 2018 ) among many other applications . f -divergences , such as the KL divergence and the Jensen Shannon divergence , are based on the idea that if two distributions are identical , they assign the same likelihood to every point . One can then define a discrepancy based on how different the likelihood ratio is from one . KL divergence underlies some of the most commonly used training objectives for both supervised and unsupervised machine learning algorithms , such as cross entropy loss . We propose a third category of divergences called H-divergences that overlaps with but also extends the set of integral probability metrics or the set f -divergences . Intuitively , H-divergence compares two distributions in terms of the optimal loss for a certain decision task . This optimal loss corresponds to a generalized notion of entropy ( DeGroot et al. , 1962 ) . Instead of measuring the best average code length of any encoding scheme ( Shannon entropy ) , the generalized entropy uses arbitrary loss function ( rather than code length ) and set of actions ( rather than encoding schemes ) , and is defined as the best expected loss among the set of actions . In particular , given two distribution p and q , we compare the generalized entropy of the mixture distribution ( p + q ) /2 and the generalized entropy of p and q individually . Intuitively , if p and q are different , it is more difficult to minimize expected loss under the mixture distribution ( p+ q ) /2 , and hence the mixture distribution should have higher generalized entropy ; if p and q are identical , then the mixture distribution is identical to p or q , and hence should have the same generalized entropy . Our divergence strictly generalizes the maximum mean discrepancy family and the Jensen Shannon divergence , which can be obtained with specific choices of the loss function . We illustrate this via the Venn diagram in Figure 1 . Our formulation allows us to choose alternative losses to leverage inductive biases and machine learning models from different problem domains . For example , if we choose the generalized entropy as the maximum log likelihood of deep generative models , we are able to leverage recent progress in modeling high dimensional images . We demonstrate the effectiveness of H-divergence in two sample tests , i.e . to decide whether two sets of samples come from the same distribution or not . A test based on a probability discrepancy declares two sets of samples different if their discrepancy exceeds some threshold . We use H-divergences based on generalized entropy defined by the log likelihood of offthe-shelf generative models . Compared to state-of-the-art tests based on MMD with deep kernels ( Liu et al. , 2020 ) , tests based on the H-divergence achieve better test power ( given identical type I error ) on a large set of benchmarks . More importantly , scientists and policy makers are often interested not only in if two distributions are different , but how two distributions are different and whether the differences affect decision making . Typical divergence measures ( such as KL ) or two sample tests only quantify if two distributions are different , while we show that H-divergence is a useful tool for quantifying how distributions are different with three application examples : studying the effect of climate change , feature selection , and sample quality evaluation . In each of these examples , we compare different aspects of the distributions by choosing specific decision loss functions . For example , climate change ( Figure 3 ) might impact agriculture in a region but not energy production , or vice versa . By choosing suitable loss functions ( related to agriculture , energy , etc ) we can quantify and test if the change in climate distribution impact different economic activities . 2 Background . 2.1 Probability Divergences . Let X denote a finite set or a finite dimensional vector space , and P ( X ) denote the set of probability distributions on X that have a density . We consider the problem of defining a probability divergence between any two distributions in P ( X ) , where a probability divergence is any function D : P ( X ) × P ( X ) → R that satisfies D ( p‖q ) ≥ 0 , D ( p‖p ) = 0 , ∀p , q ∈ P ( X ) . We call the divergence D “ strict ” if D ( p‖q ) > 0 ∀p 6= q , and “ non-strict “ otherwise . In this paper we consider both types of divergences . Integral Probability Metrics Let F denote a set of functions X → R. An integral probability metric is defined as IPMF ( p‖q ) = supf∈F |Ep [ f ( X ) ] −Eq [ f ( X ) ] | . Several important divergences belong to integral probability metrics . Examples include the Wasserstein distance , where F is the set of 1-Lipschitz functions ; the total variation distance , where F is the set of functions X → [ −1 , 1 ] . The maximum mean discrepancy ( MMD ) ( Rao , 1982 ; Burbea & Rao , 1984 ; Gretton et al. , 2012 ) chooses a kernel function k : X × X → R+ and is defined by MMD2 ( p‖q ) = Ep , pk ( X , Y ) + Eq , qk ( X , Y ) − 2Ep , qk ( X , Y ) MMD is an IPM where F is the unit norm functions in the reproducing kernel Hilbert space ( RKHS ) associated with the kernel k. f-Divergences Given any convex continuous function f : R+ → R such that f ( 1 ) = 0 , the f -Divergence is defined as ( assuming densities exist ) Df ( p‖q ) = Eq [ f ( p ( X ) /q ( X ) ) ] . Examples include the KL divergence , where f : t 7→ t log t and the Jensen Shannon divergence , where f : t 7→ ( t+ 1 ) log ( 2 t+1 ) + t log t . 2.2 H-Entropy . For any action space A and any loss function ` : X ×A → R , the H-entropy ( DeGroot et al. , 1962 ; DeGroot , 2005 ; Grünwald et al. , 2004 ) is defined as H ` ( p ) = inf a∈A Ep [ ` ( X , a ) ] In words , H-entropy is the Bayes optimal loss of a decision maker who must select some action a not for a particular x , but in expectation for a random x drawn from p ( x ) . Hentropy generalizes several important notions of uncertainty . Examples include : Shannon Entropy , where A as the set of probabilities P ( X ) , and ` ( x , a ) = − log a ( x ) ; Variance where A = X , and ` ( x , a ) = ‖x − a‖22 ; Predictive V-entropy , where A ⊂ P ( X ) is some subset of distributions , and ` ( x , a ) = − log a ( x ) ( Xu et al. , 2020 ) . A key property we will use is that H-entropy is concave ( DeGroot et al. , 1962 ) . Lemma 1 . For any choice of ` : X ×A → R , H ` is a concave function . This Lemma can be proved by observing that inf is a concave function : it is always better to pick an optimal action for p and q separately rather than a single one for both . H ` ( αp+ ( 1− α ) q ) = inf a ( αEp [ ` ( X , a ) ] + ( 1− α ) Eq [ ` ( X , a ) ] ) ≥ α inf a Ep [ ` ( X , a ) ] + ( 1− α ) inf a Eq [ ` ( X , a ) ] = αH ` ( p ) + ( 1− α ) H ` ( q ) This Lemma reflects why H ` can be thought of as a measurement of entropy or uncertainty . If the distribution is more uncertain ( e.g . a mixture of p and q , rather than p or q separately ) then decisions made under higher uncertainty will suffer a higher loss . 3 Definition and Theoretical Properties . 3.1 H-Jensen Shannon Divergence . As a warm up , we present a special case of our divergence . Definition 1 ( H-Jensen Shannon divergence ) . DJS ` ( p , q ) = H ` ( p+ q 2 ) − 1 2 ( H ` ( p ) +H ` ( q ) ) ( 1 ) DJS ` is always non-negative because H-entropy is concave ( Lemma 1 ) , and clearly D JS ` ( p , q ) = 0 whenever p = q . Therefore , DJS ` is a valid probability divergence . In particular , if we choose H ` as the Shannon entropy , Definition 1 recovers the Jensen Shannon divergence . Other special loss function choices can recover definitions in ( Burbea & Rao , 1982 ) . 3.2 General H-divergence . In addition to the H-Jensen Shannon divergence , there are other functions based on the H-entropy that satisfy the requirements of a divergence . For example , DMin ` = H ` ( p+ q 2 ) −min ( H ` ( p ) , H ` ( q ) ) ( 2 ) is also a valid divergence ( this will be proved later as a special case of Lemma 2 ) . We can define a general set of divergences that includes the above two divergences with the following definition : Definition 2 ( H-divergence ) . For two distributions p , q on X , given any continuous function φ : R2 → R such that φ ( θ , λ ) > 0 whenever θ + λ > 0 and φ ( 0 , 0 ) = 0 , define Dφ ` ( p‖q ) = φ ( H ` ( p+ q 2 ) −H ` ( p ) , H ` ( p+ q 2 ) −H ` ( q ) ) Intuitively H ` ( p+q 2 ) −H ` ( p ) and H ` ( p+q 2 ) −H ` ( q ) measure how much more difficult it is to minimize loss on the mixture distribution ( p+ q ) /2 than on p and q respectively . φ is a general class of functions that map these differences into a scalar divergence , while satisfying some desirable properties described in the next section . The following proposition shows that the H-divergence generalizes the previous definitions ( 1 ) and ( 2 ) . Therefore , any property of H-divergence is inherited by e.g . the H-Jensen Shannon divergence . Proposition 1 . If φ ( θ , λ ) = θ+λ2 then D φ ` ( p , q ) is the H-Jensen Shannon divergence in Eq. ( 1 ) . If φ ( θ , λ ) = max ( θ , λ ) then Dφ ` ( p , q ) is the H-Min divergence in Eq . ( 2 ) .

Summary: This paper proposes H-divergence, a new type of divergence based on H-entropy, to compare two probability distributions. This new divergence includes some of the commonly used integral probability metrics and f-divergences, such as Jensen Shannon divergence and MMD as special cases. A crucial property of H-divergence is that it takes into account the decision loss; namely, it compares two distributions in a way that distinguishes them based on the optimal decision loss, i.e., "two distributions are different if the optimal decision loss is higher on their mixture than on each individual distribution." It also provides an empirical estimator for H-divergence and studies its convergence properties. The paper studies several use cases of H-divergence, including two-sample tests. Experiments demonstrate that H-divergence achieves higher test power than tests based on MMD under the same type I error rate. Another important use case of H-divergence is understanding whether differences between distributions are significant enough to affect decision making (from the viewpoint of minimizing loss) in different experiments, including how climate change affects various economic activities.