Maximum Entropy competes with Maximum Likelihood

1 INTRODUCTION . The maximum entropy ( MAXENT ) method was proposed within statistical physics ( Jaynes , 1957 ; Balian , 2007 ; Pressé et al. , 2013 ) , and later on got a wide range of inter-disciplinary applications in data science , probabilistic inference , biological data modeling etc ; see e.g . ( Erickson & Smith , 2013 ) . MAXENT estimates unknown probabilities ( that generated data ) via maximizing the Boltzmann-Gibbs-Shannon entropy under certain constraints which can be derived from the observed data ( Erickson & Smith , 2013 ) . MAXENT leads to non-parametric estimators whose form does not depend on the underlying mechanism that generated data ( i.e . prior assumptions ) . Also , MAXENT avoids the zero-probability problem , i.e . when operating on a sparse data—so that certain values of the involved random quantity may not appear due to a small , but non-zero probability—MAXENT still provides a controllable non-zero estimate for this small probability . MAXENT has has several formal justification ( Jaynes , 1957 ; Chakrabarti & Chakrabarty , 2005 ; Baez et al. , 2011 ; Van Campenhout & Cover , 1981 ; Topsøe , 1979 ; Shore & Johnson , 1980 ; Paris & Vencovská , 1997 ) . But the following open problems are basic for MAXENT , because their insufficient understanding prevents its valid applications . ( i ) Which constraints of entropy maximization are to be extracted from data , which is necessarily finite and noisy ? ( ii ) When and how these constraints can lead to overfitting , where —due to a noisy data—involving more constraints leads to poorer results ? ( iii ) How predictions of MAXENT compare with those of other estimators , e.g . the ( regularized ) maximum likelihood ? Here we approach these open problems via tools of Bayesian decision theory ( Cox & Hinkley , 1979 ) . We assume that the data is given as an i.i.d . sample of a finite length M from a random quantity with n outcomes and unknown probabilities that are instanced from a non-informative prior Dirichlet density , or a mixture of such densities . Focusing on the sparse data regime M < n we calculate average KL-distances between real probabilities and their estimates , decide on the quality of MAXENT under various constraints , and compare it with the ( regularized ) maximum-likelihood ( ML ) estimator . Our main results are that MAXENT does apply to sparse data , but does demand specific prior information . We explored two different scenarios of such information . First , the unknown probabilities are most probably deterministic . Second , there are prior rank correlations between the inferred random quantity and its probabilities . Moreover , in the latter case the nonparametric MAXENT estimator is better in terms of the average KL-distance than the optimally regularized ML ( parametric ) estimator . Some of above questions were already studied in literature . ( Good , 1970 ; Christensen , 1985 ; Zhu et al. , 1997 ; Pandey & Dukkipati , 2013 ) applied formal principles of statistics ( e.g . the Minimum Description Length ) to the selection of constraints ( question ( i ) ) . Our approach to studying this question will be direct and unambiguous , since , as shown below , the Bayesian decision theory leads to clear criteria for the validity of MAXENT estimators . We can also compare all predictions with the optimal Bayesian estimator . The latter is normally not available in practice due to insufficient knowledge of prior details , but it still does provide an important theoretical benchmark . Note that ( Thomas , 1979 ; Lebanon & Lafferty , 2002 ; Kazama & Tsujii , 2005 ; Altun & Smola , 2006 ; Dudik , 2007. ; Rau , 2011 ; Campbell , 1999 ; Friedlander & Gupta , 2005 ) studied soft constraints that allow incorporation of prior assumptions into the MAXENT estimator making it effectively parametric . Here MAXENT will be taken in its original meaning as providing non-parametric estimators . This paper is organized as follows . Section 2 recall the tenets of the Bayesian decision theory and describes the data-generation set-up . Section 3 introduces and motivates the Bayesian estimator and the regularized ML estimator . Section 4 recalls the basic formulas of MAXENT , applies them to the studied set-up , and discusses their symmetry features . Section 5 compares predictions of MAXENT with the regularized ML . We close in the last section with discussing open problems . Appendix A shows how to apply MAXENT to categorical data . Appendix B presents our preliminary results on the affine symmetry of MAXENT estimators , and establishes relations with the minimum entropy principle proposed in ( Good , 1970 ; Christensen , 1985 ; Zhu et al. , 1997 ; Pandey & Dukkipati , 2013 ) . 2 BAYESIAN DECISION THEORY . Consider a random quantityZ with values ( z1 , ... , zn ) and respective probabilities q = ( q1 , ... , qn ) = ( q ( z1 ) , ... , q ( zn ) ) . We look at an i.i.d . sample of length M : D = ( Z1 , ... , ZM ) , m = { mk } nk=1 , M ≡ n∑ k=1 mk , ( 1 ) where Zu ∈ ( z1 , ... , zn ) ( u = 1 , ... , M ) , and mk is the number of appearances of zk in ( 1 ) . This sample will be an instance of our data , e.g . constraints of MAXENT will be determined from it . The conditional probability of data D reads P ( D|q1 , ... , qn ) = P ( m1 , ... , mn|q1 , ... , qn ) = M ! n∏ k=1 qmkk mk ! . ( 2 ) To check the performance of various inference methods , the probabilities q̂ ( D ) = { q̂k ( D ) } nk=1 inferred from ( 1 ) are compared with true probabilities q = { q ( zk ) } nk=1 via the KL-distance K [ q , q̂ ( D ) ] = n∑ k=1 qk ln qk q̂k ( D ) , ( 3 ) where concrete forms of q̂ ( D ) are given below . The choice of distance ( 3 ) is motivated below , where we recall that it implies the global optimality of the standard ( posterior-mean ) Bayesian estimator . Another possible choice of distance is the squared ( symmetric ) Hellinger distance : distH [ q , q̂ ] ≡ 1− ∑n k=1 √ qk q̂k . In our situation , it frequently leads to the same qualitative results as ( 3 ) . How to compare various estimators with each other , and decide on the quality of a given estimator ? Bayesian decision theory comes to answer this question ; see chapter 11 of ( Cox & Hinkley , 1979 ) . The theory assumes that the probabilities of ( z1 , ... , zn ) are generated from a known probability density P ( q1 , ... , qn ) that encapsulates the prior information about the situation . Next it decides on the quality of an estimator q̂ ( D ) via the average distance 〈K〉 = ∫ n∏ k=1 dqk P ( q1 , ... , qn ) K , K = ∑ D P ( D|q ) K [ q , q̂ ( D ) ] . ( 4 ) where K is the average of ( 3 ) over samples ( 1 ) with fixed length M . Sometimes the Bayesian decision theory replaces the distance by the utility , loss etc ( Cox & Hinkley , 1979 ) . Note the difference between the proper Bayesian approach and the Bayesian decision theory ; cf . chapters 10 and 11 in ( Cox & Hinkley , 1979 ) . The former employs the data for moving from the prior ( 5 ) to the posterior ( 7 ) . It averages over the prior , e.g . when calculating the posterior mean . The latter advises on choosing estimators , whose form may or not may not depend on the prior ; see below for examples . The decision theory averages both over the data and over the prior , as seen in ( 4 ) . For the prior density of q = { qk } nk=1 we choose the Dirichlet density ( or a mixture of such densities as seen below ) ( Frigyik et al. , 2010 ; Schafer , 1997 ) : P ( q1 , ... , qn ; α1 , ... , αn ) = Γ [ ∑n k=1 αk ] ∏n k=1 Γ [ αk ] n∏ k=1 qαk−1k δ ( n∑ k=1 qk − 1 ) , ( 5 ) where Γ [ x ] = ∫∞ 0 dy yx−1 e−y is Euler ’ s Γ-function and delta-function δ ( ∑n k=1 qk−1 ) ensures the normalization of probabilities . Parameters αk > 0 determine the prior weight of qk ( Frigyik et al. , 2010 ; Schafer , 1997 ) : 〈qk〉 ≡ ∫ ∞ 0 n∏ l=1 dql qk P ( q1 , ... , qn ; α1 , ... , αn ) = αk A , A ≡ n∑ k=1 αk , ( 6 ) where the integration range goes over the simplex 0 ≤ qk ≤ 1 , ∀k , and ∑n k=1 qk = 1 . Dirichlet density ( 5 ) is unique in holding several desired features of non-informative prior density over unknown probabilities ; see ( Frigyik et al. , 2010 ; Schafer , 1997 ) for reviews . An important feature of density ( 5 ) is that it is conjugate to the multinomial conditional probability ( 2 ) P ( q1 , ... , qn|m1 , ... , mn ) = P ( q1 , ... , qn ; α1 +m1 , ... , αn +mn ) . ( 7 ) Eq . ( 7 ) is convenient when studying i.i.d . samples ( 1 ) of discrete random quantities . Here we assume that the prior density is known exactly [ see however ( 32 ) ] . In practice , such a knowledge need not be available . For example , it may be known that the prior density belongs to the Dirichlet family , but its hyper-parameters { αk } nk=1 are unknown and should be determined from the data , e.g . via empirical Bayes procedures ; see ( Frigyik et al. , 2010 ; Schafer , 1997 ; Claesen & De Moor , 2015 ; Ran & Hu , 2017 ; Bergstra & Bengio , 2012 ) for reviews on hyper-parameter estimation . 3 BAYESIAN AND REGULARIZED MAXIMUM LIKELIHOOD ( ML ) ESTIMATORS . Starting from ( 4 ) , we find the best estimator in terms of the minimal , average KL-distance : min [ 〈K 〉 ] = ∑ D P ( D ) min [ ∫ n∏ k=1 dqk P ( q|D ) K [ q , q̂ ( D ) ] ] , ( 8 ) where the minimization goes over inferred probabilities { q̂ ( D ) } , and where P ( q|D ) is recovered from P ( D|q ) : P ( D ) P ( q|D ) = P ( D|q ) P ( q ) ; cf . ( 1 , 2 ) . The equality in ( 8 ) follows from the fact that if q̂ ( D ) minimizes ∫ ∏n k=1 dqk P ( q|D ) K [ q , q̂ ( D ) ] , then it will minimize each term of the sum for everyD , and thus will minimize the whole sum . Then implementing the constraint ∑n k=1 q̂k ( D ) = 1 via a Lagrange multiplier , we get from ( 8 ) : argmin [ ∫ n∏ k=1 dqk P ( q|D ) K [ q , q̂ ( D ) ] ] = { ∫ n∏ k=1 dqk ql P ( q|D ) } n l=1 . ( 9 ) We got in ( 9 ) the posterior average , because we employed the KL distance K [ q , q̂ ( D ) ] . The optimal estimator will be different upon using another distance , e.g . KL distance K [ q̂ ( D ) , q ] of q̂ ( D ) from q , or the Hellinger distance . Note that in the proper Bayesian approach the posterior mean is simply postulated to be an estimator , since it is just a characteristics of the posterior distribution . In the present Bayesian decision approach the posterior emerges from minimizing a specific ( viz . KL ) distance . If another distance is used , the posterior mean is not anymore optimal . If the prior is a single Dirichlet density ( 5 ) we get from ( 7 , 9 ) for the Bayesian estimator : p ( zk ) = mk + αk M +A . ( 10 ) The average KL-distance ( 4 ) for the estimator ( 10 ) reads from ( 7 , 2 ) ( denoting ψ [ x ] ≡ ddx ln Γ [ x ] ) : 〈K [ q , p ] 〉 = 1 A n∑ k=1 αkψ ( 1 + αk ) − ψ ( 1 +A ) + ln ( M +A ) − Γ [ M + 1 ] Γ [ A ] Γ [ M +A+ 1 ] n∑ k=1 M∑ m=0 Γ [ m+ 1 + αk ] Γ [ M −m+A+ αk ] ln ( m+ αk ) Γ [ αk ] Γ [ A− αk ] Γ [ m+ 1 ] Γ [ M −m+ 1 ] . ( 11 ) If the prior density is given by mixture of Dirichlet densities with weights { πa } La=1 : L∑ a=1 πaP ( q1 , ... , qn ; α [ a ] 1 , ... , α [ a ] n ) , L∑ a=1 πa = 1 , ( 12 ) then instead of ( 6 ) and ( 10 ) we have from ( 9 ) 〈qk〉 = L∑ a=1 πa α [ a ] k A [ a ] , A [ a ] ≡ n∑ k=1 α [ a ] k , ( 13 ) p ( zk ) = ∑L a=1 πa Φ [ a ] mk+α [ a ] k M+A [ a ] ∑L a=1 πa Φ [ a ] , Φ [ a ] ≡ Γ [ A [ a ] ] Γ [ M +A [ a ] ] n∏ k=1 Γ [ mk + α [ a ] k ] Γ [ α [ a ] k ] . ( 14 ) For a mixture prior density , the Bayesian estimator ( 14 ) depends on all numbers { mk ; α [ 1 ] k , ... , α [ L ] k } not just on mk . Below we illustrate that not knowing precisely details of the prior mixture can lead to serious losses when applying Bayesian estimators . It is interesting ( both conceptually and practically ) to have a simple estimator , where the dependence on the prior is reduced to a single parameter . A good candidate is the regularized maximum likelihood ( ML ) estimator ( see ( Hausser & Strimmer , 2009 ) for a review ) : pML ( zk ) ≡ mk + b M + nb = λ mk M + ( 1− λ ) 1 n , λ = M M + nb , b ≥ 0 , 0 < λ < 1 , ( 15 ) where the regularizer b ( or λ ) takes care of the fact that for a finite sample ( 1 ) not all values zk had a chance to appear ( i.e . mk = 0 for them ) . Then ( 15 ) avoids to claim a zero probability due to b > 0 . Eq . ( 15 ) is a shrinkage estimator , where the proper ML estimator mkM is shrunk towards uniform distribution 1n by the shrinkage factor λ . The proper ML estimator pML ( zk ) |b=0 will be shown to be a meaningless estimator for not very long samples ( 1 ) producing results that are worse than { q ( zk ) = 1n } n k=1 . Moreover , for such samples the correct choice of b ( based on the prior information ) is crucial , i.e . ( 15 ) is generally a parametric estimator . The estimator ( 15 ) recovers true probabilities for M →∞ ( Cox & Hinkley , 1979 ) , where n and b are fixed , hence λ→ 1 in ( 15 ) . For the optimal estimator ( 15 ) , the value of b is found by minimizing the average KL-distance ( 4 ) . When the prior is given by a Dirichlet density ( 5 ) , the average KL-distance amounts to ( 11 ) , where we need to replace ln ( M+A ) → ln ( M+nb ) and ln ( m+αk ) → ln ( m+b ) . Now ( 9 , 10 ) imply that for a homogeneous Dirichlet prior , i.e . for ( 5 ) with αk = α , we have bopt = α for the optimal value of b , i.e . the regularized ML estimator coincides with the Bayesian estimator : pML ( zk ) = p ( zk ) . This does not anymore hold for the mixture of Dirichlet prior densities .

Summary: This paper considers the maximum entropy (MAXENT) method for estimating underlying probabilities over a finite alphabet, i.e., the multinomial model. The authors compare MAXENT with the regularized maximum likelihood, that is the Bayesian estimator under the Dirichlet prior with a common hyperparameter, and the Bayesian estimator with a general Dirichlet prior in terms of the Bayes risk, i.e., the KL-divergence from the true distribution to the estimated distribution averaged over the prior. The authors also consider the case where the prior is extended to the mixture of Dirichlet distributions. These comparisons are done numerically with synthetic data.