Maximum Entropy competes with Maximum Likelihood
Authors: Armen Allahverdyan
Venue:  (2021)
Keywords: maximum entropy
Bayesian statistics

Abstract: Maximum entropy (MAXENT) method has a large number of
applications in theoretical and applied machine learning, since it
provides a convenient non-parametric tool for estimating unknown
probabilities.  The method is a major contribution of statistical
physics to probabilistic inference. However, a systematic approach
towards its validity limits is currently missing.  Here we study MAXENT
in a Bayesian decision theory set-up, i.e. assuming that there exists a
well-defined prior Dirichlet density for unknown probabilities, and that
the average Kullback-Leibler (KL) distance can be employed for deciding
on the quality and applicability of various estimators.  These allow to
evaluate the relevance of various MAXENT constraints, check its general
applicability, and compare MAXENT with estimators having various degrees
of dependence on the prior, {\it viz.} the regularized maximum
likelihood (ML) and the Bayesian estimators. We show that MAXENT applies
in sparse data regimes, but needs specific types of prior information. 
In particular, MAXENT can outperform the optimally regularized
ML provided that there are prior rank correlations between the estimated
random quantity and its probabilities. 

Under review as a conference paper at ICLR 2021
MAXIMUM ENTROPY COMPETES WITH MAXIMUM LIKELIHOOD
Anonymous authors Paper under double-bling review
ABSTRACT
Maximum entropy (MAXENT) method has large number of applications in theoretical and applied machine learning, since it provides convenient non- parametric tool for estimating unknown probabilities. The method is major con- tribution of statistical physics to probabilistic inference. However, systematic approach towards its validity limits is currently missing. Here we study MAX- ENT a Bayesian decision theory set-up, i.c. assuming that there exists well- defined prior Dirichlet density for unknown probabilities, and that the average Kullback-Leibler (KL) distance can be employed for deciding on the quality and applicability of various estimators. These allow to evaluate the relevance of var- ious MAXENT constraints, check its general applicability, and compare MAX- ENT with estimators having various degrees of dependence regularized maximum likelihood (MI.) and the Bayesian estimators. We show that MAXENT applies in sparse data regimes, but needs specific types of prior information. In particular, MAXENT can outperform the optimally regularized ML provided that there are prior rank correlations between the estimated random quantity and its probabilities.
the prior, the
INTRODUCTION
The maximum entropy (MAXENT) method was proposed within statistical physics (Jaynes 1957:
Balian 2007 Pressé al. 2013). and later on got a wide range of inter-disciplinary applica- tions in data science, probabilistic inference, biological data modeling etc; e.g. (Erickson & Smith 2013). MAXENT estimates unknown probabilities (that generated data) via maximizing the
Gibbs-Shannon entropy under certain constraints which can be derived from the observed data (Erickson & Smith 2013). MAXENT leads to non-parametric estimators whose form does not depend on the underlying mechanism that generated data (i.c. prior assumptions). Also, MAXENT avoids the zero-probability problem, i.e. when operating on a sparse data so that certain values of the involved random quantity may not appear due to a small, but non-zero still provides a controllable non-zero estimate for this small probability.
MAXENT has has several formal justification (laynes) 1957 Chakrabarti & Chakrabarty 2005 Baez et al. 2011 Van Campenhout & Cover 1981; Topsoee 1979) Shore & Johnson] 1980 Paris & Vencovská 1997). But the following open problems are basic for MAXENT, because their insuffi- cient understanding prevents its valid applications. (i) Which constraints of entropy maximization be extracted from data, which necessarily finite and noisy? (ii) When and how these con- straints can lead to overfitting, where -due to a noisy aata-innollvigg more constraints leads to poorer results? (iii) How predictions of MAXENT compare with those of other estimators, e.g. the (regularized) maximum likelihood?
Here we approach these open problems via tools of Bayesian decision theory (Cox & Hinkley, 1979). We assume that the data is given an i.i.d. sample of finite length M from a random quantity with n outcomes and unknown probabilities that are instanced from a non-informative prior Dirichlet density, calculate average KL-distances between real probabilities and their estimates, decide on the quality of MAXENT under various constraints, and compare it with the (regularized) maximum-likelinn (ML) estimator. Our main results are that MAXENT does apply to sparse data, but does demand specific prior information. We explored two different scenarios of such information. First, the
mixture of such densities. Focusing on the sparse data regime M <

---

Under review as a conference paper at ICLR 2021
unknown probabilities most probably deterministic. Second, there are prior rank correlations between the inferred random quantity and its probabilities. Moreover, in the latter case the non- parametric MAXENT estimator is better in terms of the average KL-distance than the optimally regularized MI (parametric) estimator.
Some of above questions were already studied in literature. (Good 1970 Christensen 1985 Zhu
et al. 1997 Pandey & Dukkipati 2013) applied formal principles of statistics (e.g. the Miniuum Description Length) to the selection of constraints (question (i)). Our approach to studying this question will be direct and unambiguous, since, as shown below, the Bayesian decision theory leads clear criteria for the validity of MAXENT estimators. We can also compare all predictions with the optimal Bayesian estimator. The latter is normally not available in practice due to insufficient knowledge of prior details, but still does provide an important theoretical benchmark. Note that (Tomas 1979, Lebanon & Lafferty| 2002 Kazama & Tsujii 2005 Altun & Smola 2006) Dudik, 2007.] Rau 201 [ampbell]] 1999, Friedlander Gupta (2005) studied soft constraints that allow incorporation of prior assumptions into the MAXENT estimator making it effectively parametric. Here MAXENT will be taken in its original meaning as providing non-parametric estimators.
1979,Lebanon
This paper is organized as follows. Section recall the tenets of the Bayesian decision theory and describes the data-generation set-up. Section introduces and motivates the Bayesian estimator and the regularized ML estimator. Section studied set-up, and discusses their symmetry features. Section compares predictions of MAXENT with the regularized MI. We close in the last section with discussing open problems. Appendix A shows how to apply MAXENT to categorical data. Appendix B presents our preliminary results on the affine symmetry of MAXENT estimators, and establishes relations with the minimum entropy principle proposed in (Good 1970 Christensen
the basic formulas of MAXENT, applies them to the
1985 [985] et al. 1997 Pandey & Dukkipati 2013)
2 BAYESIAN DECISION THEORY
Consider random quantity Z with values (21, 2n) and respective probabilities q (g...... pn) = (q(2))...,,(( We look at an i.idd sample of length M:
n
D (Z1, ..... Zar),
m (m+
M
mk:
where Zu E (z1,..,, -n) (u I... M), and my is the number of appearances of in 11. This sample will an instance of our data, e.g. constraints of MAXENT will be determined from it. The conditional probability of data D reads
P(D .... qn) P(m1, mn m 19....., 9n) M!
4
(2)
my!'
k=
To check the performance of various inference methods, the probabilities ((D) = {âx (D))
inferred from II) are compared with true probabilities q {{(Zz)) via the KL-distance
K[g, ((D))
Σ 9k
9k
(3)
2R (D))
where concrete forms of ((D) given below. The choice of distance is motivated below, where we recall that it implies the global optimality of the standard (posterior-mean) Bayesian estimator. Another possible choice of distance is the squared (symmetric) Hellinger distance: dista (q. @
--En k= Vqh qr. In our situation, it frequently leads to the same qualitative results as 3
How compare various estimators with each other, and decide on the quality of given estimator? Bayesian decision theory comes to answer this question; see chapter 11 of (Cox & Hinkley 1979).
The theory assumes that the probabilities of (21, 2n,) are generated from known probability density P(q,,... ..., 9n) that encapsulates the prior information about the situation. Next it decides on the quality of an estimator (D) via the average distance
K
II dgc P(q..... 9n)
K= Σ P(D)))K(a.(()))
(4)
2

---

Under review as a conference paper at ICLR 2021
where K is the average of (3) over samples with fixed length M. Sometimes the Bayesian deci- sion theory replaces the distance by the utility, loss etc (Cox & Hinkley 1979). Note the difference between the proper Bayesian approach and the Bayesian decision theory; ef. chapters 10 and 11 in (Cox & Hinkley 1979). The former employs the data for moving from the prior (5) to the poste- rior 77. It averages over the prior, e.g. when caleulating the posterior mean. The latter advises on choosing estimators, whose form may or not may not depend on the prior; see below for examples. The decision theory averages both over the data and over the prior, as seen in (4)
For the prior density of {g+ we choose the Dirichlet density (or a mixture of such densities as seen below) (Frigyik et al. 2010 Schafer 1997):
r Ogk II
72
71
P(q Iu On an..., an)
9k
4h
1),
(S)
Mas
00 dy yd e v is Euler's I'-function and delta-function 8 8(Σ 9h 1) ensures the normalization of probabilities. Parameters aj > 0 determine the prior weight of gs (Frigyik et al.) 2010 Schafer [997]:
where [|a] = l
-=
R
(gx) II dq qu P(gi.... On a1, an)
0% A
A
ak:
(6)
where the integration range goes over the simplex 0 S 9 S 1, Ak, and Σ 9k 1. Dirichlet
density (5) is unique in holding several desired features of non-informative prior density over un-
known probabilities; see (Frigyik et al. 2010; Schaaeer] 1997) for reviews. An important feature of density 5 is that it is conjugate to the multinomial conditional probability (2)
P(a....., I......///m mm.m........ m..) = P(q In 01 + +........ On + m,.)
Eq. is convenient when studying i..d. samples ( of discrete random quantities. Here assume that the prior density is known exactly [see however (32)]. In practice, such a knowledge need not be available. For example, it may be known that the prior density belongs to the Dirichlet family, but its syper-parametert {ax } are unknown and should be determined from the data, c.g. via
empirical Bayes procedures: see (Frigyik al., 2010:; [Schafer 1997| Claesen & De Moor 2015
Ran & Hu 2017 Bergstra & Bengio 2012) for reviews on Slyper-praamete estimation.
3 BAYESIAN AND REGULARIZED MAXIMUM LIKELIHOOD (ML) ESTIMATORS
Starting from
we find the best estimator in terms of the minimal, average KL-distance:
[ dg P(llD)K(o.@(D))
min| (K)] Σp(D) min
(8)
1=
where the minimization goes over inferred probabilities {.. and where P(q)D) is recovered
from P(D]g): P(g)) = P(D))) (1 The equality in (8) follows from the fact that
(D) minimizes I dq P(q D) lg. ((D),, then it will minimize each term of the sum for
every D, and thus will minimize the whole sum. Then implementing the constraint En 9h (D) via a Lagrange multiplier, get from (8:
argmin /
=(/i dg qi P(g|D)
II dg. P(g|D) Kla, (()) k=1
II k=1
(9)
I=1
We got in (9) the posterior average, because we employed the KL distance K Klg, (D)). The optimal estimator will be different upon using another distance, e.g. KL distance Klğ(D), gl of g(D) from q. or the Hellinger distance. Note that in the proper Bayesian approach the posterior mean is simply postulated to be an estimator, since it is just characteristics of the posterior distribution. In the present Bayesian decision approach the posterior emerges from minimizing a specific (viz. KL) distance. If another distance used, the posterior mean is not anymore optimal.
If the prior is a single Dirichlet density
we get from for the Bayesian estimator:
m& + 0%
p(zx)
(10)
M +
3

---

Under review as a conference paper at ICLR 2021
The average KL-distance (4) for the estimator (10) reads from
2 (denoting d[x]
d In I(x))
da
(K(.p)) Σ
+ In(M A) app(1 o4) 10 A)
r[M I]M M + A+1
M
Fim + Onl [ m+ + ond In(m + as)
(11)
D(ak] T[A 0zk
D m + 1 m + 1]
If the prior density is given by mixture of Dirichlet densities with weights { Jami
Fa P(p ..... On ald) ..... a!!i)
Σ na 1,
(12)
then instead of (6) and
10
we have from 9
(1x)
=
Ala
k
10
la]
(13)
a=1
El
a==1
Ta
mata M+16
T[4
n
Time [0]] a
p(2))
(14)
Ee= na b10)
T[M + Al1]
T(o [0)]
For mixture prior density, the Bayesian estimator (4) depends on all numbers {mii o .... Ok not just my- Below we illustrate that not knowing precisely details of the prior mixture can lead to serious losses when applying Bayesian estimators.
[1)
It interesting (both conceptually and practically) to have simple estimator, where the depen- dence on the prior is reduced to a single parameter. good candidate is the regularized maximum likelihood (ML) estimator (see (Hausser & Strimmer 2009) for review):
m + PMI. (za) =
mi M + nb M
M
+ (1-
X
b 620, O<λ<1, (15)
M nb
where the regularizer b (or X) takes care of the fact that for finite sample not all values zg had a chance to appear (i.e. my = for them). Then (15) avoids to claim zero probability due to 6 > 0. Eq. (15) is shrinkage estimator, where the proper ML estimator M is shrunk towards uniform distribution 1 by the shrinkage factor A. The proper ML estimator (= will be shown to be meaningless estimator for not very long samples (I) producing results that are worse than {9(=x) a 1 Moreover, for such samples the correct choice of 6 (based on the prior information) is crucial, i.c. (15) is generally a parametric estimator. The estimator (I5) recovers true probabilities for M + oo (Cox & Hinkley 1979), where and are fixed, hence
15)
For the optimal estimator (15). the value of is found by minimizing the average KL-distance When the prior is given by Dirichlet density (5). the average KL-distance amounts [T]), where need to replace In(M + A) + In(M+ and In( + ln(m+b). Now (9] 10) imply that for homogeneous Dirichlet prior, i.e. for (5) with or = a, we have dopt = a the optimal value of b, i.c. the regularized ML estimator coincides with the Bayesian estimator: PML(2+) p(z)). This does not anymore hold for the mixture of Dirichlet prior densities.
THE MAXIMUM ENTROPY (MAXENT) METHOD
MAXENT infers probabilities from maximizing the Boltzmann- Shannon entropy
Sig Σ ((xx) In ((x),,
(16)
under constraints taken from the sample II. The rationale of maximizing (16) is that larger
means a smaller bias (or information) according to several axiomatic schemes (Jaynes 1957
Chakrabarti & Chakrabarty 2005 Baez. et al. 2011 Van Campenhout & Cover 1981
Topspe
4

---

Under review as a conference paper at ICLR 2021
1979
Shore & Johnson 1980, Paris & Vencovská 1997 Balian 2007 Pressé et al. 2013 Note
that physical applications of MAXENT operate with constraints that are known precisely. c.g. the mean energy constraint is deduced from the corresponding conservation law (Janess 1957 Balian
2007 Pressé et al. 2013). Such situations are rare in statistics and machine learning. Hence we need understand which constraints are to be taken from the noisy data.
First we can apply no constraint and maximize the entropy:
q (z+) = 1/n..
(17)
The calculation the average distance is straightforward from 4 both for a single Dirichlet prior and mixture of such priors. We examplify the single Dirichlet case 5):
(Klg, T
Σ tau In qx) + lnn
1 A
Σ
an (1+ as)
- + A) + lun.
(18)
Now (K q,
glolj) plays an important role: once 117 completely data-independent and simply
reproduced the prior expectation on the unbiased probabilities, estimators that provide the average KL-distance larger than (18) are meaningless; see below for examples.
Next, we employ the empiric mean of
I as constraint for the expected value of Z:
M
f1
1
Zu
Σ
4k=k-
(19)
M
M
R=
R=
Maximizing (6) under constraint (19) via the Lagrange method leads to the famous Gibbs formula
Jaynes 1957
Balian 2007 Pressé et al. 2013):
n (zk)
(20)
Dz
where the Lagrange multiplier 3 is found from 20) to real data. We order the values of Z as 21 < 2n, and note a specific feature of (20) depending the sign of B, either get
19
Appendix presents example of applying
(21) S ... I (zx) or (2) ... 2 (t. (zn).
(21)
One can try to acquire further information from sample ① by looking at the second empiric moment:
M
1
1/2
z
m
Th kk
(22)
M
M
k=
1=1
Now we maximize
16 under two constraints
19
and
22
91
g
(23)
BERt P257
E1=1
where Lagrange multipliers 31 and 83 are found from solving both 19 and 22 Eqs. 20
23
make obvious how to involve other (fractional) moments. The maximizations of (T) lead to unique results, because (16) is concave function of {px 1p while the moment constraints are linear.
Let the values (21...., 2n) of Z be subject to affine transformation
E = F(zx), F(z) = g= h, 1
(24)
Hence as a result of transformation 24 : μ4l F4 gH1 + h and μ1 + P2 9 Uv h2 + 2ghp1s
(19 22). These relations show that 24 leaves the inferred probabilities 20 23 invariant,
because the resulting set of equations for the unknowns in (20 23 are identical for both the original
(21,-.., 2n.) and transformed values (24) Likewise, involving first p moments / My produces
affine-invariant probabilities. Note that involving only 22) [without involving (19 will lead to
the invariance of the probabilities with respect to a limited aifine-symmeery, where 0 in 24). Another example of limited affine symmetry is involving the fractional moment Σ 9e / (Tor
zk 0 and instead of [19 22 Then the probabilities (1/) - will stay intact only
5

---

Under review as a conference paper at ICLR 2021
under h = and g > in 24 Note in this context that the ML estimator 15 is invariant with respect (24) with an arbitrary bijective F, which keeps the values of žg different.
The symmetry features of various estimators clearly important, though lytical results that would relate them to the estimation quality quantified by (4). But we noted from numerical comparison of MAXENT estimators based on various constraints, that estimators with the
far have ana-
largest affine symmetry- i.e. (24) with arbitrary and t-tenn to be better in terms of the average KL-distance (4). Intuitively, higher (affine) symmetry should be related to higher susceptibility with respect to noises; see Appendix B for further results.
5 NUMERICAL RESULTS
5.1 A SINGLE DIRICHLET DENSITY
Recall that maximization of entropy (16) can be applied if there is no prior information that distin- guishes one probability from another. If such information is present, MAXENT is generalized to the minimum relative entropy method (Shore Johnson| 1980). We shall not study this generaliza- tion here. Hence to ensure applicability of MAXENT, we always choose prior densities such that (qx) :: i.e. all values are equally likely be generated, on average. As seen from (6) for single prior Dirichlet density (5) condition (gx) = 1 implies:
n
OEk a2
k: 1, ,,,,., n1
(25)
10 that under (55) the Bayesian and the regularized ML coincide. I.c. we conclude that the regularized ML is better estimator than MAXENT (under any constraint).
Now recall from 15
Though (gx) = - does not depend on a, the most probable values The of d do depend on the magnitude of a. Finding The from (5 25 amounts to maximizing C(g) = (a 1) Σi= In qh +
E gk. where the Lagrange multiplier ensures Σn 9x = 1. For a > 1, L(q) is a concave function of q, and its global maximum is found after differentiating it. Hence ğN holds
T = 1/n for a a> 1, k= 1,.,..,
(26)
For a < 1, L(q) convex function, it does not have local maxima with qs > (A: = I,... n). Its maxima are located at points, where maxima of C(g) are at those points, where possibly large number of Ox are zero:
0 for certain k. Repeating this argument, we see that the
/ 0 or Th 1, for <<<,
A = 1,..,,
(27)
which means deterministic probabilitics. Eq. 27 is consistent with (gx) = 1/n, because there are equivalent most probable values.
Let us start with the regime a > 1; cf. (6) Table I compares predictions of 15) with those of MAXENT solutions (20) and (23 for the Dirichlet prior (5) holding (25) with 2. It is seen that MAXENT is meaningless, because the trivial estimator (17) provides a smaller average KL-
distance; cf. 188 For the Bayesian estimator even M 1 leads meaningfull prediction; e.g.
for parameters of Table ! we have: Bayes)
0.224 < 0.225.
The above conclusion holds more generally (as we checked numerically): for the homogeneous Dirichlet prior (25) with a > 1, MAXENT estimators (20) 19) and (23 19 22 are meaningless at least in the sparse data regime M n. This puts a serious limitation on the validity of MAXENT.
The situation changes for sufficiently small values of a in the regime (27): see Table for a for 0.1.
Here the MAXENT estimators are meaningful provided that the sample length M sufficiently
large (but still in the sparse data regime M < n)): (20 19h is meaningful for M 2 9 (M < n 60).
25; see Table 2 Though predictions of MAXENT are still far from those of the Bayesian estimator, we should recall that the latter estimator is parametric, i.c. it depends on the prior (via the parameter a:) in contrast to MAXENT estimators.
while the estimator [3 19 22
meaningful for
Table 2 demonstrates the overfitting phenomenon: for 9 < M < 15 the MAXENT estimator (20) 19 is meaningful, but adding the second constraint makes the MAXENT estimator 23 19 not meaningful. The situation is worsened since (22) is again estimated from the noisy data and gathers more noise than information. This overfitting disappears for larger values of M, i.. M 2 25, as Table demonstrates.. Now adding the second constraint 222 is beneficial.

---

Under review as a conference paper at ICLR 2021
Table 1: For 60 and 2 k (k L, ..., n)) we show the average KL-distance (4 for vari- ous estimators. The full affine symmetry (24p holds for all shown probabilities. M is the length of sample ① The initial prior Dirichlet density (5) holds (25) with dg
Eq. (88) equals
(Klg. [[ = 0.225, Le. values of the average KL-distance larger than 0.225 are meaningless. K Bayes) is the averaged Kl--distance for the Bayes estimator (10) that for this case coincides with the optimally regularized ML estimator. { and (K 142) are defined (resp.) via (0 19 and (23
22 The averages are found numerically (applies to all Tables): first we generate TO instances of
19k 19k]==1 from the Dirichlet density, and then for each instance we generate 10 samples (1) Such parameters lead 3-digit precision, as reported.
M
K Bayes)
(K)
(K 1422
35
0.177
0.236
0.247
25
0.188
0.240
0.260
15
0,202
0.259
0.301
Table The same as Table but for 0ş = a = 0.1 in (5 Eq. 18
gives (K(g, g
= 1.798,
Le. values of the average KL-ddittance larger than 1.798 meaningless.
M
Bayes)
(K))
(K142)
0.233
1.756
1.685
45
0.276
1.700
1.643
35
0.338
1.723
1.680
25
0.428
1.753
1.717
15
0.606
1.770
2.164
0.730
1.762
4.946
9
0.818
1.774
11.63
7
0.916
1.848
32,24
Table The same as Table I but the initial prior density a Dirichlet mixture given by 12 30 with ap 0.3 and = 1.1. The average Kl--distance (K [g. q[o]]) for the trivial estimator 17 equals
0.212, i.c. values of the average KL-distance larger than 0.212 are meaningless; cf. 18)
Bayes)
and Bayye refer to [14 and (32). respectively. (K ML) \ and (K ML) refer to regularized ML estimator (15) under
I and the optimal value of b found from numerically minimizing
(K ML.). The optimal value baps of b changes from 2.46 for M = 35 to 2.65 for M = 1. We also
report the value of (L. with sensible value of to confirm that if is not chosen properly. then the corresponding (regularized) MI. estimator 115) is meaningless. (K,) is defined via 20
K 142 is not shown, since (K > (K,) for 35 2 M 2 1. We do not show (K,) M=1, since it larger than the average KL-distance for all other estimators.
M
[K Bayes)
Bayes)
(KML)b-b..
(ML))=1
(K))
35
0.014
0.206
0.180
0.204
0.048
25
0.015
0.207
0.188
0.210
0.053
15
0.017
0.209
0.197
0.214
0.065
0.022
0.209
0.201
0.215
0.077
0.035
0.209
0.205
0.215
0.105
0.052
0.210
0.207
0.214
0.141
0.083
0.210
0.209
0.213
0.268
0.150
0.211
0.211
0.212

---

Under review as a conference paper at ICLR 2021
Table 4: The same as in Table 3 but for different values of M. Here (K122) refers to MAXENT
estimator (23) with constraints (19 22). The Bayesian estimator found from (14 30 For this
range of sufficiently large M the MAXENT estimator 23) performs better than 23
(K122) <
(K1) < (KML))- b Bops
M
K Bayes)
K MML)b=bone
(ML))==
(K1)
(K122)
45
0.015
0.172
0.196
0.045
0.042
65
0.014
0.157
0.180
0.042
0.035
85
0.014
0.145
0.164
0.040
0.031
241
0.013
0.087
0.091
0.038
0.024
5.2 MIXTURE OF DIRICILET DENSITIES
For modeling more complex types of prior information about the unknown probabilities {qx } k==1" we shall assume that the prior density is mixture of two Dirichlet densities; see (12). Relations (n:) = (A: 1, ..... n)) will be still kept, since they are necessary for applying MAXENT. Now we assume that that there are (prior) conditional rank correlation between the values (21, zn) of
Z Z-ordered as (2) < < Zn an)-and and its probabilities (g1, .... 9n). For one component of the mixture, the probabilities (q..... 9n) prefer to be ordered as in (41 < ... < qn). For another component they tend to be ordered in the opposite way (gi ...
qn This type of prior knowledge can be modeled
via a mixture 12) of two Dirichlet priors with L -= 2, 1 = 22 2. and
[I]
< 0
[1]
an
[2]
(28)
[1
[1]
0i
b
for any k, 1,...,n,
(29)
A
Al2
where 29 ensures the needed (gx) = 1. as seen from
13 simple case that leads to 28 is
L = 2,
T1 n2
[1] ag + E(k 1), Cg
[2]
ag + E k),
k k= 1,
(30)
where Al'
2 ag + en(n-1), Recall that for a mixture of Dirichlet densities the Bayes 14 and the optimally regularized ML estimator (15) are different.
estimator
For numerical illustration we choose {zx - Prior probability densities generated via (30)
will be employed with <... < Zn Now Tables 3
and 4 show that for M > 5 the MAXENT
estimator 20
19p is clearly better than the optimally regularized ML estimator (5):
(K)) < (LL))=bopp
(31)
where the optimal value of b is found from minimizing the averaged KL-distance M 2 7, we see that (K₂) is closer to the optimal (K Bayes) than to (K ML))b=bops= Note that such threshold values for M do depend on the assumed prior density and on n.
Moreover, for
For M 30 the performance of the optimally regularized ML estimator (15) (for a fixed b 1) will be better than MAXENT with any finite number of constraints, since the regularized MI. converges to the true probabilities for M Table MAXENT with constraints (19) 199
ao (Cox Hinkley
1979), while MAXENT does not. But as
+ 22) still performs better than the optimally
regularized ML even for M as large as 241 (for n
60).
Table 3 shows that MAXENT with two constraints 19 22 performs worse than the method under the single constraint 19 although the affine invariance (24] of probabilities holds. This overfitting
situation changes for larger values of M. i.c. M 45, as Table demonstrates.
To stress the relevance of rank correlations, we note that the advantage (3I) of MAXENT closely relates to the agreement between (28) and the ordering (21
2,) of Z. If the vector (21 < < 2n) is randomly permuted, and employed for values of Z, predictions of MAXENT become meaningless even for rather large values of M >
Recall that both the Bayesian 14 and the regularized ML estimator (15) are parametric, iec the very their form depends on the prior, which is frequently not available in practice. Hence need
8

---

Under review as a conference paper at ICLR 2021
to understand how strong is dependence. Let us assume that one has to employ Bayesian estimator
without knowing the full form of the equgl- mixture (30). Instead one knows the average
values of ag 2 [ao + e(k: 1)] + [Ao + + k)] from 30 preseribes them to single Dirichlet
prior 5 25 and builds up from (TO) an estimator
m& + 0o c(n - 11/2
((*))
(32)
M now + en(n 1)/2"
The performance of this perturbed Bayesian estimator deteriorates and gets worse than that of the
MAXENT solution: (K1) 1) Bayes) see Table 3 Likewise, the choice of b in the regularized ML
estimator (15) is important. If just some reasonable value is chosen instead of the optimal one e.g. b 1 instead of b ~ 2.5 in Table 3 then the ML estimator can turn meaningless; see Table 3 for M < 15. In this context, we emphasize that the Bayes estimator and the optimally regularized ML estimator are never meaningless even for M = 1. For parameters of Table 3 the MAXENT estimator 20 becomes meaningless for M S 3.
SUMMARY AND DISCUSSION
The maximum entropy (MAXENT) method provides non-parametric estimators for inferring un- known probabilities (Erickson & Smith 2013). MAXENT is widely applied both in statistical physics and probabilistic inference. However, its physical applications are mostly data-free and are based additional principles (e.g. conservation laws (Balian 2007 Pressé et al. 2013 that are normally absent in statistics and machine learning. Hence we needed a systematic approach towards understanding the validity limits of MAXENT an inference tool.
Here we presented a Bayesian decision theory approach that allows to determine on whether MAX- ENT is applicable all, i.e. whether it is better than random guess. It also allows to compare different estimators with each other (e.g. to compare MAXENT with the regularized maximum likelihood), and study the relevance various constraints employed in MAXENT.
Our results are summarized as follows. MAXENT does apply to sparse data, but demands specific prior information. Here sparse means M n, i.. the sample length M is smaller than the number of probabilities n to be inferred. We explored two different scenarios of such prior information. First, the unknown probabilities generated by homogeneous Dirichlet density (25) are most prob- ably deterministic. Second, there are prior rank correlations between the random quantity and its probabilities. This seems to be the simplest prior information that makes MAXENT applicable and superior over the optimally regularized maximum-likelibee describing several phenomena that are relevant for applying and understanding estimators: overfit- ting (i.e. adding more noisy constraints leads to poorer inference), instability of optimal Bayesian parametric estimators with respect to variation of prior details, inapplicability of non non-parametric MAXENT estimators very short samples etc.
estimator. Our approach is capable of
Several important problems were uncovered by this study and should be addressed in future. First of all, this concerns the applicability of MAXENT to categorical data, where the values (21,..., knp) of the random variable Z in sample (1) are not numerical, but instead refer to certain distinguishable categories. The major difference between maximum likelihood and MAXENT estimator is that the former freely applies to categorical data. In contrast, MAXENT does depend on the concrete nu- merical implementation (i.e. encoding) of data, though this dependence somewhat weakened by the affine symmetry (24). Thus an open problem demands considering various encoding schemes in view of their applicability to MAXENT estimators. (In this paper we in fact assumed the simplest encoding via natural numbers; see Tables.) Appendix A reports preliminary results in this direc- tion along with real data example. The second open problem relates to the influence of affine symmetries on the performance of various MAXENT estimators. We observed numerically that the constraints which produce affine-invarian probabilities produce better estimators; see after (4). Preliminary results along this direction are given in Appendix B where we also show relations of our results with the minimum entropy principle proposed in (Good 1970; Christensen
1985 Zhu
et al. 1997 Pandey & Dukkipati 2013) for contraint selection.

---

Under review as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
REFERENCES
Yasemin Altun and Alex Smola. Unifying divergence minimization and statistical inference via convex duality. International Conference on Computational Learning Theory, pp. 139-153. Springer, 2006.
John C Baez, Tobias Fritz, and Tom Leinster. A characterization of entropy in terms of information loss. Entropy, (3(11):1455-1955 2011.
Roger Balian. From Microphysies to Macrophysics: Methods and Applications of Statistical Physics. Volumes II. Springer Science & Business Media, 2007.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1):281-305 2012.
L. Lome Campbell. Minimum cross-entropy estimation with inaccurate side information. 1EEE Transactions on Information Theory, 45(7):2650-2652, 1999.
CG Chakrabarti and Indranil Chakrabarty, Shannon entropy: axiomatic characterization and appli- cation. International Journal of Mathematics and Mathematical Sciences, 2005, 2005.
Ronald Christensen. Entropy minimax multivariate statistical modeling-i: Theory. International
Journal Of General System, 11(3):231-277, 1985.
Mare Claesen and Bart De Moor. Hyperparameter search in machine learning. arXiv preprint arXiv:1502.02117, 2015.
David Roxbee Cox and David Victor Hinkley. Theoretical statisties. CRC Press, 1979.
Miroslav Dudik. Maximum entropy density estimation and modeling geographic distributions of species. phd dissertation presented to the princeton university, 2007.
Gary Erickson and C Ray Smith. Maximamm-Entropy and Bayesian Methods in Science and Engi- neering: Volume 2: Applications, volume 31. Springer Science & Business Media, 2013.
Michael P Friedlander and Maya R Gupta. On minimizing distortion and relative entropy. JEEE Transactions Information Theory, 52(1):238-22 2005.
Bela A Frigyik, Amol Kapila, and Maya R Gupta. Introduction the dirichlet distribution and related processes. Department of Electrical Engineering, University of Washignton, UWEETR- 2010-0006, (0006):1-2) 2010.
Andrew Gelman, John Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald Rubin. Bayesian data analysis. CRC press, 2013.
IJ Good. Some statistical methods in machine intelligence research. Mathematical Biosciences, 6: 185-208, 1970.
Jean Hausser and Korbinian Strimmer. Entropy inference and the james-stein estimator, with ap- plication to nonlinear gene association networks. Journal of Machine Learning Research, 10(7), 2009.
Edwin Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Junichi Kazama and Junichi Tsujii. Maximum entropy models with inequality constraints: A case study on text categorization. Machine Learning, 60(1-3):159-19) 2005.
Guy Lebanon and John D Lafferty. Boosting and maximum likelihood for exponential models. In Advances in neural information processing systems, pp. 447-454, 2002.
Gaurav Pandey and Ambedkar Dukkipati. Minimum description length principle for maximum entropy model selection. In 2013 IEEE International Symposium on Information Theory, pp- 1521-1525. IEEE, 2013.
10

---

Under review as a conference paper at ICLR 2021
Jeff Paris and Alena Vencovská. In defense of the maximum entropy inference process. International Journal of approximate reasoning, 17(1):77-103, 1997.
Steve Pressé, Kingshuk Ghosh, Julian Lee, and Ken A Dill. Principles of maximum entropy and maximum caliber in statistical physics. Reviews of Modern Physics, 85(3):1115, 2013.
Zhi- Yong Ran and Bao-Gang Hu. Parameter identifiability in statistical machine learning: a review. Neural Computation, 29(5)11111-1033 2017.
Jochen Rau. Inferring the gibbs state of small quantum system. Physical Review A. 84(1)012201, 2011.
Joseph L Schafer. Analysis of incomplete multivariate data. CRC press, 1997.
John Shore and Rodney Johnson. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Transactions on information theory, 26(1):26-37, 1980.
Marlin U Thomas. A generalized maximum entropy principle. Operations Research, 27(6): 1888- 1196, 1979.
Flemming Topsøe. Information-theorer optimization techniques. Kybernetika, 15(1):8-27, 1979.
Jan Van Campenhout and T Cover. Maximum entropy and conditional probability. IEEE Transac- fions on Information Theory, 27(4):483 1981.
Song Chun Zhu, Ying Nian Wu, and David Mumford. Minimax entropy principle and its application to texture modeling. Neural computation, 988::6277-160, 1997.
A APPENDIX: MAXENT APPLIES TO CATEGORICAL DATA
The MAXENT method can be applied to any multinomial data T.. provided that numeric values (Z1,..., 2n) of the random quantity Z are given. MAXENT estimators depend on (31, 2n) modulo the affine symmetry (24). This creates a problem in applying MAXENT to categorical data, since for MAXENT one now needs specific encoding of categorical into a numeric representation (Z1,..,, 2n.) of categories. Recall that there is degree of arbitrariness in choosing the regularizer in maximum likelihood (ML) estimator, or in choosing prior parameters for Bayesian inference. Here the arbitrariness lies in different encodings. In practice, the proper encoding of categories arises in any problem dealing with categorical data. categories are ordinal (e.g. military ranks, education levels), then one ethnicity, preference, disease) there is no such ordering.
21 22 ... 2n encoding. However, for nominal categories (e.g.
Let us illustrate the MAXENT method with the simple data from pre-election presidential polling conducted in 1988, where out of M = 1447 voters m, = 727 preferred Bush, m2 = 583 preferred Dukakis, and m3 137 preferred other candidates had preference. The data together with its Bayesian analysis is taken from (Gelman et al., 2013).
Here our random variable Z is voter's preference with three outcomes (21, 22, 22) ('Bush', 'Dukakis', 'Other') and unknowns (gi, 4a), which just represent the fractions of the popula- estimate Q1 - 92. i.c. whether Bush has more support tion with each preference. The goal here noninformative prior distribution for (41 Q2, g3) with than Dukakis. One can assume the Dirichlet
parameters a1 02 0g 1, compute the posterior means of g and 92 (タ1, 92) and take the difference (Gelman et al. 2013). The results show that Bush has more support: Q1 > 42.
Since the data purely categorical, we shall apply the frequency encoding for MAXENT: each category is represented with its frequency in the data set, e.g. in this example (2), 22, 2g) (0.502, 0.403, 0.095). Now empiric mean is equal to 0.42 and the maximizing solutions of (6) with Σ 1 2 = 0.42 are (21), (22),, ql (2x)) = (0.535, 0.36, 0.105). Thus, also the MAXENT result shows more support for Bush.
To see if the prediction of MAXENT is reliable (on average) here, the same Bayesian decision model for these samples is up, where first sample of (q), T2, q3) is drawn from the Dirichlet distribution
11

---

Under review as a conference paper at ICLR 2021
with 01 02 0g 1, and then using this sample category probabilities, categorical data sets of size M are generated with categories replaced by its frequency encodings. The process is repeated and the average (K1) from (4) is computed via generating 10 instances of {ęx k==1
and 10 categorical samples. For the present case O11 02 = 03 = 1 and n = 3, we have
(Klg, I 0.265 from (8
Now for M > 15 we get that (K1) (Klg, ( i.c. the MAXENT solution is reliable. For example, at M = 17 we have (K Bayes) = 0.046 < (K1) = 0.116 < (K(g, q(1]) = 0.265, where (K Bayes) refers to the Bayesian (posterior mean) estimator 10p. Already for M = 47 predictions of
MAXENT are close to those of the optimal Bayesian estimator (K Bayes) = 0.019 < (K1) = 0.034.
For the actual sample size M = 1447, we get even closer results (K Bayes) = 0.00087 < (K1)
0.014.
To summarize the present real categorical data example, we saw that the frequency encoding of the categorical variable allows to apply MAXENT. The MAXENT estimator (19) agrees with Bayesian estimator, and is going be reliable already for modest sample sizes M > 15. For sufficiently
large M the average KL distance of the MAXENT estimator gets close to that of the (optimal) Bayes estimator.
B APPENDIX: AFFINE SYMMETRY AND THE MINIMUM ENTROPY PRINCIPLE
Above we focused on MAXENT estimators 20 19 (the first empiric moment fixed) or 23 19
22) (the first and second empiric moments are fixed). As discussed around (24). both (19) and 22
lead to affine-invariant probabilities. We studied several alternative constraints that do not have
full affine symmetry, i.e. this symmetry is partial and relates to restriction on the parameters in 24
An example of this is constraining the square-root (fractional) moment [cf. 20
B1/2VEE
(2x)
(33)
E1=1 E
M
as 22 = M
Zu,
(34)
where B1/2 is determined from 334 and where we assumed 2 > 0. For estimator (33 the sym- metry (24) is kept under g > 0 and 0. We denote the corresponding average KL distances by (K 1/2)-
Let us now compare two different MAXENT estimators each one employing its own constraint; e.g- we compare (20) with (33). We saw from extensive numeric simulations that whenever these con- straints have different degrees the affine symmetry, then the estimator having the largest symmetry provides a smaller average KL. distance. A particular example of this general relation is:
(i) (K 1//2),
(35)
which was verified on parameters of Tables
Recall that Refs. (Good 1970, Christensen 1985 Zhu et al. 1997) proposed the minimum entropy principle: when comparing two possible contraints to be employed in the maximum entropy method, then it is preferable to use the one that provides the smaller (maximized) entropy. The heuristic motivation of the principle is that it avoids overfitting by not insisting too much the entropy maximization. This principle was motivated via the minimum description length in (Pandey & Dukkipati 2013).
We ask whether in cases similar to (35) we can compare the average entropics, i.c. for 335 we compare (zx))) II) and (1)g (2x }), where the averages are defined as in (4). In all cases we were able to check, relations similar to (35) are accompanied by the result that the constraint which
provide a smaller average KI. distance (i.e. better costraint) also has a smaller average entropy, e.g.
(S1g)1/2( (sx))
(36)
12

---

Under review as a conference paper at ICLR 2021
The theoretical origin of this relation between the average KL distance and the average (maximized)
entropy is not yet clear. Here a concrete numerical example that illustrates (35 36). For parameters
of Table we noted for M = 55: (K1) = 1.756 < (K 1/2) 1.758 and (S 1 2k = 4.006 <
(S]g 2))) 4.008.
13


Reviews:
[{"review_id": "uJOYR4OilJH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors discussed the maximum entropy method of obtaining estimators for discrete probability distribution. They also gave an exposition regarding different moment type constraints and how they behave under affine data transformation. They authors then performed numerical experiments comparing the maximum entropy method under different constraints and the regularized maximum likelihood estimator.\n\nI find that overall the writing is fine but some parts were quite difficult to understand. For example, I found it hard to follow the discussions concerning equations (20) to (24). What do you mean by fixing the second moment? Are you setting (23) to a constant?\n\nHowever I think the main problem of this paper is the use of average KL to measure performance given in (5). The use of (5) as a lost function does not seem to be correct. True $q$ is a fixed quantity and you cannot directly put a prior on true parameter values. It is easy to be confused when you are talking about Bayesian procedures and true parameter values. So let me try to break this down. Let us assume that the data is generated from some unknown true distribution $Z_{i_1},\\dotsc,Z_{i_M}\\sim P(Z=z_k)=q_k^0$ for $k=1,\\dotsc,n$, where $q_k^0$ is unknown. Since we do not know the true distribution, we come up with a model for the observed data $Z_{i_1},\\dotsc,Z_{i_M}$, say $Z_{i_1},\\dotsc,Z_{i_M}\\sim P(Z=z_k)=q_k$. So the $q_k$'s here are the parameters for our model which we need to estimate. Let us do the Bayesian way by putting a prior on the $q_k$'s, for example $(q_1,\\dotsc,q_n)\\sim\\mathrm{Dirichlet}(\\alpha_1,\\dotsc,\\alpha_n)$. For there we can get the posterior $\\Pi(q_1,\\dotsc,q_n | Z_{i_1},\\dotsc,Z_{i_M})$ by Bayes theorem and the posterior mean $\\hat{q}_k=\\mathrm{E}(q_k|Z_{i_1},\\dotsc,Z_{i_M})$. So the $hat{q}_k$ is supposed to be the estimator for the unknown $q_k^0$ and we can quantify their distance using KL by\n$$K(q^0,\\widehat{q})=\\sum_{k=1}^nq_k^0\\log\\frac{q_k^0}{\\widehat{q}_k}.$$\n\nIn the paper, the authors endowed a Dirictlet prior on the $q_k^0$'s, and this is not correct. In Bayesian statistics, you endow a prior on the model of the truth and not directly on the truth. From here you can talk about things like how does my estimator performed across all different kinds of truth, i.e., \n$$\n\\sup_{q^0\\in\\mathcal{S}}K(q^0,\\widehat{q})\n$$\nwhere $\\mathcal{S}$ is the simplex in $\\mathrm{R}^n$.\n\nTypos:\n1. Page 1, 2nd line from below, main results are that...\n\n2. The line after (15), do you mean $\\alpha_k^{[1]},\\dotsc,\\alpha_k^{[L]}$? Then the line after this,\nBelow we show some concrete...\n\n3. The 3rd line after (24), (17) is a concave function in $\\{q_k\\}_{k=1}^n$\n\n4. In (7), the range of integration is over the simplex $\\{(q_1,\\dotsc,q_n):\\sum_{k=1}^np_k=1,0\\leq p_k\\leq1,\\forall k\\}$.", "extra_scores": {}, "extra_text": {}}, {"review_id": "e_6Ggquqt04", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper investigates maximum entropy (MaxEnt) inference and compares it to a Bayesian estimator and regularized maximum likelihood for finite models. To assess the accuracy of the different estimators, the authors use the average KL-divergence between the ground truth and the estimator, where the average is computed over all datasets of a given size $M$ and all probabilistic models of size $n$ (generated by some prior, either a single Dirichlet or a mixture of Dirichlets). Using numerical experiments, the authors find that the performance of MaxEnt deteriorates for sparse data generated from uniform models. However, by exploiting knowledge about the order of probabilities, MaxEnt can outperform regularized maximum likelihood. \n\n# Assessment\n\nI have to admit that I had different expectations based on the title and found the paper of limited interest. After reading the paper, I'm a bit puzzled as to what I learned and how much the findings can be generalized. Therefore, I would rather reject the paper in its current version.\n\nLet me try to detail some of the problems that I have with the paper:\n\n* The whole setting seems arbitrary to me. The author call their setup a \"Bayesian approach\" because the correct prior is used to assess the quality of an estimator (when computing $\\langle \\overline K\\rangle$). In my understanding, a Bayesian approach would try to make reasonable assumptions about the likelihood and prior, integrate out any uninteresting parameters, and look at the resulting posterior. \n\n* Typically, MaxEnt is used to incorporate expectations that are assumed to be known exactly. The paper seems to use MaxEnt as a blackbox method in order to circumvent the need of defining a prior. To apply MaxEnt, the data are transformed to moments (by computing sample means), which are then used to constrain the model. It is clear that for small $M$ the estimated expectations obtained by computing sample means are not very reliable and therefore MaxEnt (as used by the authors) performs purely on sparse data. \n\n* The numerical experiments are run for very specific choices of $n$, $\\alpha_k$, $z_k$, etc. There is no motivation for these choices. It remains unclear to me if the results generalize to other choices. \n\n* As the authors acknowledge themselves the approach is limited to finite models with $n$ states. What about continuous or categorical variables?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bwGgDGyRX0r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths: \n1.\tThe paper is basically clear. The claims and method are correct.\n2.\tThis paper proposes a novel MAXENT method to compete with maximum likelihood in sparse data. The topic of the paper is highly relevant to the ICLR community. The idea is original, and the motivation is strong in theoretical analysis. The theoretical analysis is presented in a fine manner. The numerical experiments show the validity of the proposed method.\n3.    By assuming a well-defined prior Dirichlet density for unknown probabilities, it employs the average KL distance in evaluating the relevance of various MAXENT constraints, checking its general applicability, and comparing MAXENT with estimators having various degrees of dependence on the prior, the regularized ML and the Bayesian estimators.\n\nWeakness: \n1.\tI’m confused about the application scenarios of the proposed method. The authors did not show any application value, though they claimed that MAXENT has a large number of applications in applied machine learning. The proposed method is not easy to understand and implement. The author should at least add one real data application.\n2.\tDifferences between the proposed method and the existing methods (Good, 1970; Zhu et al,. 1997; Pandey & Dukkipati, 2013) should be stressed more. The differences can be presented theoretically and numerically.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aVT1_QFNOEr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the maximum entropy (MAXENT) method for estimating underlying probabilities over a finite alphabet, i.e., the multinomial model. The authors compare MAXENT with the regularized maximum likelihood, that is the Bayesian estimator under the Dirichlet prior with a common hyperparameter, and the Bayesian estimator with a general Dirichlet prior in terms of the Bayes risk, i.e., the KL-divergence from the true distribution to the estimated distribution averaged over the prior. The authors also consider the case where the prior is extended to the mixture of Dirichlet distributions. These comparisons are done numerically with synthetic data.\n\nAlthough the practical performance of MAXENT is of interest, the paper provides little novel knowledge about it.\n\n- The numerical experiments are at a too small scale. Without theoretical results, the experiments are required to be more comprehensive. The reported experiments are small scale both in terms of the alphabet size n and the data sequence length M.\n\n- Isn’t it possible to include any discussion on hyperparameter estimation, for example, empirical Bayes since the hyperparameter estimation of the Dirichlet prior has long been studied?\n\n- p.7, last paragraph: I wonder why the random perturbation of the ordering of Z devastates MAXENT since the hyperparametes \\alpha can also be permutated accordingly.\n\nMinor:\nRight after eq. (15): \\alpha_k^{[1]},..., \\alpha_k^{[L]} (The last one should be \\alpha_k^{[L]}.)\np.7, l.12 from the bottom: be be --> be\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As one of the reviewers concisely summarized: This paper investigates maximum entropy (MaxEnt) inference and compares it to a Bayesian estimator and regularized maximum likelihood for finite models. 

Two reviewers specifically question whether they have learned anything new after reading. This combined with various other drawbacks described during the review phase led to strong agreement among the reviewers about a variety of deficiencies in this paper. One reviewer initially gave a relatively high score but has since revised his/her opinion in light of the other reviews and discussion. I find that the significance of this work is not high enough to warrant acceptance at this time, but the authors would do well to incorporate the reviewers suggestions to improve the paper. 
Decision: Reject