Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors discussed the maximum entropy method of obtaining estimators for discrete probability distribution. They also gave an exposition regarding different moment type constraints and how they behave under affine data transformation. They authors then performed numerical experiments comparing the maximum entropy method under different constraints and the regularized maximum likelihood estimator.

I find that overall the writing is fine but some parts were quite difficult to understand. For example, I found it hard to follow the discussions concerning equations (20) to (24). What do you mean by fixing the second moment? Are you setting (23) to a constant?

However I think the main problem of this paper is the use of average KL to measure performance given in (5). The use of (5) as a lost function does not seem to be correct. True $q$ is a fixed quantity and you cannot directly put a prior on true parameter values. It is easy to be confused when you are talking about Bayesian procedures and true parameter values. So let me try to break this down. Let us assume that the data is generated from some unknown true distribution $Z_{i_1},\dotsc,Z_{i_M}\sim P(Z=z_k)=q_k^0$ for $k=1,\dotsc,n$, where $q_k^0$ is unknown. Since we do not know the true distribution, we come up with a model for the observed data $Z_{i_1},\dotsc,Z_{i_M}$, say $Z_{i_1},\dotsc,Z_{i_M}\sim P(Z=z_k)=q_k$. So the $q_k$'s here are the parameters for our model which we need to estimate. Let us do the Bayesian way by putting a prior on the $q_k$'s, for example $(q_1,\dotsc,q_n)\sim\mathrm{Dirichlet}(\alpha_1,\dotsc,\alpha_n)$. For there we can get the posterior $\Pi(q_1,\dotsc,q_n | Z_{i_1},\dotsc,Z_{i_M})$ by Bayes theorem and the posterior mean $\hat{q}_k=\mathrm{E}(q_k|Z_{i_1},\dotsc,Z_{i_M})$. So the $hat{q}_k$ is supposed to be the estimator for the unknown $q_k^0$ and we can quantify their distance using KL by
$$K(q^0,\widehat{q})=\sum_{k=1}^nq_k^0\log\frac{q_k^0}{\widehat{q}_k}.$$

In the paper, the authors endowed a Dirictlet prior on the $q_k^0$'s, and this is not correct. In Bayesian statistics, you endow a prior on the model of the truth and not directly on the truth. From here you can talk about things like how does my estimator performed across all different kinds of truth, i.e., 
$$
\sup_{q^0\in\mathcal{S}}K(q^0,\widehat{q})
$$
where $\mathcal{S}$ is the simplex in $\mathrm{R}^n$.

Typos:
1. Page 1, 2nd line from below, main results are that...

2. The line after (15), do you mean $\alpha_k^{[1]},\dotsc,\alpha_k^{[L]}$? Then the line after this,
Below we show some concrete...

3. The 3rd line after (24), (17) is a concave function in $\{q_k\}_{k=1}^n$

4. In (7), the range of integration is over the simplex $\{(q_1,\dotsc,q_n):\sum_{k=1}^np_k=1,0\leq p_k\leq1,\forall k\}$.
review_title: Average KL does not seem to be correct

final_score: rating: 4: Ok but not good enough - rejection
review_content: # Summary

This paper investigates maximum entropy (MaxEnt) inference and compares it to a Bayesian estimator and regularized maximum likelihood for finite models. To assess the accuracy of the different estimators, the authors use the average KL-divergence between the ground truth and the estimator, where the average is computed over all datasets of a given size $M$ and all probabilistic models of size $n$ (generated by some prior, either a single Dirichlet or a mixture of Dirichlets). Using numerical experiments, the authors find that the performance of MaxEnt deteriorates for sparse data generated from uniform models. However, by exploiting knowledge about the order of probabilities, MaxEnt can outperform regularized maximum likelihood. 

# Assessment

I have to admit that I had different expectations based on the title and found the paper of limited interest. After reading the paper, I'm a bit puzzled as to what I learned and how much the findings can be generalized. Therefore, I would rather reject the paper in its current version.

Let me try to detail some of the problems that I have with the paper:

* The whole setting seems arbitrary to me. The author call their setup a "Bayesian approach" because the correct prior is used to assess the quality of an estimator (when computing $\langle \overline K\rangle$). In my understanding, a Bayesian approach would try to make reasonable assumptions about the likelihood and prior, integrate out any uninteresting parameters, and look at the resulting posterior. 

* Typically, MaxEnt is used to incorporate expectations that are assumed to be known exactly. The paper seems to use MaxEnt as a blackbox method in order to circumvent the need of defining a prior. To apply MaxEnt, the data are transformed to moments (by computing sample means), which are then used to constrain the model. It is clear that for small $M$ the estimated expectations obtained by computing sample means are not very reliable and therefore MaxEnt (as used by the authors) performs purely on sparse data. 

* The numerical experiments are run for very specific choices of $n$, $\alpha_k$, $z_k$, etc. There is no motivation for these choices. It remains unclear to me if the results generalize to other choices. 

* As the authors acknowledge themselves the approach is limited to finite models with $n$ states. What about continuous or categorical variables?

review_title: A limited comparison between MaxEnt and regularized Maximum Likelihood using a Bayesian preformance measure

final_score: rating: 3: Clear rejection
review_content: This paper considers the maximum entropy (MAXENT) method for estimating underlying probabilities over a finite alphabet, i.e., the multinomial model. The authors compare MAXENT with the regularized maximum likelihood, that is the Bayesian estimator under the Dirichlet prior with a common hyperparameter, and the Bayesian estimator with a general Dirichlet prior in terms of the Bayes risk, i.e., the KL-divergence from the true distribution to the estimated distribution averaged over the prior. The authors also consider the case where the prior is extended to the mixture of Dirichlet distributions. These comparisons are done numerically with synthetic data.

Although the practical performance of MAXENT is of interest, the paper provides little novel knowledge about it.

- The numerical experiments are at a too small scale. Without theoretical results, the experiments are required to be more comprehensive. The reported experiments are small scale both in terms of the alphabet size n and the data sequence length M.

- Isn’t it possible to include any discussion on hyperparameter estimation, for example, empirical Bayes since the hyperparameter estimation of the Dirichlet prior has long been studied?

- p.7, last paragraph: I wonder why the random perturbation of the ordering of Z devastates MAXENT since the hyperparametes \alpha can also be permutated accordingly.

Minor:
Right after eq. (15): \alpha_k^{[1]},..., \alpha_k^{[L]} (The last one should be \alpha_k^{[L]}.)
p.7, l.12 from the bottom: be be --> be

review_title: Too small numerical experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: Strengths: 
1.	The paper is basically clear. The claims and method are correct.
2.	This paper proposes a novel MAXENT method to compete with maximum likelihood in sparse data. The topic of the paper is highly relevant to the ICLR community. The idea is original, and the motivation is strong in theoretical analysis. The theoretical analysis is presented in a fine manner. The numerical experiments show the validity of the proposed method.
3.    By assuming a well-defined prior Dirichlet density for unknown probabilities, it employs the average KL distance in evaluating the relevance of various MAXENT constraints, checking its general applicability, and comparing MAXENT with estimators having various degrees of dependence on the prior, the regularized ML and the Bayesian estimators.

Weakness: 
1.	I’m confused about the application scenarios of the proposed method. The authors did not show any application value, though they claimed that MAXENT has a large number of applications in applied machine learning. The proposed method is not easy to understand and implement. The author should at least add one real data application.
2.	Differences between the proposed method and the existing methods (Good, 1970; Zhu et al,. 1997; Pandey & Dukkipati, 2013) should be stressed more. The differences can be presented theoretically and numerically.

review_title: Maximum Entropy Competes with Maximum Likelihood. 

Meta-review: As one of the reviewers concisely summarized: This paper investigates maximum entropy (MaxEnt) inference and compares it to a Bayesian estimator and regularized maximum likelihood for finite models. 

Two reviewers specifically question whether they have learned anything new after reading. This combined with various other drawbacks described during the review phase led to strong agreement among the reviewers about a variety of deficiencies in this paper. One reviewer initially gave a relatively high score but has since revised his/her opinion in light of the other reviews and discussion. I find that the significance of this work is not high enough to warrant acceptance at this time, but the authors would do well to incorporate the reviewers suggestions to improve the paper. 
Decision: Reject