Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides theoretical results that favor MLE estimators, in terms of the excess square loss risk,  compared to empirical risk estimators under mild assumption. In particular, the paper devises an estimator for Poisson regression and employs an existing estimator for heavy-tail Pareto regression, and subsequently derives upper bounds on the excess square loss risk that are tighter compared to the least squares estimator. Finally, a mixture of distributions (with each component coming from a different distribution class) is proposed. The MLE for the proposed mixture outperforms empirical risk estimators under different objective losses. 
main_review: The theoretical novelty of the paper lies in the fact that it is the first work that determines the performance of a MLE in terms of the performance of existing estimators using finite samples, as opposed to asymptomatic convergence to the true value. Subsequently, it derives tighter bounds, than existing results, for mle poisson and pareto regression. 

(1) The results seem to hold in low-dimensional cases where (n -- samples> d- covariates). The practical utility of the paper would be stronger if it could be applied in the high-dimensional regime ( for example in corollary 2, the d^2 factor seems to be warning in this scenario).

(2) Similarly, it would be interesting if the authors could compare also with ridge/lasso estimators in section (6).

(3) The bound of corollary (2) is the tightest that can be achieved (one cannot find a different \theta_est satisfying the conditions of theorem (2) that could yield better excess square loss risk for the MLE?) 

(4) Regarding the bound of lemma 1, and given that I am not an expert in the area, I cannot grasp its tightness (how large is \lambda_max (\Sigma) ) in practical settings. I think it would help if the authors could show experimentally ( for experiments that do not make use of the mixture of distributions suggested- but only the poisson and/or pareto distribution separately ) and demonstrate empirically the tightness of the bounds.

(5) Moreover, I think it would be useful if the authors could summarize in a Table n and d for the datasets considered.

(6) Given that the resulting loss  that stems from the proposed mixture is con-convex,  it would be useful if the authors could demonstrate convergence empirically by providing some learning curves in the appendix.

(7) Finally, it is not clear to me why theorem 2 can be applied on the mixture (even when it holds separately for each component).
summary_of_the_review: The paper makes theoretical contributions on the performance of mle compared to empirical risk losses. 
However, the applicability of the derived results in real-world regimes could be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper compares two inferential methods for regression models, the maximum likelihood estimation and the estimation based on loss functions. For that purpose, a quantity is proposed to measure the correctness of an estimator of a regression model. Then it is shown that, under certain conditions on the quantity of an estimator, the proposed quantity of the maximum likelihood estimator can be evaluated. This result is applied to two regression models based on the Poisson distribution and Pareto distribution. Choices of the probability distributions and target metrics are discussed for the maximum likelihood estimation. Experiments are given to compare the performance of the maximum likelihood estimator and the estimators based on some loss functions.
main_review: **Strengths:**

(a) The paper tackles an interesting and challenging topic to compare two well-known inferential methods for regression, namely, the maximum likelihood estimation (MLE) and the estimation based on loss functions (TMO).

(b) A new theorem, Theorem 2, is given to compare the performance of the MLE with that of an estimator in terms of a quantity which measures the correctness of estimators of regression coefficients.

(c) Applying Theorem 2, detailed discussion is given to compare MLE and TMO for Poisson regression and Pareto regression.

**Weaknesses:**

(d) I wonder the mathematical correctness of the discussion in Section 5 related to the three component mixture distribution which consists of the point distribution at 0, the negative binomial distribution and Pareto distribution. The point distribution and negative binomial distribution are discrete distributions, while Pareto distribution is a continuous distribution. Therefore the likelihood function of the mixture of these distributions should not be evaluated based on the function $\sum_{j=1}^3 w_j p_j(y| \boldsymbol{x} ; \boldsymbol{\theta}_j)$, where $p_1$ and $p_2$ are the probability mass functions of the point distribution and the negative binomial distribution, respectively, and $p_3$ is the probability density function of the Pareto distribution. I am worried from the discussion in Section 5 that this approach is used for the maximum likelihood estimation of the mixture distribution. There should be more explanation about the maximum likelihood estimation of this mixture model.

(e) If my worry in Comment (e) is correct, part of the results of the experiments in Section 6 regarding the three component mixture should be reconsidered.

(f) In the paper, MLE and TMO are compared through the quantity given in Equation (1). However I am not sure how meaningful this comparison will be because MLE and TMO are generally the minimizers of loss functions which are not based on the square loss risk as in (1). Since the two estimators are obtained through different loss functions, I think it is difficult to compare the goodness of the estimators through the single quantity such as the quantity (1). Or is there any special meaning to use the quantity (1) to evaluate the MLE and TMO?

**Other Comments:**

(g) I am confused about the MLE for regression models used in the paper. In the ordinary MLE for regression, the parameters of a probability distribution and regression coefficients are estimated simultaneously via the maximization of the likelihood function. However it seems from the paragraph titled "MLE and post-hoc inference" on page 4 that the parameters of the distribution $\boldsymbol{\theta}$ and the link function $\tilde{h}$ are estimated in different steps. Is this approach used for Theorem 2 and other results of the paper? This should be explained in the paper because the approach discussed there is different from the common approach for MLE.

(h) p.2, Section 1, Competitiveness of MLE, l.2 up: MLE can competitive  ===> MLE can be competitive

(i) p.3, Section 3, Notation, ll.1-2 up: the sphere centered at ... ===> the ball centered at ... (I think this expression is less confusing.)

(j) p.3, Definition 1, l.3: The definition of $| \cdot |$ in $|\tilde{\cal F}| \leq T$ should be given.

(k) p.3, Theorem 1, l.2 up: Is "the MLE based estimator" exactly the same as the MLE or something different?

(l) p.6, Equation (4):  As a comment related to Comment (g), how is the parameter $b$ in the Pareto regression model (4) estimated?

(m) p.8, Section 6, Common Experimental Protocol, 2nd paragraph, l.2 up: that employs the mixture ===> that employ the mixture

(n) p.28, l.7 up: be keep the number ===> keep the number
summary_of_the_review: The paper tackles a challenging topic to compare two well-known inferential methods for regression. Some new theoretical results are presented on this topic. However I have concerns about the mathematical correctness of the paper (see Comments (d) and (e)) and the motivation for the study (see Comment (f)).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper promotes the maximum likelihood estimation over target metric optimization. The main theoretical results is a finite sample error bound. Specific examples and numerical results are provide to further illustrate the investigation.
main_review: Strengths:

- It is well known that the MLE is efficient under very mild assumptions, so although less intuitive compared with the TMO, it is important to promote its usage. One disadvantage of MLE is the likelihood is different to determine and it is often misspecified in practice. The paper discusses this point and recommend using a mixture of distributions. 
- The finite sample error bound is obtained in a more general context than Acharya et al. (2017).
- Specific examples (Poisson regression and Pareto regression) are provided to illustrate the theoretical results.

Weaknesses:

- The notations and presentation are difficult to follow. Some key notations do not seem to be properly defined. Here are some examples. What is the meaning of $\ell_{1}$ norm for probability distributions? $f$ stands for the probability density function or cumulative distribution function or the underlying probability measure? What is the meaning of $|\tilde{F}|$? From Definition 1, it seems to be the size of the set, but this does not seem to be the case when I check the proof. Due to these issues, I was not able to fully understand and appropriately evaluate the significance of the theoretical results, and I have reflect this in my confidence level.
- The assumptions in Theorem 1 seems to already require that the MLE has a tight bound. I am not clear how to interpret the main result.
- I am not sure if it is true that the proof for discrete distributions can be easily extended to continuous distributions. This is usually not the case in existing MLE theory partly due to fact that the support for continuous distributions are not countable.
- Why both $\ell_{1}$ norm and KL-divergence are required.
- For the two distributions in section 4, it is well known that the MLE is more efficient than the least squares. 
- Minor issues:
  - "w.p" should be "w.p." in multiple places
  - "s.t" should be "s.t."
  - The result in (Bai & Yin, 2008) for random covariance matrices is for Gaussian distributed covariates only. 
  - It might be better to give the definition of MAPE, WAPE and RMSE in the main paper than in the appendix so that readers can follow the main paper without going back and forth. 

summary_of_the_review: This paper in about an interesting and important idea, but the notations and presentation need some improvement.
review_title: null

Meta-review: This paper has been independently assessed by three expert reviewers. The results place it at the borderline of acceptance decision: while one of the reviewers gave it a straight accept evaluation, two others assessed it as marginally rejectable, even after discussion with the authors. All of the reviewers agreed that the theoretical results provided should help promote the use of MLE estimators over perhaps more prevalently used in current practice TMO, and that is the main contribution of this work. The reviewers were concerned with the clarity of the presentation and with a confusing notation used. Some of these issues have been addressed in the authors' responses. All things considered, I conclude that this work can be of some interest to the ICLR audience, and as such it can be assessed as marginally acceptable for this conference: "accept if needed". I will recommend it as such for consideration by the Senior Area Chair and the Program Committee.
Decision: Accept (Poster)