On the benefits of maximum likelihood estimation for Regression and Forecasting

1 INTRODUCTION . The task of fitting a regression model for a response variable y against a covariate vector x ∈ Rd is ubiquitous in supervised learning in both linear and non-linear settings ( Lathuilière et al. , 2020 ; Mohri et al. , 2018 ) as well as non-i.i.d settings like multi-variate forecasting ( Salinas et al. , 2020 ; Wang et al. , 2019 ) . The end goal in regression and forecasting problems is often to use the resulting model to obtain good performance in terms of some target metric of interest on the population level ( usually measured on a previously unseen test set ) . The mean-squared error or the mean absolute deviation are examples of common target metrics . In this paper , our focus is on the choice of loss function that is used to train such models , which is an important question that is often overlooked , especially in the deep neural networks context where the emphasis is more on the choice of network architecture ( Lathuilière et al. , 2020 ) . Perhaps the most common method used by practitioners for choosing the loss function for a particular regression model is to directly use the target metric of interest as the loss function for empirical risk minimization ( ERM ) over a function class on the training set . We denote this approach for choosing a loss function as Target Metric Optimization ( TMO ) . This is especially more common with the advent of powerful general purpose function optimizers like deep networks and has also been rigorously analyzed for simpler function classes ( Mohri et al. , 2018 ) . Target Metric Optimization seems like a reasonable approach - if the practitioner knows about the target metric of interest for prediction using the model , it seems intuitive to optimize for the same objective on the training data . Prior work ( both theoretical and applied ) has both advocated for and argued against TMO for regression problems . Many prominent works on regression ( Goldberger et al. , 1964 ; Lecue & Mendelson , 2016 ) use the TMO approach , though most of them assume that the data is well behaved ( e.g . sub-Gaussian noise ) . In terms of applications , many recent works on time-series forecasting ( Wu et al. , 2020 ; Oreshkin et al. , 2019 ; Sen et al. , 2019 ) also use the TMO approach directly on the target metric . On the other hand , the robust regression literature has long advocated for not using the target metric directly for ERM in the case of contamination or heavy tailed response/covariate behaviour ( Huber , 1992 ; Hsu & Sabato , 2016 ; Zhu & Zhou , 2021 ; Lugosi & Mendelson , 2019a ; Audibert et al. , 2011 ; Brownlees et al. , 2015 ) on account of its suboptimal high-probability risk bounds . However , as noted in ( Prasad et al. , 2020 ) , many of these methods are either not practical ( Lugosi & Mendelson , 2019a ; Brownlees et al. , 2015 ) or have sub-optimal empirical performance ( Hsu & Sabato , 2016 ) . Even more practical methods such as ( Prasad et al. , 2020 ) would lead to sufficiently more computational overhead over standard TMO . Another well known approach for designing a loss function is Maximum Likelihood Estimation ( MLE ) . Here one assumes that the conditional distribution of y given x belongs to a family of distributions p ( y|x ; θ ) parameterized by θ ∈ Θ ( McCullagh & Nelder , 2019 ) . Then one can choose the negative log likelihood as the loss function to optimize using the training set , to obtain an estimate θ̂mle . This approach is sometimes used in the forecasting literature ( Salinas et al. , 2020 ; Davis & Wu , 2009 ) where the choice of a likelihood can encode prior knowledge about the data . For instance a negative binomial distribution can be used to model count data . During inference , given a new instance x′ , one can output the statistic from p ( y|x′ ; θ̂mle ) that optimizes the target metric , as the prediction value ( Gneiting , 2011 ) . MLE also seems like a reasonable approach for loss function design - it is folklore that the MLE is asymptotically optimal for parameter estimation , in terms of having the smallest asymptotic variance among all estimators ( Heyde , 1978 ; Rao , 1963 ) , when the likelihood is well-specified . However , much less is known about finite-sample , fixed-dimension analysis of MLE , which is the typical regime of interest for the regression problems we consider in this paper . An important practical advantage for MLE is that model training is agnostic to the choice of the target metric - the same trained model can output estimators for different target metrics at inference time . Perhaps the biggest argument against the MLE approach is the requirement of knowing the likelihood distribution family . We address both these topics in Section 5 . Both TMO and MLE can be viewed as offering different approaches to selecting the loss function for a given regression model . In this paper , we argue that in several settings , both from a practical and theoretical perspective , MLE might be a better approach than TMO . This result might not be immediately obvious apriori - while MLE does benefit from prior knowledge of the distribution , TMO also benefits from prior knowledge of the target metric at training time . Our main contributions are as follows : Competitiveness of MLE : In Section 3 , we prove that under some general conditions on the family of distributions and a property of interest , the MLE approach is competitive with any estimator for the property . We show that this result can be applied to fixed design regression problems in order to prove that MLE can be competitive ( up to logarithmic terms ) with any estimator in terms of excess square loss risk , under some assumptions . Example Applications : In Section 4.1 , we apply our general theorem to prove an excess square loss bound for the the MLE estimator for Poisson regression with the identity link ( Nelder & Wedderburn , 1972 ; Lawless , 1987 ) . We show that these bounds can be better than those of the TMO estimator , which in this case is least-squares regression . Then in Section 4.2 , we show a similar application in the context of Pareto regression i.e y|x follows a Pareto distribution . We show that MLE can be competitive with robust estimators like the one in ( Hsu & Sabato , 2016 ) and therefore can be better than TMO ( least-squares ) . Empirical Results : We propose the use of a general purpose mixture likelihood family ( see Section 5 ) that can capture a wide variety of prior knowledge across datasets , including zero-inflated or bursty data , count data , sub-Gaussian continuous data as well as heavy tailed data , through different choices of ( learnt ) mixture weights . Then we empirically show that the MLE approach with this likelihood can outperform ERM for many different commonly used target metrics like WAPE , MAPE and RMSE ( Hyndman & Koehler , 2006 ) for two popular forecasting and two regression datasets . Moreover the MLE approach is also shown to have better probabilistic forecasts ( measured by quantile losses ( Wen et al. , 2017 ) ) than quantile regression ( Koenker & Bassett Jr , 1978 ; Gasthaus et al. , 2019 ; Wen et al. , 2017 ) which is the TMO approach in this case . 2 PRIOR WORK ON MLE . Maximum likelihood estimators ( MLE ) have been studied extensively in statistics starting with the work of Wald ( 1949 ) ; Redner ( 1981 ) , who showed the maximum likelihood estimates are asymptotically consistent for parametric families . Fahrmeir & Kaufmann ( 1985 ) showed asymptotic normality for MLE for generalized linear models . It is also known that under some regularity assumptions , MLE achieves the Cramer-Rao lower bound asymptotically ( Lehmann & Casella , 2006 ) . However , we note that none of these asymptotic results directly yield finite sample guarantees . Finite sample guarantees have been shown for certain problem scenarios . Geer & van de Geer ( 2000 ) ; Zhang ( 2006 ) provided uniform convergence bounds in Hellinger distance for maximum likelihood estimation . These ideas were recently used by Foster & Krishnamurthy ( 2021 ) to provide algorithms for contextual bandits . There are other works which study MLE for non-parametric distributions e.g. , Dümbgen & Rufibach ( 2009 ) showed convergence rates for log-concave distributions . There has been some works ( Sur & Candès , 2019 ; Bean et al. , 2013 ; Donoho & Montanari , 2016 ; El Karoui , 2018 ) that show that MLE can be sub-optimal for high dimensional regression i.e when the dimension grows with the number of samples . In this work we focus on the setting where the dimension does not scale with the number of samples . Our MLE results differ from the above work as we provide finite sample competitiveness guarantees . Instead of showing that the maximum likelihood estimator converges in some distance metric , we show that under some mild assumptions it can work as well as other estimators . Hence , our methods are orthogonal to known well established results in statistics . Perhaps the closest to our work is the competitiveness result of Acharya et al . ( 2017 ) , who showed that MLE is competitive when the size of the output alphabet is bounded and applied to profile maximum likelihood estimation . In contrast , our work applies for unbounded output alphabets and can provide stronger guarantees in many scenarios . 3 COMPETITIVENESS OF MLE . In this section , we will show that under some reasonable assumptions on the likelihood family , the MLE is competitive with any estimator in terms of estimating any property of a distribution from the family . We will then show that this result can be applied to derive bounds on the MLE in some fixed design regression settings that can be better than that of TMO . We will first setup some notation . Notation : Given a positive semi-definite symmetric matrix M , ‖x‖M : = xTMx is the matrix norm of the vector x. λ ( M ) denotes an eigenvalue of a symmetric square matrix M ; specifically λmax ( M ) and λmin ( M ) denote the maximum and minimum eigenvalues respectively . The letter f is used to denote general probability densities . We use p to denote the conditional probability density of the response given the covariate . ‖·‖1 will be overloaded to denote the ` 1 norm between two probability distributions i.e ‖p− p′‖1 : = ∫ |p ( z ) − p′ ( z ) |dz . DKL ( p1 ; p2 ) will be used to denote the KL-divergence between the two distributions . If Z is a set equipped with a norm ‖·‖ , then N ( , Z ) will denote an -net i.e any point z ∈ Z has a corresponding point z′ ∈ N ( , Z ) s.t ‖z − z′‖ ≤ . Bdr denotes the ball centered at the origin with radius r and Sd−1r denotes its surface . We define [ n ] : = { 1 , 2 , · · · , n } . | · | denotes the size of the enclosed set . General Competitiveness : We first consider a general family of distributions F over the space Z . For a sample z ∼ f ( for z ∈ Z and f ∈ F ) , the MLE distribution is defined as fz = argmaxf∈F f ( z ) . We are interested in estimating a property π : F → W of these distributions from an observed sample . The following definition will be required to impose some joint conditions on the distribution family and the property being estimated , that are needed for our result . Definition 1 . The tuple ( F , π ) , where F is a set of distributions and π : F → W a property of those distributions , is said to be ( T , , δ1 , δ2 ) -approximable , if there exists a set of distributions F̃ ⊆ F s.t |F̃ | ≤ T and for every f ∈ F , there exists a f̃ such that ‖f − f̃‖1 ≤ δ1 and Prz∼f ( ∥∥∥π ( fz ) − π ( f̃z ) ∥∥∥ ≥ ) ≤ δ2 , where f̃z = argmaxf̃∈F̃ f̃ ( z ) andW has a norm ‖·‖ . The above definition states that the set of distributions F has a finite δ-cover , F̃ in terms of the ` 1 distance . Moreover the cover is such that solving MLE on the cover and applying the property π on the result of the MLE is not too far from π applied on the MLE over the whole set F . This property is satisfied trivially if F is finite . We note that it is also satisfied by some commonly used set of distributions and corresponding properties . Now we state our main result . Theorem 1 . Let π̂ be an estimator such that for any f ∈ F and z ∼ f , Pr ( ‖π ( f ) − π̂ ( z ) ‖ ≥ ) ≤ δ . Let Ff be a subset of F that contains f such that with probability at least 1− δ , fz ∈ Ff and ( Ff , π ) is ( T , , δ1 , δ2 ) -approximable . Then the MLE satisfies the following bound , Pr ( ‖π ( f ) − π ( fz ) ‖ ≥ 3 ) ≤ ( T + 3 ) δ + δ1 + δ2 . We provide the proof of Theorem 1 in Appendix A . We also provide a simpler version of this result for finite distribution families as Theorem 3 in Appendix A for the benefit of the reader . Competitiveness in Fixed Design Regression : Theorem 1 can be used to show that MLE is competitive with respect to any estimator for square loss minimization in fixed design regression . We will first formally introduce the setting . Consider a fixed design matrix X ∈ Rn×d where n is the number of samples and d the feature dimension . We will work in a setting where n d. The target vector is a random vector given by yn ∈ Rn . Let yi be the i-th coordinate of yn and xi denote the i-th row of the design matrix . We assume that the target is generated from the conditional distribution given xi such that , yi ∼ p ( ·|xi ; θ∗ ) , θ∗ ∈ Θ . We are interested in optimizing a target metric ` ( · , · ) given an instance of the random vector yn . The final objective is to optimize , min h∈H Eyi∼p ( ·|xi ; θ∗ ) [ 1 n n∑ i=1 ` ( yi , h ( xi ) ) ] , whereH is a class of functions . In this context , we are interested in comparing two approaches . TMO ( see ( Mohri et al. , 2018 ) ) . This is standard empirical risk minimization on the target metric where given an instance of the random vector yn one outputs the estimator ĥ = minh∈H 1 n ∑n i=1 ` ( yi , h ( xi ) ) . MLE and post-hoc inference ( see ( Gneiting , 2011 ) ) . In this method one first solves for the parameter in the distribution family that best explains the empirical data by MLE i.e. , θ̂mle : = argmin θ∈Θ L ( yn ; θ ) , where L ( yn ; θ ) : = n∑ i=1 − log p ( yi|xi ; θ ) Then during inference given a sample xi the predictor is defined as , h̃ ( xi ) : = argminŷ Ey∈p ( ·|xi ; θ̂mle ) [ ` ( y , ŷ ) ] or in other words we output the statistic from the MLE distribution that optimizes the loss function of interest . For instance if ` is the square loss , then h̃ ( xi ) will be the mean of the conditional distribution p ( y|xi ; θ̂mle ) . We will prove a general result using Theorem 1 when the target metric ` is the square loss andH is a linear function class . Moreover , the true distribution p ( ·|xi ; θ∗ ) is such that E [ yi ] = 〈θ∗ , xi〉 for all i ∈ [ n ] i.e we are in the linear realizable setting . In this case the quantity of interest is the excess square loss risk given by , E ( θ ) : = 1 n n∑ i=1 Eyn‖yn −Xθ‖22 − 1 n n∑ i=1 Eyn‖yn −Xθ∗‖22 = ‖θ − θ ∗‖2Σ , ( 1 ) where Σ : = ( ∑ i xix T i ) /n is the normalized covariance matrix , and θ ∗ is the population minimizer of the target metric over the linear function class . Now we are ready to state the main result . Theorem 2 . Consider a fixed design regression setting where the likelihood family is parameterized by θ ∈ Θ ⊆ Bdw and |N ( , Θ ∩ Bdw ) | ≤ |N ( , Bdw ) | for a small enough . Further the following conditions hold , 1 . DKL ( p ( yi ; θ ) ; p ( yi ; θ′ ) ) ≤ L‖θ − θ′‖2 . 2 . The negative log-likelihood L ( yn ; θ ) as a function of θ is α-strongly convex and β-smooth , w.p . at least 1− δ . Further suppose there exists an estimator θest such that E ( θest ) ≤ ( c1 + c2 log ( 1/δ ) ) η/n , where c1 , c2 are problem dependent quantities and η > 0 . Then the MLE estimator also satisfies , E ( θ̂mle ) = O ( ( c1 + c2d ( log n+ log ( wLλmax ( Σ ) ) + log ( β/α ) + log 1 δ ) ) η n ) w.p at least 1− δ . We provide the proof in Appendix C. The proof involves proving the conditions in Theorem 1 and bounding the size of the cover T . In order to better understand Theorem 2 , let us consider a typical case where there exists a possibly complicated estimator such that E ( θest ) = O ( ( d + log ( 1/δ ) ) /n ) . In this case the above theorem implies that MLE will be competitive with this estimator up to a log n factor . In many cases the MLE might be much simpler to implement than the original estimator but would essentially match the same error bound . We now provide specific examples in subsequent sections .

Summary: The paper provides theoretical results that favor MLE estimators, in terms of the excess square loss risk,  compared to empirical risk estimators under mild assumption. In particular, the paper devises an estimator for Poisson regression and employs an existing estimator for heavy-tail Pareto regression, and subsequently derives upper bounds on the excess square loss risk that are tighter compared to the least squares estimator. Finally, a mixture of distributions (with each component coming from a different distribution class) is proposed. The MLE for the proposed mixture outperforms empirical risk estimators under different objective losses.