Repository: statsmodels/statsmodels

Issue:
Error in formula for WLS loglike
I think there's a square missing in the math equation for the WCS loglike here:

https://github.com/statsmodels/statsmodels/blob/14fb572ce70202f970786986179dbc91efdd0aa3/statsmodels/regression/linear_model.py#L713

Rendered equation is here:
https://www.statsmodels.org/devel/generated/statsmodels.regression.linear_model.WLS.loglike.html#statsmodels.regression.linear_model.WLS.loglike

It says `log(Y - Y_hat)`, but the code says `log( (y-y_hat)^2 )`.

---

Also, only if you have time to answer:
Do you have a reference that explains where this `log( (y-y_hat)^2 )` is coming from?
The code comment says "concentrated likelihood", but I'm not familiar with that.

Is this likelihood derived somewhere on Wikipedia or somewhere?

I'm struggling to write the likelihood that reproduces the correct covariance matrix / parameter errors that correspond to what WLS does here:
https://github.com/gammapy/gammapy/blob/5ceba58594953f7dede9b541727f9d79a66e20e6/gammapy/utils/fitting/tests/test_fit.py

If you have a tip that would be much appreciated.


Discussion:
(I will be offline for the next few hours)

Quick answer:
concentrated loglikelihood is when we substitute in the solution for one of the parameters (in statistics profile loglikelihood is similar)
In this case we have a expression for the MLE of the scale/variance  (wresid**2).sum() / nobs.
substituted it back into the expression for loglike for sigma^2.

This assumes scale is estimated by MLE, and not e.g. assumed to be known, otherwise we should/can not concentrate out the scale/variance.
concentrating the scale out of the likelihood is "standard econometric", i.e. in text books.

a brief google search finds this:
https://davegiles.blogspot.com/2012/07/concentrating-or-profiling-likelihood.html
equation (2) and (3), but, AFAICS, he drops the part that is just constant for the optimization ( 2 * pi)

or section 1.4 in
https://faculty.washington.edu/ezivot/econ583/mleLectures.pdf
We get the additional term that depends on the weights (inverse variance without scale) for WLS.

> It says log(Y - Y_hat), but the code says log( (y-y_hat)^2 )

that's a mistake in the doc string

I briefly skimmed you gammapy test module, but didn't see where the loglike would come in.
OLS/WLS are derived as least square (which is equivalent to MLE, except that we usually use a degrees of freedom correction in the scale/var estimate).
The cov_params do not rely on the likelihood function, they are just `sigma*2 * inv(X' X)` or the WLS equivalent using `wexog`.


@josef-pkt - Thank you for the explanation and links, I'll study those.

What I'm looking for is the likelihood function for this:
https://www.itl.nist.gov/div898/strd/nls/data/ratkowsky2.shtml
that will yield the correct parameter errors (std dev).

When I just do `wresid ** 2`, the best-fit parameters come out OK, but the parameter errors are different.

I know that I can scale the covariance matrix after the fit. But I wonder if there's a cost function that will directly yield the reference covariance matrix such that `error_i = sqrt(covar_i_i)` if I use some tool like `hesse()` in `minuit` (see [here](http://nbviewer.jupyter.org/github/iminuit/iminuit/blob/master/tutorial/basic_tutorial.ipynb)) to compute the covariance matrix via the second derivatives of the log likelihood.

> that's a mistake in the doc string

@josef-pkt - Can you fix it, please? Or should I add a commit here?
Using the log-likelihood function that includes the scale is a bit messy. 
It is easier to work directly with a nonlinear least squares solution and estimate the scale/variance separately. 
The main problem is that the MLE scale estimate divides by nobs, but we want most of the time, and NIST will divide squared residuals by `nobs - k_params`.

In GLM we compute the hessian for the cov_params, but we treat the scale/variance/dispersion as given. That is we only compute the hessian with respect to the mean parameters.
This is fine with these distributions (linear exponential family like Gaussian) because the asymptotic or expected Hessian is block-diagonal between mean parameters and scale estimate.

So I guess minuit will give you for the hessian the part of the mean parameter, which essentially replaces wexog by the Jacobian of the mean function. The scale estimate needs to be added as a multiplicative factor as in WLS.

The alternative is to write the full loglikelihood for both mean parameters and scale without concentrating out the scale, and then compute the hessian for both sets of parameters. The relevant part for the mean parameters will be numerically a bit different from the least squares cov_params because block-diagonality of the hessian is not imposed and the scale estimate is MLE. Asymptotically they are the same, but in small samples as in the NIST example with 9 observations there is most likely a visible difference.

(Several models like MixedLM and statespace models go the MLE route of estimating variances simultaneously with the mean parameters.)





Resolving patch:
diff --git a/statsmodels/regression/linear_model.py b/statsmodels/regression/linear_model.py
--- a/statsmodels/regression/linear_model.py
+++ b/statsmodels/regression/linear_model.py
@@ -711,7 +711,7 @@ def whiten(self, X):
             return np.sqrt(self.weights)[:, None]*X
 
     def loglike(self, params):
-        """
+        r"""
         Returns the value of the gaussian log-likelihood function at params.
 
         Given the whitened design matrix, the log-likelihood is evaluated
@@ -729,9 +729,11 @@ def loglike(self, params):
 
         Notes
         --------
-        .. math:: -\\frac{n}{2}\\log\\left(Y-\\hat{Y}\\right)-\\frac{n}{2}\\left(1+\\log\\left(\\frac{2\\pi}{n}\\right)\\right)-\\frac{1}{2}log\\left(\\left|W\\right|\\right)
+        .. math:: -\frac{n}{2}\log SSR -\frac{n}{2}\left(1+\log\left(\frac{2\pi}{n}\right)\right)-\frac{1}{2}\log\left(\left|W\right|\right)
 
-        where :math:`W` is a diagonal matrix
+        where :math:`W` is a diagonal weight matrix matrix and
+        :math:`SSR=\left(Y-\hat{Y}\right)^\prime W \left(Y-\hat{Y}\right)` is
+        the sum of the squared weighted residuals.
         """
         nobs2 = self.nobs / 2.0
         SSR = np.sum((self.wendog - np.dot(self.wexog, params))**2, axis=0)


Test patch: