Repository: statsmodels/statsmodels

Issue:
BUG: GLM Binomial count, incorrect score, score_obs
It looks like GLM Binomial score and score_obs methods ignore n_trials.

Needs to be confirmed. I am debugging a failure of a new test for penalized GLM Binomial count #4576 and score at IRLS params are very large.

p2 is model estimated with IRLS
res2.model has convergence problems with `bfgs`

```
(Pdb) res2.model.score(p2.params)
array([ 1.82844758, -0.57849861,  1.19638513, -0.32730399])
(Pdb) res2.model.score_obs(p2.params).sum(0)
array([ 1.82844758, -0.57849861,  1.19638513, -0.32730399])
(Pdb) res2.model.score_obs(p2.params).T.dot(res2.model.n_trials)
array([  1.10841891e-11,   1.00175789e-11,   7.95967285e-12,
         7.00894074e-12])
```

maybe the unit tests for the GLM refactoring doesn't have a Binomial count case


Discussion:
Thanks for finding this, Josef. That's probably my doing. I agree with your other comment about our test case suite, although huge, has some pretty gaping holes. GLM binomial is a little different because it has its own bespoke "weight"-like thing, `ntrials`. 

While I tried to make it as generic as possible, I also got confused because there is overlap with with `ntrials`, `var_weights`, and `freq_weights`. It might make sense to assume/require that the user is either allowed to use `ntrials` or `freq_weigths/var_weights`, but not both. 
I'm not sure what to do with Binomial counts. There are too many special case handling for it in the current code, and it has been neglected as a standard model for unit testing (including by myself).

The user doesn't use n_trials directly, but is allowed to specify endog as (successes, failures), except that I added n_trials now to kwargs for internal use.

Some idea: 
- just fix bugs and add more unit tests (immediate approach)
- use var_weights internally instead of n_trials once var_weights has stabilized and a bit battle tested
- split BinomialCount from Binomial as separate family and keep endog as counts (similar to Poisson), avoids confusing endog as count versus endog as proportion.
- drop (successes, failures) as permitted endog, but allow proportion with n_trials or var_weights similar to R.  (this is what I use now for replicating a model)
- related: we also have exposure as possible equivalent of n_trials if we generalize exposure, which is again almost or essentially equivalent to what we can do with var_weights.



It looks like there is also a bug in the hessian with observed=True

from a failing log-binomial test case with n_trials=10:
there are convergence problem with bfgs, but if we start close to irls params, then we get to the same params as irls (at convergence tolerance)

however, hessian is not invertible, and the difference between OIM and EIM is very large especially for the second parameter which has the wrong sign on the diagonal of OIM.

```
(Pdb) rslt_irls.model.score(rslt_irls.params)
array([  3.35782900e-07,   3.83264057e-05,  -5.74391843e-05])
(Pdb) rslt_irls.model.hessian(rslt_irls.params)
array([[ -577.12454715,  1267.24619673, -1228.98579486],
       [ 1267.24619673,  4501.71344637,   131.42380418],
       [-1228.98579486,   131.42380418, -5607.05729572]])

(Pdb) rslt_irls.model.hessian(rslt_irls.params, observed=False)
array([[-780.98173872, -270.96784394, -198.61793915],
       [-270.96784394, -789.92625106,   31.97257933],
       [-198.61793915,   31.97257933, -766.91849269]])
```

log-binomial can be a difficult case to optimize, but the fittedvalues at IRLS params are between 0.3 and 0.6 and shouldn't run into problems with domain violations.
And another bug in hessian with observed=True
It looks like I'm using eimfactor twice in the tmp part, 
tmp already includes eim_factor and then we multiply again with it.
`oim_factor = eim_factor * (1 + tmp)`

I don't remember the derivation for the expressions here (of course not), but unit test pass if I replace this line with
`oim_factor = eim_factor * (1) + tmp`

**edit** this change should be correct
based on collecting terms in the SPSS algorithm documentation, which I also did initially for hessian factor
there is still a test failure with fit_constrained but that's just that tolerance is to tight

```
>        assert_allclose(res1.cov_params()[self.idx_p_uc[:, None],
>                       self.idx_p_uc], res2.cov_params(), rtol=1e-10, atol=1e-1
5)
E       AssertionError:
E       Not equal to tolerance rtol=1e-10, atol=1e-15
E
E       (mismatch 1.5625%)
E        x: array([[  2.877556e+00,  -1.294920e-03,   6.044716e-04,   9.005805e-
04,
E                 6.786921e-04,   2.364194e-02,  -1.869324e-01,  -5.049150e-02,
E                -2.058958e-02,  -3.015464e-04,  -5.586616e-04,   5.540279e-04,.
..
E        y: array([[  2.877556e+00,  -1.294920e-03,   6.044716e-04,   9.005805e-
04,
E                 6.786921e-04,   2.364194e-02,  -1.869324e-01,  -5.049150e-02,
E                -2.058958e-02,  -3.015464e-04,  -5.586616e-04,   5.540279e-04,.
..

..\eclipse_ws\statsmodels\statsmodels_py34_pr\statsmodels\genmod\tests\test_cons
trained.py:58: AssertionError
```

```
(Pdb) np.max(np.abs(res1.cov_params()[self.idx_p_uc[:, None], self.idx_p_uc] -
 res2.cov_params()))
3.3864466786326375e-11

(Pdb) np.max(np.abs(res1.cov_params()[self.idx_p_uc[:, None], self.idx_p_uc] /
 res2.cov_params()-1))
1.0756122659216771e-09
```
this reminds me:
There was a problem comparing hessian between GLM Binomial with probit link and discrete.Probit where agreement was not very high compared to other cases. 
AFAIR, that was the only non-canonical link that I had checked when I wrote hessian_factor.
I don't remember for which unit test that was.

discrete/tests/test_sandwich_cov.py has a disabled test `T_estGLMProbit(CheckDiscreteGLM):` but based on the comment because of the link and not because of hessian.
discrete/tests/test_constrained.py compares discrete and GLM but only Poisson and Logit.

https://github.com/statsmodels/statsmodels/pull/1781/files
added a CheckComparisonMixin in test_glm.py but it is used only for Logit and Poisson which are the only two cases in test_glm that overlap with discrete.

discrete test_discrete.py doesn't have any GLM comparison tests.


Resolving patch:
diff --git a/statsmodels/genmod/generalized_linear_model.py b/statsmodels/genmod/generalized_linear_model.py
--- a/statsmodels/genmod/generalized_linear_model.py
+++ b/statsmodels/genmod/generalized_linear_model.py
@@ -526,7 +526,7 @@ def score_factor(self, params, scale=None):
 
         score_factor = (self.endog - mu) / self.family.link.deriv(mu)
         score_factor /= self.family.variance(mu)
-        score_factor *= self.iweights
+        score_factor *= self.iweights * self.n_trials
 
         if not scale == 1:
             score_factor /= scale
@@ -578,9 +578,9 @@ def hessian_factor(self, params, scale=None, observed=True):
         tmp = self.family.variance(mu) * self.family.link.deriv2(mu)
         tmp += self.family.variance.deriv(mu) * self.family.link.deriv(mu)
 
-        tmp = score_factor * eim_factor * tmp
+        tmp = score_factor * tmp
         # correct for duplicatee iweights in oim_factor and score_factor
-        tmp /= self.iweights
+        tmp /= self.iweights * self.n_trials
         oim_factor = eim_factor * (1 + tmp)
 
         if tmp.ndim > 1:
@@ -1044,6 +1044,10 @@ def _fit_gradient(self, start_params=None, method="newton",
         using the scipy gradient optimizers.
         """
 
+        # fix scale during optimization, see #4616
+        scaletype = self.scaletype
+        self.scaletype = 1.
+
         if (max_start_irls > 0) and (start_params is None):
             irls_rslt = self._fit_irls(start_params=start_params,
                                        maxiter=max_start_irls,
@@ -1056,6 +1060,9 @@ def _fit_gradient(self, start_params=None, method="newton",
                                     maxiter=maxiter, full_output=full_output,
                                     method=method, disp=disp, **kwargs)
 
+        # reset scaletype to original
+        self.scaletype = scaletype
+
         mu = self.predict(rslt.params)
         scale = self.estimate_scale(mu)
 


Test patch:
diff --git a/statsmodels/discrete/tests/test_sandwich_cov.py b/statsmodels/discrete/tests/test_sandwich_cov.py
--- a/statsmodels/discrete/tests/test_sandwich_cov.py
+++ b/statsmodels/discrete/tests/test_sandwich_cov.py
@@ -479,13 +479,9 @@ def test_basic(self):
         assert_equal(res1.cov_type, self.cov_type)
         assert_equal(res2.cov_type, self.cov_type)
 
-        assert_allclose(res1.params, res2.params, rtol=1e-13)
-        # bug TODO res1.scale missing ?  in Gaussian/OLS
-        assert_allclose(res1.bse, res2.bse, rtol=1e-13)
-#         if not self.cov_type == 'nonrobust':
-#             assert_allclose(res1.bse * res1.scale, res2.bse, rtol=1e-13)
-#         else:
-#             assert_allclose(res1.bse, res2.bse, rtol=1e-13)
+        rtol = getattr(res1, 'rtol', 1e-13)
+        assert_allclose(res1.params, res2.params, rtol=rtol)
+        assert_allclose(res1.bse, res2.bse, rtol=1e-10)
 
 
 class TestGLMLogit(CheckDiscreteGLM):
@@ -502,19 +498,32 @@ def setup_class(cls):
         cls.res2 = mod1.fit(cov_type='cluster', cov_kwds=dict(groups=group))
 
 
-class T_estGLMProbit(CheckDiscreteGLM):
-    # invalid link. What's Probit as GLM?
+class TestGLMProbit(CheckDiscreteGLM):
 
     @classmethod
     def setup_class(cls):
         endog_bin = (endog > endog.mean()).astype(int)
         cls.cov_type = 'cluster'
 
-        mod1 = GLM(endog_bin, exog, family=families.Gaussian(link=links.CDFLink()))
-        cls.res1 = mod1.fit(cov_type='cluster', cov_kwds=dict(groups=group))
+        mod1 = GLM(endog_bin, exog, family=families.Binomial(link=links.probit()))
+        cls.res1 = mod1.fit(method='newton',
+                            cov_type='cluster', cov_kwds=dict(groups=group))
 
         mod1 = smd.Probit(endog_bin, exog)
         cls.res2 = mod1.fit(cov_type='cluster', cov_kwds=dict(groups=group))
+        cls.rtol = 1e-6
+
+    def test_score_hessian(self):
+        res1 = self.res1
+        res2 = self.res2
+        # Note scale is fixed at 1, so we don't need to fix it explicitly
+        score1 = res1.model.score(res1.params * 0.98)
+        score2 = res2.model.score(res1.params * 0.98)
+        assert_allclose(score1, score2, rtol=1e-13)
+
+        hess1 = res1.model.hessian(res1.params)
+        hess2 = res2.model.hessian(res1.params)
+        assert_allclose(hess1, hess2, rtol=1e-10)
 
 
 class TestGLMGaussNonRobust(CheckDiscreteGLM):
diff --git a/statsmodels/genmod/tests/test_constrained.py b/statsmodels/genmod/tests/test_constrained.py
--- a/statsmodels/genmod/tests/test_constrained.py
+++ b/statsmodels/genmod/tests/test_constrained.py
@@ -55,7 +55,7 @@ def test_se(self):
         assert_allclose(res1.scale, res2.scale, rtol=1e-10)
         assert_allclose(res1.bse[self.idx_p_uc], res2.bse, rtol=1e-10)
         assert_allclose(res1.cov_params()[self.idx_p_uc[:, None],
-                        self.idx_p_uc], res2.cov_params(), rtol=1e-10, atol=1e-15)
+                        self.idx_p_uc], res2.cov_params(), rtol=5e-9, atol=1e-15)
 
     def test_resid(self):
         res1 = self.res1
diff --git a/statsmodels/genmod/tests/test_glm.py b/statsmodels/genmod/tests/test_glm.py
--- a/statsmodels/genmod/tests/test_glm.py
+++ b/statsmodels/genmod/tests/test_glm.py
@@ -16,6 +16,8 @@
 from statsmodels.tools.sm_exceptions import PerfectSeparationError
 from statsmodels.discrete import discrete_model as discrete
 from statsmodels.tools.sm_exceptions import DomainWarning
+from statsmodels.tools.numdiff import (approx_fprime, approx_fprime_cs,
+                                       approx_hess)
 import pytest
 import warnings
 
@@ -987,6 +989,27 @@ def test_summary():
         rslt = model.fit(method=method)
         s = rslt.summary()
 
+
+def check_score_hessian(results):
+    # compare models core and hessian with numerical derivatives
+
+    params = results.params
+    # avoid checking score at MLE, score close to zero
+    sc = results.model.score(params * 0.98, scale=1)
+    # cs currently (0.9) does not work for all families
+    # sc2 = approx_fprime_cs(params * 0.98, results.model.loglike)
+    llfunc = lambda x: results.model.loglike(x, scale=1)
+    sc2 = approx_fprime(params * 0.98, llfunc)
+    assert_allclose(sc, sc2, rtol=0.05)
+
+    hess = results.model.hessian(params, scale=1)
+    hess2 = approx_hess(params, llfunc)
+    assert_allclose(hess, hess2, rtol=0.05)
+    scfunc = lambda x: results.model.score(x, scale=1)
+    hess3 = approx_fprime(params, scfunc)
+    assert_allclose(hess, hess3, rtol=0.05)
+
+
 def test_gradient_irls():
     # Compare the results when using gradient optimization and IRLS.
 
@@ -1045,6 +1068,9 @@ def test_gradient_irls():
                elif (family_class, link) == (fam.Gaussian, lnk.inverse_power):
                    # adding skip because of convergence failure
                    skip_one = True
+               # the following fails with identity link, because endog < 0
+               # elif family_class == fam.Gamma:
+               #     lin_pred = 0.5 * exog.sum(1) + np.random.uniform(size=exog.shape[0])
                else:
                    lin_pred = np.random.uniform(size=exog.shape[0])
 
@@ -1055,6 +1081,12 @@ def test_gradient_irls():
                    mod_irls = sm.GLM(endog, exog, family=family_class(link=link()))
                rslt_irls = mod_irls.fit(method="IRLS")
 
+               if not (family_class, link) in [(fam.Poisson, lnk.sqrt),
+                                               (fam.Gamma, lnk.inverse_power),
+                                               (fam.InverseGaussian, lnk.identity)
+                                               ]:
+                   check_score_hessian(rslt_irls)
+
                # Try with and without starting values.
                for max_start_irls, start_params in (0, rslt_irls.params), (3, None):
                    # TODO: skip convergence failures for now
@@ -1065,7 +1097,7 @@ def test_gradient_irls():
                        mod_gradient = sm.GLM(endog, exog, family=family_class(link=link()))
                    rslt_gradient = mod_gradient.fit(max_start_irls=max_start_irls,
                                                     start_params=start_params,
-                                                    method="newton")
+                                                    method="newton", maxiter=300)
 
                    assert_allclose(rslt_gradient.params,
                                    rslt_irls.params, rtol=1e-6, atol=5e-5)
@@ -1083,6 +1115,8 @@ def test_gradient_irls():
                    assert_allclose(gradient_bse, rslt_irls.bse, rtol=1e-6, atol=5e-5)
 
 
+
+
 def test_gradient_irls_eim():
     # Compare the results when using eime gradient optimization and IRLS.