Repository: statsmodels/statsmodels

Issue:
BUG/ENH: poisson deviance
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/families/family.py#L372
Should the deviance function of the poisson distribution have the term `- (endog - mu)` added inside the sum, cf. function `resid_dev`? If the model does not contain an intercept term `np.sum(endog - mu)` might not be zero.
```python
return 2 * np.sum(freq_weights * (endog * np.log(endog_mu) - (endog - mu))) / scale
```


Discussion:
Yes, looks like a bug to me for the no-intercept case. (I'm adding corner-case label to signal that this bug only affects "uncommon" use cases.)

We don't use deviance in any optimization, so redundant calculation doesn't make much difference in performance.
I think I was the culprit who introduced this when I added `freq_weights`.

For my education, can one of you 2 explain this problem further? I was always under the impression that when you fit a poisson model `np.sum(endog - mu)` approaches zero. I'm aware this behavior is specific to Poisson (or maybe more specifically to the related variance function). 

Additionally, what is the exact drawback of having `np.sum(endog - mu) != 0`? Does this bias some statistical test (only Chi2 comes to mind as something that relies on deviance)? 

Thanks! 
@thequackdaddy 

It is or can be nonzero if no constant and no implicit constant is included. 
It is a property of all GLM/LEF models, including OLS that the sum of residuals is zero *if* a constant is included.

**edit** Most likely it's true only for canonical links. I didn't check the math for non-canonical links but I guess it would be a weighted average instead of sum.

However, we don't rule out the case that a user specifies a model without a constant. In the OLS case this would correspond to a regression through the zero point.

The match between sum(endog) and sum(fittedvalues) also occurs for subgroups with categorical dummy variables, e.g. in contingency tables.
related when I looked at this recently for Logit
https://mail.python.org/pipermail/scikit-learn/2016-December/001010.html

When the constant is penalized, then it cannot force `sum(endog) = sum(fittedvalues)`.

This is an old behavior, the term is missing also in 0.6, and not related to freq_weights addition



Resolving patch:
diff --git a/statsmodels/genmod/families/family.py b/statsmodels/genmod/families/family.py
--- a/statsmodels/genmod/families/family.py
+++ b/statsmodels/genmod/families/family.py
@@ -380,7 +380,9 @@ def deviance(self, endog, mu, freq_weights=1., scale=1.):
            D = 2 * \sum_i (freq\_weights_i * Y_i * \log(Y_i / \mu_i))/ scale
         '''
         endog_mu = self._clean(endog / mu)
-        return 2 * np.sum(endog * freq_weights * np.log(endog_mu)) / scale
+        return 2 * np.sum(freq_weights * (endog * np.log(endog_mu) -
+                                          (endog - mu))) / scale
+
 
     def loglike(self, endog, mu, freq_weights=1., scale=1.):
         r"""


Test patch:
diff --git a/statsmodels/discrete/tests/test_constrained.py b/statsmodels/discrete/tests/test_constrained.py
--- a/statsmodels/discrete/tests/test_constrained.py
+++ b/statsmodels/discrete/tests/test_constrained.py
@@ -338,9 +338,10 @@ def setup_class(cls):
 
         constr = 'C(agecat)[T.4] = C(agecat)[T.5]'
         lc = patsy.DesignInfo(mod.exog_names).linear_constraint(constr)
-        cls.res1 = fit_constrained(mod, lc.coefs, lc.constants)
+        cls.res1 = fit_constrained(mod, lc.coefs, lc.constants,
+                                   fit_kwds={'atol': 1e-10})
         cls.constraints = lc
-        cls.res1m = mod.fit_constrained(constr)
+        cls.res1m = mod.fit_constrained(constr, atol=1e-10)
 
 
 class TestGLMPoissonConstrained1b(CheckPoissonConstrainedMixin):
@@ -386,7 +387,7 @@ def test_compare_glm_poisson(self):
 
         # basic, just as check that we have the same model
         assert_allclose(res1.params, res2.params, rtol=1e-12)
-        assert_allclose(res1.bse, res2.bse, rtol=1e-12)
+        assert_allclose(res1.bse, res2.bse, rtol=1e-11)
 
         # check predict, fitted, ...
 
diff --git a/statsmodels/genmod/tests/test_glm.py b/statsmodels/genmod/tests/test_glm.py
--- a/statsmodels/genmod/tests/test_glm.py
+++ b/statsmodels/genmod/tests/test_glm.py
@@ -1826,6 +1826,39 @@ def test_convergence_atol_rtol_params(self):
                      actual_iterations)
 
 
+def test_poisson_deviance():
+    # see #3355 missing term in deviance if resid_response.sum() != 0
+    np.random.seed(123987)
+    nobs, k_vars = 50, 3-1
+    x = sm.add_constant(np.random.randn(nobs, k_vars))
+
+    mu_true = np.exp(x.sum(1))
+    y = np.random.poisson(mu_true, size=nobs)
+
+    mod = sm.GLM(y, x[:, :], family=sm.genmod.families.Poisson())
+    res = mod.fit()
+
+    d_i = res.resid_deviance
+    d = res.deviance
+    lr = (mod.family.loglike(y, y+1e-20) -
+          mod.family.loglike(y, res.fittedvalues)) * 2
+
+    assert_allclose(d, (d_i**2).sum(), rtol=1e-12)
+    assert_allclose(d, lr, rtol=1e-12)
+
+    # case without constant, resid_response.sum() != 0
+    mod_nc = sm.GLM(y, x[:, 1:], family=sm.genmod.families.Poisson())
+    res_nc = mod_nc.fit()
+
+    d_i = res_nc.resid_deviance
+    d = res_nc.deviance
+    lr = (mod.family.loglike(y, y+1e-20) -
+          mod.family.loglike(y, res_nc.fittedvalues)) * 2
+
+    assert_allclose(d, (d_i**2).sum(), rtol=1e-12)
+    assert_allclose(d, lr, rtol=1e-12)
+
+
 if __name__ == "__main__":
     # run_module_suite()
     # taken from Fernando Perez: