Repository: statsmodels/statsmodels

Issue:
BUG/TST: resid_deviance, GEE, GLM, family
https://github.com/statsmodels/statsmodels/issues/4208#issuecomment-357811452

https://github.com/statsmodels/statsmodels/commit/6b8a24b68f8b1ade2dcba7740d7f96a4e2ca6739#diff-20016798d17fceba2ff1d3dfec9f2e61
has most likely introduced a bug in GEEResults.resid_deviance by changing the behavior of the family resid method. GLMResults has a compensating change but not GEEResults.

AFAIR, there are no unit tests for GEEResults resid_xxx. Those could be compared to the GLM counterparts by using independence covariance structure.

families don't have unit tests by themselves

Needs checking before 0.9


Discussion:
Bug for resid_deviance in GEE confirmed, other resid are fine:

```
In [14]:
for resid in ['resid_response', 'resid_pearson', 'resid_deviance', 'resid_working', 'resid_anscombe']:
    r1 = getattr(rslt1, resid)
    r2 = getattr(rslt2, resid)
    print(resid, np.max(np.abs(r1 - r2)))

resid_response 1.88737914186e-14
resid_pearson 4.46309655899e-14
resid_deviance 2.81714847623
resid_working 1.31894495325e-13
resid_anscombe 3.60822483003e-14
```

based on test case in test_gee.py (using GLM instead of discrete logit)
```
    #def test_compare_logit(self):

vs = Independence()
family = families.Binomial()

Y = 1 * (np.random.normal(size=100) < 0)
X1 = np.random.normal(size=100)
X2 = np.random.normal(size=100)
X3 = np.random.normal(size=100)
groups = np.random.randint(0, 4, size=100)

D = pd.DataFrame({"Y": Y, "X1": X1, "X2": X2, "X3": X3})

mod1 = GEE.from_formula("Y ~ X1 + X2 + X3", groups, D,
                        family=family, cov_struct=vs)
rslt1 = mod1.fit()

mod2 = GLM.from_formula("Y ~ X1 + X2 + X3", data=D, family=family)
rslt2 = mod2.fit(disp=False)

assert_almost_equal(rslt1.params.values, rslt2.params.values,
                    decimal=10)
```

the values agree using directly the changed part that is now in GLM

```
In [16]:
np.max(np.abs(np.sign(rslt1.resid_response) * np.sqrt(rslt1.resid_deviance) - rslt2.resid_deviance))

Out[16]:
3.2640556923979602e-14
```
suggested fix as mentioned in comment in #4208, which still requires minimal code duplication

- rename family specific resid_dev to have leading underscore
- change `resid_dev` in family superclass to handle the generic sign sqrt var_weights and scale.
- delete the corresponding part again from GLM resid_deviance
- minimal adjustment to docs (no full templating for now) generic Family.resid_dev should mention that details are in family specific `_resid_dev`

This should make family resid_dev backwards compatible with 0.8 and fix the GEE bug.

Resolving patch:
diff --git a/statsmodels/genmod/families/family.py b/statsmodels/genmod/families/family.py
--- a/statsmodels/genmod/families/family.py
+++ b/statsmodels/genmod/families/family.py
@@ -161,8 +161,8 @@ def deviance(self, endog, mu, var_weights=1., freq_weights=1., scale=1.):
 
         .. math::
 
-           D = 2\sum_i (freq\_weights_i * (llf(endog_i, endog_i) -
-               llf(endog_i, \mu_i)))
+           D = 2\sum_i (freq\_weights_i * var\_weights *
+           (llf(endog_i, endog_i) - llf(endog_i, \mu_i)))
 
         where y is the endogenous variable. The deviance functions are
         analytically defined for each family.
@@ -170,23 +170,25 @@ def deviance(self, endog, mu, var_weights=1., freq_weights=1., scale=1.):
         Internally, we calculate deviance as:
 
         .. math::
-           D = \sum_i freq\_weights_i * resid\_dev_i  / scale
+           D = \sum_i freq\_weights_i * var\_weights * resid\_dev_i  / scale
         """
-        resid_dev = self.resid_dev(endog, mu, var_weights)
-        return np.sum(resid_dev * freq_weights / scale)
+        resid_dev = self._resid_dev(endog, mu)
+        return np.sum(resid_dev * freq_weights * var_weights / scale)
 
-    def resid_dev(self, endog, mu, var_weights=1.):
+    def resid_dev(self, endog, mu, var_weights=1., scale=1.):
         r"""
         The deviance residuals
 
         Parameters
         ----------
-        endog : array
-            The endogenous response variable.
-        mu : array
+        endog : array-like
+            The endogenous response variable
+        mu : array-like
             The inverse of the link function at the linear predicted values.
-        var_weights : array
+        var_weights : array-like
             1d array of variance (analytic) weights. The default is 1.
+        scale : float, optional
+            An optional scale argument. The default is 1.
 
         Returns
         -------
@@ -200,8 +202,15 @@ def resid_dev(self, endog, mu, var_weights=1.):
 
         .. math::
            resid\_dev_i = sign(y_i-\mu_i) \sqrt{D_i}
+           
+        D_i is calculated from the _resid_dev method in each family.
+        Distribution-specific documentation of the calculation is available
+        there.
         """
-        raise NotImplementedError
+        resid_dev = self._resid_dev(endog, mu)
+        resid_dev *= var_weights / scale
+        return np.sign(endog - mu) * np.sqrt(np.clip(resid_dev, 0., np.inf))
+
 
     def fitted(self, lin_pred):
         r"""
@@ -391,7 +400,7 @@ def __init__(self, link=None):
             link = L.log()
         super(Poisson, self).__init__(link=link, variance=Poisson.variance)
 
-    def resid_dev(self, endog, mu, var_weights=1.):
+    def _resid_dev(self, endog, mu):
         r"""
         Poisson deviance residuals
 
@@ -401,8 +410,6 @@ def resid_dev(self, endog, mu, var_weights=1.):
             The endogenous response variable.
         mu : array
             The inverse of the link function at the linear predicted values.
-        var_weights : array
-            1d array of variance (analytic) weights. The default is 1.
 
         Returns
         -------
@@ -413,12 +420,12 @@ def resid_dev(self, endog, mu, var_weights=1.):
         -----
         .. math::
 
-           resid\_dev_i = 2 * var\_weights_i * (endog_i * \ln(endog_i /
-           \mu_i) - (endog_i - \mu_i))
+           resid\_dev_i = 2 * (endog_i * \ln(endog_i / \mu_i) -
+           (endog_i - \mu_i))
         """
         endog_mu = self._clean(endog / mu)
         resid_dev = endog * np.log(endog_mu) - (endog - mu)
-        return 2 * var_weights * resid_dev
+        return 2 * resid_dev
 
     def loglike_obs(self, endog, mu, var_weights=1., scale=1.):
         r"""
@@ -519,7 +526,7 @@ def __init__(self, link=None):
             link = L.identity()
         super(Gaussian, self).__init__(link=link, variance=Gaussian.variance)
 
-    def resid_dev(self, endog, mu, var_weights=1.):
+    def _resid_dev(self, endog, mu):
         r"""
         Gaussian deviance residuals
 
@@ -529,8 +536,6 @@ def resid_dev(self, endog, mu, var_weights=1.):
             The endogenous response variable.
         mu : array
             The inverse of the link function at the linear predicted values.
-        var_weights : array
-            1d array of variance (analytic) weights. The default is 1.
 
         Returns
         -------
@@ -541,9 +546,9 @@ def resid_dev(self, endog, mu, var_weights=1.):
         --------
         .. math::
 
-           resid\_dev_i = var\_weights_i * (endog_i - \mu_i) ** 2
+           resid\_dev_i = (endog_i - \mu_i) ** 2
         """
-        return var_weights * (endog - mu) ** 2
+        return (endog - mu) ** 2
 
     def loglike_obs(self, endog, mu, var_weights=1., scale=1.):
         r"""
@@ -664,7 +669,7 @@ def __init__(self, link=None):
             link = L.inverse_power()
         super(Gamma, self).__init__(link=link, variance=Gamma.variance)
 
-    def resid_dev(self, endog, mu, var_weights=1.):
+    def _resid_dev(self, endog, mu):
         r"""
         Gamma deviance residuals
 
@@ -674,8 +679,6 @@ def resid_dev(self, endog, mu, var_weights=1.):
             The endogenous response variable.
         mu : array
             The inverse of the link function at the linear predicted values.
-        var_weights : array
-            1d array of variance (analytic) weights. The default is 1.
 
         Returns
         -------
@@ -686,12 +689,12 @@ def resid_dev(self, endog, mu, var_weights=1.):
         -----
         .. math::
 
-           resid\_dev_i = 2 * var\_weights_i * ((endog_i - \mu_i) / \mu_i -
+           resid\_dev_i = 2 * ((endog_i - \mu_i) / \mu_i -
            \log(endog_i / \mu_i))
         """
         endog_mu = self._clean(endog / mu)
         resid_dev = -np.log(endog_mu) + (endog - mu) / mu
-        return 2 * var_weights * resid_dev
+        return 2 * resid_dev
 
     def loglike_obs(self, endog, mu, var_weights=1., scale=1.):
         r"""
@@ -850,7 +853,7 @@ def initialize(self, endog, freq_weights):
         else:
             return endog, np.ones(endog.shape[0])
 
-    def resid_dev(self, endog, mu, var_weights=1.):
+    def _resid_dev(self, endog, mu):
         r"""
         Binomial deviance residuals
 
@@ -860,8 +863,6 @@ def resid_dev(self, endog, mu, var_weights=1.):
             The endogenous response variable.
         mu : array
             The inverse of the link function at the linear predicted values.
-        var_weights : array
-            1d array of variance (analytic) weights. The default is 1.
 
         Returns
         -------
@@ -872,13 +873,13 @@ def resid_dev(self, endog, mu, var_weights=1.):
         -----
         .. math::
 
-           resid\_dev_i = 2 * var\_weights * n * (endog_i * \ln(endog_i /
-           \mu_i) + (1 - endog_i) * \ln((1 - endog_i) / (1 - \mu_i)))
+           resid\_dev_i = 2 * n * (endog_i * \ln(endog_i /\mu_i) +
+           (1 - endog_i) * \ln((1 - endog_i) / (1 - \mu_i)))
         """
         endog_mu = self._clean(endog / mu)
         n_endog_mu = self._clean((1. - endog) / (1. - mu))
         resid_dev = endog * np.log(endog_mu) + (1 - endog) * np.log(n_endog_mu)
-        return 2 * var_weights * self.n * resid_dev
+        return 2 * self.n * resid_dev
 
     def loglike_obs(self, endog, mu, var_weights=1., scale=1.):
         r"""
@@ -1037,7 +1038,7 @@ def __init__(self, link=None):
         super(InverseGaussian, self).__init__(
             link=link, variance=InverseGaussian.variance)
 
-    def resid_dev(self, endog, mu, var_weights=1.):
+    def _resid_dev(self, endog, mu):
         r"""
         Inverse Gaussian deviance residuals
 
@@ -1047,8 +1048,6 @@ def resid_dev(self, endog, mu, var_weights=1.):
             The endogenous response variable.
         mu : array
             The inverse of the link function at the linear predicted values.
-        var_weights : array
-            1d array of variance (analytic) weights. The default is 1.
 
         Returns
         -------
@@ -1059,10 +1058,9 @@ def resid_dev(self, endog, mu, var_weights=1.):
         -----
         .. math::
 
-           resid\_dev_i = var\_weights_i / (endog_i * \mu_i^2) * (endog_i -
-                          \mu_i)^2
+           resid\_dev_i = 1 / (endog_i * \mu_i^2) * (endog_i - \mu_i)^2
         """
-        return var_weights / (endog * mu ** 2) * (endog - mu) ** 2
+        return 1. / (endog * mu ** 2) * (endog - mu) ** 2
 
     def loglike_obs(self, endog, mu, var_weights=1., scale=1.):
         r"""
@@ -1187,7 +1185,7 @@ def __init__(self, link=None, alpha=1.):
         super(NegativeBinomial, self).__init__(
             link=link, variance=V.NegativeBinomial(alpha=self.alpha))
 
-    def resid_dev(self, endog, mu, var_weights=1.):
+    def _resid_dev(self, endog, mu):
         r"""
         Negative Binomial deviance residuals
 
@@ -1197,8 +1195,6 @@ def resid_dev(self, endog, mu, var_weights=1.):
             The endogenous response variable.
         mu : array
             The inverse of the link function at the linear predicted values.
-        var_weights : array
-            1d array of variance (analytic) weights. The default is 1.
 
         Returns
         -------
@@ -1209,7 +1205,7 @@ def resid_dev(self, endog, mu, var_weights=1.):
         -----
         .. math::
 
-            resid_dev_i = 2 * var\_weights_i * (endog_i * \ln(endog_i /
+            resid_dev_i = 2 * (endog_i * \ln(endog_i /
             \mu_i) - (endog_i + 1 / \alpha) * \ln((endog_i + 1 / \alpha) /
             (\mu_i + 1 / \alpha)))
         """
@@ -1218,7 +1214,7 @@ def resid_dev(self, endog, mu, var_weights=1.):
         mu_alpha = mu + 1 / self.alpha
         resid_dev = endog * np.log(endog_mu)
         resid_dev -= endog_alpha * np.log(endog_alpha / mu_alpha)
-        return 2 * var_weights * resid_dev
+        return 2 * resid_dev
 
     def loglike_obs(self, endog, mu, var_weights=1., scale=1.):
         r"""
@@ -1365,7 +1361,7 @@ def __init__(self, link=None, var_power=1.):
         super(Tweedie, self).__init__(
             link=link, variance=V.Power(power=var_power * 1.))
 
-    def resid_dev(self, endog, mu, var_weights=1.):
+    def _resid_dev(self, endog, mu):
         r"""
         Tweedie deviance residuals
 
@@ -1375,8 +1371,6 @@ def resid_dev(self, endog, mu, var_weights=1.):
             The endogenous response variable.
         mu : array
             The inverse of the link function at the linear predicted values.
-        var_weights : array
-            1d array of variance (analytic) weights. The default is 1.
 
         Returns
         -------
@@ -1417,7 +1411,7 @@ def resid_dev(self, endog, mu, var_weights=1.):
 
         .. math::
 
-            resid\_dev_i = 2 * var\_weights_i * dev_i
+            resid\_dev_i = 2 * dev_i
         """
         p = self.var_power
         if p == 1:
@@ -1430,7 +1424,7 @@ def resid_dev(self, endog, mu, var_weights=1.):
         else:
             dev = (endog ** (2 - p) / ((1 - p) * (2 - p)) -
                    endog * mu ** (1-p) / (1 - p) + mu ** (2 - p) / (2 - p))
-        return 2 * dev * var_weights
+        return 2 * dev
 
     def loglike_obs(self, endog, mu, var_weights=1., scale=1.):
         r"""
diff --git a/statsmodels/genmod/generalized_linear_model.py b/statsmodels/genmod/generalized_linear_model.py
--- a/statsmodels/genmod/generalized_linear_model.py
+++ b/statsmodels/genmod/generalized_linear_model.py
@@ -1504,8 +1504,9 @@ def resid_anscombe_unscaled(self):
     @cache_readonly
     def resid_deviance(self):
         dev = self.family.resid_dev(self._endog, self.fittedvalues,
-                                    self._var_weights)
-        return np.sign(self._endog - self.fittedvalues) * np.sqrt(dev)
+                                    var_weights=self._var_weights,
+                                    scale=1.)
+        return dev
 
     @cache_readonly
     def pearson_chi2(self):


Test patch:
diff --git a/statsmodels/genmod/tests/test_gee_glm.py b/statsmodels/genmod/tests/test_gee_glm.py
new file mode 100644
--- /dev/null
+++ b/statsmodels/genmod/tests/test_gee_glm.py
@@ -0,0 +1,130 @@
+import numpy as np
+import pandas as pd
+
+
+from statsmodels.genmod.generalized_linear_model import GLM
+from statsmodels.genmod import families
+from statsmodels.genmod.families import links
+from statsmodels.genmod.generalized_estimating_equations import GEE
+from statsmodels.genmod.cov_struct import Independence
+
+from numpy.testing import assert_allclose
+
+
+class CheckGEEGLM(object):
+
+    def test_basic(self):
+        res1 = self.result1
+        res2 = self.result2
+        assert_allclose(res1.params.values, res2.params.values,
+                        rtol=1e-6, atol=1e-10)
+
+
+    def test_resid(self):
+        res1 = self.result1
+        res2 = self.result2
+        assert_allclose(res1.resid_response, res2.resid_response,
+                        rtol=1e-6, atol=1e-10)
+        assert_allclose(res1.resid_pearson, res2.resid_pearson,
+                        rtol=1e-6, atol=1e-10)
+        assert_allclose(res1.resid_deviance, res2.resid_deviance,
+                        rtol=1e-6, atol=1e-10)
+        assert_allclose(res1.resid_anscombe, res2.resid_anscombe,
+                        rtol=1e-6, atol=1e-10)
+        assert_allclose(res1.resid_working, res2.resid_working,
+                        rtol=1e-6, atol=1e-10)
+
+
+#def test_compare_logit(self):
+class TestCompareLogit(CheckGEEGLM):
+
+    @classmethod
+    def setup_class(cls):
+        vs = Independence()
+        family = families.Binomial()
+        np.random.seed(987126)
+        Y = 1 * (np.random.normal(size=100) < 0)
+        X1 = np.random.normal(size=100)
+        X2 = np.random.normal(size=100)
+        X3 = np.random.normal(size=100)
+        groups = np.random.randint(0, 4, size=100)
+
+        D = pd.DataFrame({"Y": Y, "X1": X1, "X2": X2, "X3": X3})
+
+        mod1 = GEE.from_formula("Y ~ X1 + X2 + X3", groups, D,
+                            family=family, cov_struct=vs)
+        cls.result1 = mod1.fit()
+
+        mod2 = GLM.from_formula("Y ~ X1 + X2 + X3", data=D, family=family)
+        cls.result2 = mod2.fit(disp=False)
+
+
+class TestComparePoisson(CheckGEEGLM):
+
+    @classmethod
+    def setup_class(cls):
+        vs = Independence()
+        family = families.Poisson()
+        np.random.seed(987126)
+        Y = np.exp(1 + np.random.normal(size=100))
+        X1 = np.random.normal(size=100)
+        X2 = np.random.normal(size=100)
+        X3 = np.random.normal(size=100)
+        groups = np.random.randint(0, 4, size=100)
+
+        D = pd.DataFrame({"Y": Y, "X1": X1, "X2": X2, "X3": X3})
+
+        mod1 = GEE.from_formula("Y ~ X1 + X2 + X3", groups, D,
+                            family=family, cov_struct=vs)
+        cls.result1 = mod1.fit()
+
+        mod2 = GLM.from_formula("Y ~ X1 + X2 + X3", data=D, family=family)
+        cls.result2 = mod2.fit(disp=False)
+
+
+class TestCompareGaussian(CheckGEEGLM):
+
+    @classmethod
+    def setup_class(cls):
+
+        vs = Independence()
+        family = families.Gaussian()
+        np.random.seed(987126)
+        Y = np.random.normal(size=100)
+        X1 = np.random.normal(size=100)
+        X2 = np.random.normal(size=100)
+        X3 = np.random.normal(size=100)
+        groups = np.kron(np.arange(20), np.ones(5))
+
+        D = pd.DataFrame({"Y": Y, "X1": X1, "X2": X2, "X3": X3})
+
+        md = GEE.from_formula("Y ~ X1 + X2 + X3", groups, D,
+                              family=family, cov_struct=vs)
+        cls.result1 = md.fit()
+
+        cls.result2 = GLM.from_formula("Y ~ X1 + X2 + X3", data=D).fit()
+
+
+class TestCompareGamma(CheckGEEGLM):
+
+    @classmethod
+    def setup_class(cls):
+        # adjusted for Gamma, not in test_gee.py
+        vs = Independence()
+        family = families.Gamma(link=links.log)
+        np.random.seed(987126)
+        #Y = np.random.normal(size=100)**2
+        Y = np.exp(0.1 + np.random.normal(size=100))   # log-normal
+        X1 = np.random.normal(size=100)
+        X2 = np.random.normal(size=100)
+        X3 = np.random.normal(size=100)
+        groups = np.random.randint(0, 4, size=100)
+
+        D = pd.DataFrame({"Y": Y, "X1": X1, "X2": X2, "X3": X3})
+
+        mod1 = GEE.from_formula("Y ~ X1 + X2 + X3", groups, D,
+                                family=family, cov_struct=vs)
+        cls.result1 = mod1.fit()
+
+        mod2 = GLM.from_formula("Y ~ X1 + X2 + X3", data=D, family=family)
+        cls.result2 = mod2.fit(disp=False)