Repository: statsmodels/statsmodels

Issue:
More numerical issues in genmod families
#2028 doesn't fix one of the issues I am having with GLM: if any endog == 0 for negative binomial regression, the residual deviances are NaN.  The changes as recently merged via #2028 effectively have a term `notzero * log(endog/mu)`, which is still NaN even when endog is zero.



Discussion:
A related issue, the shape parameter in a NegativeBinomial family should be converted to a float when saving as an attribute.

Right now, if I pass an integer greater than 1 into the NegativeBinomial family, e.g. alpha=2 (easy to do by accident), then in the arithmetic for loglike we have 1/self.alpha, so we get 0.  In particular if any endog == 0, we end up taking gammaln(endog + 1/alpha) = NaN, so I get a NaN value for llf (but we would have gotten the wrong llf in any case).

Is it possible to make a small example that fails?

I get NaN in the last line from the following example:

import numpy as np
import statsmodels.api as sm

n = 200
p = 3

exog = np.random.normal(size=(n,p))
exog[:, 0] = 1

lin_pred = 4 + exog[:, 1] + exog[:, 2]
expval = np.exp(lin_pred)

endog = np.random.poisson(expval)
endog[0] = 0

model = sm.GLM(endog, exog, family=sm.families.NegativeBinomial(alpha=1.))
result = model.fit()
print result.resid_deviance[0]

On Fri, Oct 10, 2014 at 12:00 AM, Skipper Seabold notifications@github.com
wrote:

> Is it possible to make a small example that fails?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/statsmodels/statsmodels/issues/2037#issuecomment-58609970
> .

family NegativeBinomial doesn't have 0log0 protection. But, AFAICS this only appears in resid_dev.

I guess this would mean that it is broken for all data with endog=0. Do we have unit tests?


Resolving patch:
diff --git a/statsmodels/genmod/families/family.py b/statsmodels/genmod/families/family.py
--- a/statsmodels/genmod/families/family.py
+++ b/statsmodels/genmod/families/family.py
@@ -1145,7 +1145,7 @@ class NegativeBinomial(Family):
     safe_links = [L.Log,]
 
     def __init__(self, link=L.log, alpha=1.):
-        self.alpha = alpha
+        self.alpha = 1. * alpha # make it at least float
         self.variance = V.NegativeBinomial(alpha=self.alpha)
         if isinstance(link, L.NegativeBinomial):
             self.link = link(alpha=self.alpha)
@@ -1202,7 +1202,6 @@ def deviance(self, endog, mu, scale=1.):
         """
         iszero = np.equal(endog, 0)
         notzero = 1 - iszero
-        tmp = np.zeros(len(endog))
         endog_mu = self._clean(endog/mu)
         tmp = iszero * 2 * np.log(1 + self.alpha * mu)/self.alpha
         tmp += notzero * (2 * endog * np.log(endog_mu) - 2/self.alpha *
@@ -1249,9 +1248,9 @@ def resid_dev(self, endog, mu, scale=1.):
         '''
         iszero = np.equal(endog, 0)
         notzero = 1 - iszero
-        tmp = np.zeros(len(endog))
+        endog_mu = self._clean(endog/mu)
         tmp = iszero * 2 * np.log(1 + self.alpha * mu)/self.alpha
-        tmp += notzero * (2 * endog * np.log(endog/mu) - 2/self.alpha *
+        tmp += notzero * (2 * endog * np.log(endog_mu) - 2/self.alpha *
                           (1 + self.alpha * endog) *
                           np.log((1 + self.alpha * endog) /
                                  (1 + self.alpha * mu)))


Test patch: