Repository: statsmodels/statsmodels

Issue:
Robust linear model: divide by zero error for perfect fits
If you run this python code:
```
import statsmodels.api as sm

x = [80, 70, 60, 50, 40, 30, 20, 10, 0]
y = [-4, -5, -6, -7, -8, -9, -10, -11, -12]

x_with_const = sm.add_constant(x)
rlm_model = sm.RLM(y, x_with_const, M=sm.robust.norms.TukeyBiweight())
rlm_results = rlm_model.fit()
```
the results is a divide by zero exception in robust_linear_model.py. This happens even if there are outliers -- as soon as the algorithm decides to remove them, the divide by zero comes back.
Eg the same code as above except for:
```
x = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
y = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,1000]
```
still results in the divide by zero error.

I'm running statsmodels 0.6.1.

I edited robust_linear_model.py to make the couple offending scale locations (_estimate_scale() and deviance()) have logic basically like:
```
if scale == 0:
   scale = sys.float_info.epsilon
```
and this fixed the issues, but this seems more like a workaround than a real fix.


Discussion:
I'm putting temporarily prio-high.

I tried to solve this problem some years ago, instead of fixing the symptoms. But then I got confused about the perfect fit policies or behaviors, and was not sure what to do with it. #1341
I was recently again reading books and articles that argue that the perfect fit property with variance=0 is a feature and not a problem, but I don't "like" it. It wasn't clear to me how to add user options for it.

**edit** perfect fit happens if at least half of the observations are on a line or subspace. This happens because we use MAD for the scale estimate. If we don't want a breakdown point of around one half, then the scale should be adjusted. But Huber scale also did not look very plausible.

The point is essentially to decide whether, for example, the x.5 y-values in this case are outliers or not:
x = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
y = [1,2,3,4,5,6,7,8,9,10,11,12, 12.5, 14.5, 15.5, 16.5]
or to decide whether the code should make a decision or should ask the user.

Perhaps the way I would expect this to show up is in the fit_history. If a perfect fit resulted in NaNs in the fit history, I could detect this and go back in the history to the most recent fit that worked for me. If the weights were also saved off in fit history, I could do some basic decision making based on what fraction of weights are non-zero. 

The main issue I'm having right now is the fit() blows up completely, so I don't even have access to the history to make a decision for myself.

I guess it would also be my preference would be for the perfect fit case to have an epsilon scale such that I would never see NaNs, but that's might be mostly for practical reasons for my particular use case.
@SamNPowers  What kind of data do you have to get perfect fit? And how many observations?

I think that saving the weights in the history will not be practical, because it looks too much like a memory leak, i.e. we are adding a full nobs length array in each iteration.

The main workaround for the perfect fit problem that I thought about is to add a tiny bit of noise to the data which would put the noise standard deviation as a lower bound on the scale.

There are currently some limitation on the scale options, and I just saw another bug with an unused scale option that would allow users to provide a scale function or callable.

From what I can see now (being 3 years older than in the previous round), we need to
- handle the perfect fit case with nans where needed but no exceptions as mostly done in my old PR
- fix and add options for the scale estimate, which would allow, for example, the user to specify `mad` with lower bound on the scale. I also worked in the meantime on other alternative scale estimators which would allow more flexibility. Those are not yet ready for merge.
The data I'm using to get a perfect fit is simulated data - we need to have a set of data with a well-known fit to use for testing some of our other logic. I considered adding noise to the data but that seemed like a hack around the issue.

I am very in favor of allowing the user to specify a lower bound on scale.

Thanks for being so responsive about this!
Thinking about it some more, I start to like the idea of adding an epsilon lower bound on the scale by default, that can be set to zero in a new fit keyword. This would be independent of the "proper" handling of the perfect fit case.

I guess the epsilon lower bound would be a interior solution that is numerically close to the full perfect fit solution.
@SamNPowers  Did all the numbers look reasonable with your epsilon perfect fit, or does it look like we need to adjust other parts?

I'm not planning to work on the full solution for some time, but I think it should be possible to quickly get some changes merged to avoid the current exceptions and give the user some options as workarounds.


BTW: I'm glad to get some feedback on this, because last time I either couldn't figure out what should be done or I couldn't make up my mind about which solution would be best. 
(And I spend large parts of the last 2 months reading again about outlier robust estimation although with a different focus, mainly on multivariate models.)
The two main scenarios I tested were: 1. a simple line, 2. a simple line with a couple outliers. Both of them gave me the robust fit parameters I expected with just my simple scale lower bound. It's not exactly a stress test, but it at least indicated to me that there weren't other issues.
original issue #55 (It's the 6th oldest issue, and resurfaces every few years. Some of the oldest issues have been fixed over time but not been closed.)
Do we have an estimate on when the code to avoid the current exceptions + give the user some workarounds might be in?

Resolving patch:
diff --git a/statsmodels/robust/norms.py b/statsmodels/robust/norms.py
--- a/statsmodels/robust/norms.py
+++ b/statsmodels/robust/norms.py
@@ -189,7 +189,7 @@ def _subset(self, z):
         Huber's T is defined piecewise over the range for z
         """
         z = np.asarray(z)
-        return np.less_equal(np.fabs(z), self.t)
+        return np.less_equal(np.abs(z), self.t)
 
     def rho(self, z):
         r"""
@@ -210,7 +210,7 @@ def rho(self, z):
         z = np.asarray(z)
         test = self._subset(z)
         return (test * 0.5 * z**2 +
-                (1 - test) * (np.fabs(z) * self.t - 0.5 * self.t**2))
+                (1 - test) * (np.abs(z) * self.t - 0.5 * self.t**2))
 
     def psi(self, z):
         r"""
@@ -254,7 +254,7 @@ def weights(self, z):
         """
         z = np.asarray(z)
         test = self._subset(z)
-        absz = np.fabs(z)
+        absz = np.abs(z)
         absz[test] = 1.0
         return test + (1 - test) * self.t / absz
 
@@ -266,10 +266,10 @@ def psi_deriv(self, z):
         -----
         Used to estimate the robust covariance matrix.
         """
-        return np.less_equal(np.fabs(z), self.t)
+        return np.less_equal(np.abs(z), self.t)
 
 
-#TODO: untested, but looks right.  RamsayE not available in R or SAS?
+# TODO: untested, but looks right.  RamsayE not available in R or SAS?
 class RamsayE(RobustNorm):
     """
     Ramsay's Ea for M estimation.
@@ -303,8 +303,8 @@ def rho(self, z):
             rho(z) = a**-2 * (1 - exp(-a*\|z\|)*(1 + a*\|z\|))
         """
         z = np.asarray(z)
-        return (1 - np.exp(-self.a * np.fabs(z)) *
-                (1 + self.a * np.fabs(z))) / self.a**2
+        return (1 - np.exp(-self.a * np.abs(z)) *
+                (1 + self.a * np.abs(z))) / self.a**2
 
     def psi(self, z):
         r"""
@@ -323,7 +323,7 @@ def psi(self, z):
             psi(z) = z*exp(-a*\|z\|)
         """
         z = np.asarray(z)
-        return z * np.exp(-self.a * np.fabs(z))
+        return z * np.exp(-self.a * np.abs(z))
 
     def weights(self, z):
         r"""
@@ -343,7 +343,7 @@ def weights(self, z):
         """
 
         z = np.asarray(z)
-        return np.exp(-self.a * np.fabs(z))
+        return np.exp(-self.a * np.abs(z))
 
     def psi_deriv(self, z):
         """
@@ -353,9 +353,12 @@ def psi_deriv(self, z):
         -----
         Used to estimate the robust covariance matrix.
         """
-
-        return np.exp(-self.a * np.fabs(z)) + z**2*\
-                np.exp(-self.a*np.fabs(z))*-self.a/np.fabs(z)
+        a = self.a
+        x = np.exp(-a * np.abs(z))
+        dx = -a * x * np.sign(z)
+        y = z
+        dy = 1
+        return x * dy + y * dx
 
 
 class AndrewWave(RobustNorm):
@@ -381,7 +384,7 @@ def _subset(self, z):
         Andrew's wave is defined piecewise over the range of z.
         """
         z = np.asarray(z)
-        return np.less_equal(np.fabs(z), self.a * np.pi)
+        return np.less_equal(np.abs(z), self.a * np.pi)
 
     def rho(self, z):
         r"""
@@ -451,7 +454,16 @@ def weights(self, z):
         a = self.a
         z = np.asarray(z)
         test = self._subset(z)
-        return test * np.sin(z / a) / (z / a)
+        ratio = z / a
+        small = np.abs(ratio) < np.finfo(np.double).eps
+        if np.any(small):
+            weights = np.ones_like(ratio)
+            large = ~small
+            ratio = ratio[large]
+            weights[large] = test[large] * np.sin(ratio) / ratio
+        else:
+            weights = test * np.sin(ratio) / ratio
+        return weights
 
     def psi_deriv(self, z):
         """
@@ -491,7 +503,7 @@ def _subset(self, z):
         """
 
         z = np.asarray(z)
-        return np.less_equal(np.fabs(z), self.c)
+        return np.less_equal(np.abs(z), self.c)
 
     def rho(self, z):
         r"""
@@ -599,7 +611,7 @@ def _subset(self, z):
         """
         Hampel's function is defined piecewise over the range of z
         """
-        z = np.fabs(np.asarray(z))
+        z = np.abs(np.asarray(z))
         t1 = np.less_equal(z, self.a)
         t2 = np.less_equal(z, self.b) * np.greater(z, self.a)
         t3 = np.less_equal(z, self.c) * np.greater(z, self.b)
@@ -626,7 +638,7 @@ def rho(self, z):
             rho(z) = a*(b + c - a)                  for \|z\| > c
         """
 
-        z = np.fabs(z)
+        z = np.abs(z)
         a = self.a
         b = self.b
         c = self.c
@@ -665,7 +677,7 @@ def psi(self, z):
         c = self.c
         t1, t2, t3 = self._subset(z)
         s = np.sign(z)
-        z = np.fabs(z)
+        z = np.abs(z)
         v = s * (t1 * z +
                  t2 * a*s +
                  t3 * a*s * (c - z) / (c - b))
@@ -699,15 +711,25 @@ def weights(self, z):
         b = self.b
         c = self.c
         t1, t2, t3 = self._subset(z)
-        v = (t1 +
-            t2 * a/np.fabs(z) +
-            t3 * a*(c-np.fabs(z))/(np.fabs(z)*(c-b)))
-        v[np.where(np.isnan(v))]=1. # for some reason 0 returns a nan?
+
+        v = np.zeros_like(z)
+        v[t1] = 1.0
+        abs_z = np.abs(z)
+        v[t2] = a / abs_z[t2]
+        abs_zt3 = abs_z[t3]
+        v[t3] = a * (c - abs_zt3) / (abs_zt3 * (c - b))
+        v[np.where(np.isnan(v))] = 1.  # for some reason 0 returns a nan?
         return v
 
     def psi_deriv(self, z):
         t1, t2, t3 = self._subset(z)
-        return t1 + t3 * (self.a*np.sign(z)*z)/(np.fabs(z)*(self.c-self.b))
+        a, b, c = self.a, self.b, self.c
+        # default is t1
+        d = np.zeros_like(z)
+        d[t1] = 1.0
+        zt3 = z[t3]
+        d[t3] = (a * np.sign(zt3) * zt3) / (np.abs(zt3) * (c - b))
+        return d
 
 
 class TukeyBiweight(RobustNorm):
@@ -733,7 +755,7 @@ def _subset(self, z):
         """
         Tukey's biweight is defined piecewise over the range of z
         """
-        z = np.fabs(np.asarray(z))
+        z = np.abs(np.asarray(z))
         return np.less_equal(z, self.c)
 
     def rho(self, z):
@@ -856,7 +878,7 @@ def estimate_location(a, scale, norm=None, axis=0, initial=None,
     for iter in range(maxiter):
         W = norm.weights((a-mu)/scale)
         nmu = np.sum(W*a, axis) / np.sum(W, axis)
-        if np.alltrue(np.less(np.fabs(mu - nmu), scale * tol)):
+        if np.alltrue(np.less(np.abs(mu - nmu), scale * tol)):
             return nmu
         else:
             mu = nmu
diff --git a/statsmodels/robust/robust_linear_model.py b/statsmodels/robust/robust_linear_model.py
--- a/statsmodels/robust/robust_linear_model.py
+++ b/statsmodels/robust/robust_linear_model.py
@@ -17,6 +17,7 @@
 import scipy.stats as stats
 
 from statsmodels.tools.decorators import cache_readonly
+from statsmodels.tools.sm_exceptions import ConvergenceWarning
 import statsmodels.regression.linear_model as lm
 import statsmodels.regression._tools as reg_tools
 import statsmodels.robust.norms as norms
@@ -28,7 +29,7 @@
 
 
 def _check_convergence(criterion, iteration, tol, maxiter):
-    cond = np.fabs(criterion[iteration] - criterion[iteration - 1])
+    cond = np.abs(criterion[iteration] - criterion[iteration - 1])
     return not (np.any(cond > tol) and iteration < maxiter)
 
 
@@ -140,7 +141,7 @@ def predict(self, params, exog=None):
 
         Parameters
         ----------
-        params : array_like, optional after fit has been called
+        params : array_like
             Parameters of a linear model
         exog : array_like, optional.
             Design / exogenous data. Model exog is used if None.
@@ -148,10 +149,6 @@ def predict(self, params, exog=None):
         Returns
         -------
         An array of fitted values
-
-        Notes
-        -----
-        If the model as not yet been fit, params is not optional.
         """
         # copied from linear_model
         if exog is None:
@@ -240,8 +237,8 @@ def fit(self, maxiter=50, tol=1e-8, scale_est='mad', init=None, cov='H1',
 
         Returns
         -------
-        results : object
-            statsmodels.rlm.RLMresults
+        results : statsmodels.rlm.RLMresults
+            Results instance
         """
         if cov.upper() not in ["H1", "H2", "H3"]:
             raise ValueError("Covariance matrix %s not understood" % cov)
@@ -286,6 +283,12 @@ def fit(self, maxiter=50, tol=1e-8, scale_est='mad', init=None, cov='H1',
         iteration = 1
         converged = 0
         while not converged:
+            if self.scale == 0.0:
+                import warnings
+                warnings.warn('Estimated scale is 0.0 indicating that the most'
+                              ' last iteration produced a perfect fit of the '
+                              'weighted data.', ConvergenceWarning)
+                break
             self.weights = self.M.weights(wls_results.resid / self.scale)
             wls_results = reg_tools._MinimalWLS(self.endog, self.exog,
                                                 weights=self.weights,
@@ -347,7 +350,7 @@ class RLMResults(base.LikelihoodModelResults):
         errors are taken from the robust covariance matrix specified in the
         argument to fit.
     chisq : array
-        An array of the chi-squared values of the paramter estimates.
+        An array of the chi-squared values of the parameter estimates.
     df_model
         See RLM.df_model
     df_resid
@@ -419,6 +422,10 @@ def resid(self):
 
     @cache_readonly
     def sresid(self):
+        if self.scale == 0.0:
+            sresid = self.resid.copy()
+            sresid[:] = 0.0
+            return sresid
         return self.resid / self.scale
 
     @cache_readonly
diff --git a/statsmodels/robust/scale.py b/statsmodels/robust/scale.py
--- a/statsmodels/robust/scale.py
+++ b/statsmodels/robust/scale.py
@@ -42,7 +42,7 @@ def mad(a, c=Gaussian.ppf(3/4.), axis=0, center=np.median):
     a = np.asarray(a)
     if callable(center):
         center = np.apply_over_axes(center, a, axis)
-    return np.median((np.fabs(a-center))/c, axis=axis)
+    return np.median((np.abs(a-center))/c, axis=axis)
 
 
 class Huber(object):
@@ -157,16 +157,18 @@ def _estimate_both(self, a, scale, mu, axis, est_mu, n):
                 nmu = mu.squeeze()
             nmu = tools.unsqueeze(nmu, axis, a.shape)
 
-            subset = np.less_equal(np.fabs((a - mu)/scale), self.c)
+            subset = np.less_equal(np.abs((a - mu)/scale), self.c)
             card = subset.sum(axis)
 
-            nscale = np.sqrt(np.sum(subset * (a - nmu)**2, axis) \
-                    / (n * self.gamma - (a.shape[axis] - card) * self.c**2))
+            nscale = np.sqrt(np.sum(subset * (a - nmu) ** 2, axis)
+                             / (n * self.gamma -
+                                (a.shape[axis] - card) * self.c ** 2))
             nscale = tools.unsqueeze(nscale, axis, a.shape)
 
-            test1 = np.alltrue(np.less_equal(np.fabs(scale - nscale),
-                        nscale * self.tol))
-            test2 = np.alltrue(np.less_equal(np.fabs(mu - nmu), nscale*self.tol))
+            test1 = np.alltrue(np.less_equal(np.abs(scale - nscale),
+                                             nscale * self.tol))
+            test2 = np.alltrue(
+                np.less_equal(np.abs(mu - nmu), nscale * self.tol))
             if not (test1 and test2):
                 mu = nmu
                 scale = nscale
@@ -219,21 +221,28 @@ def __init__(self, d=2.5, tol=1e-08, maxiter=30):
         self.maxiter = maxiter
 
     def __call__(self, df_resid, nobs, resid):
-        h = (df_resid)/nobs*(self.d**2 + (1-self.d**2)*\
-                    Gaussian.cdf(self.d)-.5 - self.d/(np.sqrt(2*np.pi))*\
-                    np.exp(-.5*self.d**2))
+        h = (df_resid) / nobs * (self.d ** 2 + (1 - self.d ** 2) *
+                                 Gaussian.cdf(self.d) - .5 - self.d / (
+                                     np.sqrt(2 * np.pi)) *
+                                 np.exp(-.5 * self.d ** 2))
         s = mad(resid)
-        subset = lambda x: np.less(np.fabs(resid/x),self.d)
-        chi = lambda s: subset(s)*(resid/s)**2/2+(1-subset(s))*(self.d**2/2)
-        scalehist = [np.inf,s]
+
+        def subset(x):
+            return np.less(np.abs(resid / x), self.d)
+
+        def chi(s):
+            return subset(s) * (resid / s) ** 2 / 2 + (1 - subset(s)) * \
+                   (self.d ** 2 / 2)
+
+        scalehist = [np.inf, s]
         niter = 1
-        while (np.abs(scalehist[niter-1] - scalehist[niter])>self.tol \
-                and niter < self.maxiter):
-            nscale = np.sqrt(1/(nobs*h)*np.sum(chi(scalehist[-1]))*\
-                    scalehist[-1]**2)
+        while (np.abs(scalehist[niter - 1] - scalehist[niter]) > self.tol
+               and niter < self.maxiter):
+            nscale = np.sqrt(1 / (nobs * h) * np.sum(chi(scalehist[-1])) *
+                             scalehist[-1] ** 2)
             scalehist.append(nscale)
             niter += 1
-            #if niter == self.maxiter:
+            # if niter == self.maxiter:
             #    raise ValueError("Huber's scale failed to converge")
         return scalehist[-1]
 


Test patch:
diff --git a/statsmodels/robust/tests/test_rlm.py b/statsmodels/robust/tests/test_rlm.py
--- a/statsmodels/robust/tests/test_rlm.py
+++ b/statsmodels/robust/tests/test_rlm.py
@@ -107,6 +107,7 @@ def setup_class(cls):
         cls.decimal_scale = DECIMAL_3
 
         model = RLM(cls.data.endog, cls.data.exog, M=norms.HuberT())
+        cls.model = model
         results = model.fit()
         h2 = model.fit(cov="H2").bcov_scaled
         h3 = model.fit(cov="H3").bcov_scaled
@@ -122,6 +123,18 @@ def setup(self):
     def test_summary(self):
         self.res1.summary()
 
+    @pytest.mark.smoke
+    def test_summary2(self):
+        self.res1.summary2()
+
+    @pytest.mark.smoke
+    def test_chisq(self):
+        assert isinstance(self.res1.chisq, np.ndarray)
+
+    @pytest.mark.smoke
+    def test_predict(self):
+        assert isinstance(self.model.predict(self.res1.params), np.ndarray)
+
 
 class TestHampel(TestRlm):
     @classmethod
@@ -314,3 +327,48 @@ def test_rlm_start_values_errors():
     start_params = np.array([start_params, start_params]).T
     with pytest.raises(ValueError):
         model.fit(start_params=start_params)
+
+
+@pytest.fixture(scope='module',
+                params=[norms.AndrewWave, norms.LeastSquares, norms.HuberT,
+                        norms.TrimmedMean, norms.TukeyBiweight, norms.Hampel,
+                        norms.RamsayE])
+def norm(request):
+    return request.param()
+
+
+@pytest.fixture(scope='module')
+def perfect_fit_data(request):
+    from statsmodels.tools.tools import Bunch
+    rs = np.random.RandomState(1249328932)
+    exog = rs.standard_normal((1000, 1))
+    endog = exog + exog ** 2
+    exog = sm.add_constant(np.c_[exog, exog ** 2])
+    return Bunch(endog=endog, exog=exog, const=(3.2 * np.ones_like(endog)))
+
+
+def test_perfect_fit(perfect_fit_data, norm):
+    res = RLM(perfect_fit_data.endog, perfect_fit_data.exog, M=norm).fit()
+    assert_allclose(res.params, np.array([0, 1, 1]), atol=1e-8)
+
+
+def test_perfect_const(perfect_fit_data, norm):
+    res = RLM(perfect_fit_data.const, perfect_fit_data.exog, M=norm).fit()
+    assert_allclose(res.params, np.array([3.2, 0, 0]), atol=1e-8)
+
+
+@pytest.mark.parametrize('conv', ('weights', 'coefs', 'sresid'))
+def test_alt_criterion(conv):
+    data = sm.datasets.stackloss.load(as_pandas=True)
+    data.exog = sm.add_constant(data.exog, prepend=False)
+    base = RLM(data.endog, data.exog, M=norms.HuberT()).fit()
+    alt = RLM(data.endog, data.exog, M=norms.HuberT()).fit(conv=conv)
+    assert_allclose(base.params, alt.params)
+
+
+def test_bad_criterion():
+    data = sm.datasets.stackloss.load(as_pandas=True)
+    data.exog = sm.add_constant(data.exog, prepend=False)
+    mod = RLM(data.endog, data.exog, M=norms.HuberT())
+    with pytest.raises(ValueError, match='Convergence argument unknown'):
+        mod.fit(conv='unknown')