Repository: statsmodels/statsmodels

Issue:
QuantReg fails when a residual is exactly zero.
I would like to use quantile regression on a dataset with many factor variables. It seems it is easy for the model to get at least one residual that is exactly zero with this dataset... When this happens, I get nans in xstar, and the code raises an exception in the next round. The cause is that the first two lines in this snippet of the fit method

```
        mask = np.abs(resid) < .000001
        resid[mask] = np.sign(resid[mask]) * .000001
        resid = np.where(resid < 0, q * resid, (1-q) * resid)
        resid = np.abs(resid)
        xstar = exog / resid[:, np.newaxis]
```

reset residuals to a small number only if they are close to zero but not zero. This is because np.sign(0) is 0, so the second line does nothing when the residual is 0. In my mind, the easy solution is to arbitrarily choose a side (say, the positive one) when a residual is zero, How would that sound?

QuantReg fails when a residual is exactly zero.
I would like to use quantile regression on a dataset with many factor variables. It seems it is easy for the model to get at least one residual that is exactly zero with this dataset... When this happens, I get nans in xstar, and the code raises an exception in the next round. The cause is that the first two lines in this snippet of the fit method

```
        mask = np.abs(resid) < .000001
        resid[mask] = np.sign(resid[mask]) * .000001
        resid = np.where(resid < 0, q * resid, (1-q) * resid)
        resid = np.abs(resid)
        xstar = exog / resid[:, np.newaxis]
```

reset residuals to a small number only if they are close to zero but not zero. This is because np.sign(0) is 0, so the second line does nothing when the residual is 0. In my mind, the easy solution is to arbitrarily choose a side (say, the positive one) when a residual is zero, How would that sound?



Discussion:
Thanks for catching this.
Do you have a short example that can be used to test this case?
Setting the the zeros positive sounds fine, but there is a chance that the solution becomes asymmetric in q and 1-q if there are many ties at zero (discretized or rounded observations instead of continuous observations)
Do you have just one or very residuals at zero or some larger number?

related aside: I was thinking of also trying a smoothed version (locally quadratic around zero) to avoid some problems with zero or near zero residuals. (there was a convergence cycle reported on stats.stackexchange)

we could just replace the sign function `np.sign(resid[mask])`

by `((resid[mask]) >= 0) *2 - 1)`

for trimming to positive

another option to look into (as a more general solution to near zero residuals)

regularize the division, i.e. limit the weights

`xstar = exog / (a0 + resid[:, np.newaxis]`

with a small `a0 = 1e-6`
(Fan and Li 2001 do something similar with data dependent a0 as function of quantiles for L1 penalization around origin. I didn't read that part carefully.)

I haven't checked whether near zero residuals also create other problems

**edit**
additive a0 is not just shifting the residuals, because resid at that point is positive `resid_abs`

Thanks for the quick reply. The dataset I'm working with is kind of large, but it should be possible to reproduce this also with a synthetic dataset, no? The times I've looked at the number of zeroes in the debugger, it was just a few or more like one, actually. And given that my model matrix is very tall and not that predictable, it would be surprising to get a lot of zeroes for the residual during these iterations.

I'll let people who are more knowledgeable about statistics than me select the best way of keeping that `xstar` variable not too large. I'd be fine with all the solutions proposed above.

What do you need from me to help fix this bug?

It would be helpful if you could make up a small example that shows the behavior.
One problem with these type of changes is that we need a collection of corner cases to verify and test the behavior.

Replacing the sign function by the weak inequality would be a quick fix, but I would prefer the regularization solution, but I would have to do more testing for that.

The following snippet of code reproduces the problem:

``` python
import numpy as np
from statsmodels.api import QuantReg

X = np.array([[1, 0], [0, 1]], dtype=np.float64)
y = np.array([0, 1], dtype=np.float64)

QuantReg(y, X).fit(0.5)
```

I confirmed (with a debugger) that the residuals are zero after the first iteration.

The regularization solution looks like the better fix to me too, at least at an intuitive level.

@chrish42  Thanks for the example, I'm finally going back to this and will fix it for statsmodels 0.7

A simple way to trigger this case is to include a outlier (one-observation) dummy variable, which makes it a pretty general problem.

Thanks for catching this.
Do you have a short example that can be used to test this case?
Setting the the zeros positive sounds fine, but there is a chance that the solution becomes asymmetric in q and 1-q if there are many ties at zero (discretized or rounded observations instead of continuous observations)
Do you have just one or very residuals at zero or some larger number?

related aside: I was thinking of also trying a smoothed version (locally quadratic around zero) to avoid some problems with zero or near zero residuals. (there was a convergence cycle reported on stats.stackexchange)

we could just replace the sign function `np.sign(resid[mask])`

by `((resid[mask]) >= 0) *2 - 1)`

for trimming to positive

another option to look into (as a more general solution to near zero residuals)

regularize the division, i.e. limit the weights

`xstar = exog / (a0 + resid[:, np.newaxis]`

with a small `a0 = 1e-6`
(Fan and Li 2001 do something similar with data dependent a0 as function of quantiles for L1 penalization around origin. I didn't read that part carefully.)

I haven't checked whether near zero residuals also create other problems

**edit**
additive a0 is not just shifting the residuals, because resid at that point is positive `resid_abs`

Thanks for the quick reply. The dataset I'm working with is kind of large, but it should be possible to reproduce this also with a synthetic dataset, no? The times I've looked at the number of zeroes in the debugger, it was just a few or more like one, actually. And given that my model matrix is very tall and not that predictable, it would be surprising to get a lot of zeroes for the residual during these iterations.

I'll let people who are more knowledgeable about statistics than me select the best way of keeping that `xstar` variable not too large. I'd be fine with all the solutions proposed above.

What do you need from me to help fix this bug?

It would be helpful if you could make up a small example that shows the behavior.
One problem with these type of changes is that we need a collection of corner cases to verify and test the behavior.

Replacing the sign function by the weak inequality would be a quick fix, but I would prefer the regularization solution, but I would have to do more testing for that.

The following snippet of code reproduces the problem:

``` python
import numpy as np
from statsmodels.api import QuantReg

X = np.array([[1, 0], [0, 1]], dtype=np.float64)
y = np.array([0, 1], dtype=np.float64)

QuantReg(y, X).fit(0.5)
```

I confirmed (with a debugger) that the residuals are zero after the first iteration.

The regularization solution looks like the better fix to me too, at least at an intuitive level.

@chrish42  Thanks for the example, I'm finally going back to this and will fix it for statsmodels 0.7

A simple way to trigger this case is to include a outlier (one-observation) dummy variable, which makes it a pretty general problem.


Resolving patch:
diff --git a/statsmodels/regression/quantile_regression.py b/statsmodels/regression/quantile_regression.py
--- a/statsmodels/regression/quantile_regression.py
+++ b/statsmodels/regression/quantile_regression.py
@@ -172,7 +172,7 @@ def fit(self, q=.5, vcov='robust', kernel='epa', bandwidth='hsheather',
             resid = endog - np.dot(exog, beta)
 
             mask = np.abs(resid) < .000001
-            resid[mask] = np.sign(resid[mask]) * .000001
+            resid[mask] = ((resid[mask] >= 0) * 2 - 1) * .000001
             resid = np.where(resid < 0, q * resid, (1-q) * resid)
             resid = np.abs(resid)
             xstar = exog / resid[:, np.newaxis]
@@ -366,18 +366,18 @@ def summary(self, yname=None, xname=None, title=None, alpha=.05):
 
         """
 
-        #TODO: import where we need it (for now), add as cached attributes
-        from statsmodels.stats.stattools import (jarque_bera,
-                omni_normtest, durbin_watson)
-        jb, jbpv, skew, kurtosis = jarque_bera(self.wresid)
-        omni, omnipv = omni_normtest(self.wresid)
-
+        # #TODO: import where we need it (for now), add as cached attributes
+        # from statsmodels.stats.stattools import (jarque_bera,
+        #         omni_normtest, durbin_watson)
+        # jb, jbpv, skew, kurtosis = jarque_bera(self.wresid)
+        # omni, omnipv = omni_normtest(self.wresid)
+        #
         eigvals = self.eigenvals
         condno = self.condition_number
-
-        self.diagn = dict(jb=jb, jbpv=jbpv, skew=skew, kurtosis=kurtosis,
-                          omni=omni, omnipv=omnipv, condno=condno,
-                          mineigval=eigvals[0])
+        #
+        # self.diagn = dict(jb=jb, jbpv=jbpv, skew=skew, kurtosis=kurtosis,
+        #                   omni=omni, omnipv=omnipv, condno=condno,
+        #                   mineigval=eigvals[0])
 
         top_left = [('Dep. Variable:', None),
                     ('Model:', None),
@@ -394,17 +394,17 @@ def summary(self, yname=None, xname=None, title=None, alpha=.05):
                      ('Df Model:', None) #[self.df_model])
                     ]
 
-        diagn_left = [('Omnibus:', ["%#6.3f" % omni]),
-                      ('Prob(Omnibus):', ["%#6.3f" % omnipv]),
-                      ('Skew:', ["%#6.3f" % skew]),
-                      ('Kurtosis:', ["%#6.3f" % kurtosis])
-                      ]
-
-        diagn_right = [('Durbin-Watson:', ["%#8.3f" % durbin_watson(self.wresid)]),
-                       ('Jarque-Bera (JB):', ["%#8.3f" % jb]),
-                       ('Prob(JB):', ["%#8.3g" % jbpv]),
-                       ('Cond. No.', ["%#8.3g" % condno])
-                       ]
+        # diagn_left = [('Omnibus:', ["%#6.3f" % omni]),
+        #               ('Prob(Omnibus):', ["%#6.3f" % omnipv]),
+        #               ('Skew:', ["%#6.3f" % skew]),
+        #               ('Kurtosis:', ["%#6.3f" % kurtosis])
+        #               ]
+        #
+        # diagn_right = [('Durbin-Watson:', ["%#8.3f" % durbin_watson(self.wresid)]),
+        #                ('Jarque-Bera (JB):', ["%#8.3f" % jb]),
+        #                ('Prob(JB):', ["%#8.3g" % jbpv]),
+        #                ('Cond. No.', ["%#8.3g" % condno])
+        #                ]
 
 
         if title is None:


Test patch:
diff --git a/statsmodels/regression/tests/test_quantile_regression.py b/statsmodels/regression/tests/test_quantile_regression.py
--- a/statsmodels/regression/tests/test_quantile_regression.py
+++ b/statsmodels/regression/tests/test_quantile_regression.py
@@ -208,3 +208,30 @@ def setUp(cls):
     #@classmethod
     #def setUp(cls):
         #cls.res1, cls.res2 = setup_fun('tri', 'hsheather')
+
+def test_zero_resid():
+    # smoke and regression tests
+
+    X = np.array([[1, 0], [0, 1], [0, 2.1], [0, 3.1]], dtype=np.float64)
+    y = np.array([0, 1, 2, 3], dtype=np.float64)
+
+    res = QuantReg(y, X).fit(0.5, bandwidth='chamberlain') #'bofinger')
+    res.summary()
+
+    assert_allclose(res.params, np.array([0.0,  0.96774163]), rtol=1e-4, atol=1e-20)
+    assert_allclose(res.bse, np.array([0.0447576, 0.01154867]), rtol=1e-4, atol=1e-20)
+    assert_allclose(res.resid, np.array([0.0,  3.22583680e-02,  -3.22574272e-02,
+         9.40732912e-07]), rtol=1e-4, atol=1e-20)
+
+
+    X = np.array([[1, 0], [0.1, 1], [0, 2.1], [0, 3.1]], dtype=np.float64)
+    y = np.array([0, 1, 2, 3], dtype=np.float64)
+
+    res = QuantReg(y, X).fit(0.5, bandwidth='chamberlain')
+    res.summary()
+
+    assert_allclose(res.params, np.array([9.99982796e-08, 9.67741630e-01]),
+                    rtol=1e-4, atol=1e-20)
+    assert_allclose(res.bse, np.array([0.04455029, 0.01155251]), rtol=1e-4, atol=1e-20)
+    assert_allclose(res.resid, np.array([-9.99982796e-08, 3.22583598e-02,
+                            -3.22574234e-02, 9.46361860e-07]), rtol=1e-4, atol=1e-20)