Repository: statsmodels/statsmodels

Issue:
ENH: diagnostic.recursive_olsresiduals should check that the initial X'X matrix has full rank
#### Describe the bug

The recursive residues computations sometimes returns np.nan values in rresid_scaled (after the usual initial nan), coming from negative values in rvarraw. It depends on the dataset. For the same dataset, changing the order of the exogenous variables changes the result

#### Code Sample, a copy-pastable example if possible

```
import numpy as np
import pandas as pd
import statsmodels.stats.api as sms
import statsmodels.formula.api as smf
import statsmodels.stats.diagnostic as smdiag

dataset=np.array([0,7.014799322267146,13.47334785849311,-1.3169578969248168,
3,6.185878051672474,13.47334785849311,-0.7629096520666107,
6,5.630920325521993,13.47334785849311,-0.3563785047244501,
9,6.363851632277281,13.47334785849311,-0.881373587019543,
12,7.257486347135948,14.379600111877165,-1.3169578969248168,
15,6.423712775322329,14.379600111877165,-0.7629096520666107,
18,5.83737750947789,14.379600111877165,-0.3563785047244501,
21,6.687553844007199,14.379600111877165,-0.881373587019543,
24,7.389791926029563,15.063152274677005,-1.3169578969248168,
27,6.639826281881416,15.063152274677005,-0.7629096520666107,
30,5.999933200118575,15.063152274677005,-0.3563785047244501,
33,6.963107389152293,15.063152274677005,-0.881373587019543,
36,7.824891534944816,15.117859463727847,-1.7354151626688945,
39,7.377190721069367,15.117859463727847,-1.1701612351210517,
42,6.982662217105705,15.117859463727847,-0.881373587019543,
45,6.314446025665141,15.117859463727847,-0.5493061443340549,
48,7.792171774757407,14.236482060109118,-1.7354151626688945,
51,7.036776429548548,14.236482060109118,-1.0106831886830212,
54,6.076902278267195,14.236482060109118,-0.3563785047244501,
57,6.806714858421174,14.236482060109118,-0.881373587019543,
60,7.780656513758164,14.547462532301932,-1.7354151626688945,
63,7.302058440982015,14.547462532301932,-1.1542345536088656,
66,7.079919367665778,14.547462532301932,-1.0106831886830212,
69,6.428518822142567,14.547462532301932,-0.652836579719798,
72,5.927316797773215,14.547462532301932,-0.3563785047244501,
75,7.530361052000997,14.453841928391206,-1.3169578969248168,
78,7.20545815055174,14.453841928391206,-1.0106831886830212,
81,6.660119862361395,14.453841928391206,-0.652836579719798,
84,6.287908640385098,14.453841928391206,-0.3563785047244501,
87,7.167990144322255,13.204435732336696,-1.3169578969248168,
90,6.6817942288765915,13.204435732336696,-1.0106831886830212,
93,6.214582128770743,13.204435732336696,-0.652836579719798,
96,5.836659121689873,13.204435732336696,-0.3563785047244501])

dataset=np.reshape(dataset[np.newaxis,:],(33,4))

df=pd.DataFrame({'y':dataset[:,1],'x1':dataset[:,2],'x2':dataset[:,3]})

def hv_test(data,formula):
    model=smf.ols(formula=formula,data=data)
    model_fit=model.fit()
    rr=smdiag.recursive_olsresiduals(model_fit,skip=3)
    rresid=rr[0]
    rresid_scaled=rr[4]
    hc=sms.linear_harvey_collier(model_fit)
    return rresid,rresid_scaled,hc

rresid,rresid_scaled,hc=hv_test(df,'y ~ x1 + x2')
print('Harvey Collier test\n')
print(hc)
print('rresid_scaled\n')
print(rresid_scaled)
print('')
rresid,rresid_scaled,hc=hv_test(df,'y ~ x2 + x1')
print('Harvey Collier test\n')
print(hc)
print('rresid_scaled\n')
print(rresid_scaled)
print('')

```

<details>

I found this issue by running the Harvey Collier test. Even when no nan values appear, depending on the dataset, changing the order of the exogenous variables changes the result of the harvey Collier test.

By digging a little bit in the recursive_olsresiduals function, regarding the nan values, the problem seems to come from the computation of rvarraw, which may be sommetimes negative. In turn, when the sqrt is computed, it results in a nan.

Sorry if the problem has aleardy been solved or disussed... i'm using the conda version of statsmodels, which may be older that the one on github?

</details>

#### Expected Output

No difference in recursive_olsresiduals output for the same dataset, or at least no np.nan

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>
INSTALLED VERSIONS
------------------
Python: 3.7.6.final.0
OS: Linux 5.3.0-42-generic #34~18.04.1-Ubuntu SMP Fri Feb 28 13:42:26 UTC 2020 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.11.0 (/home/username/anaconda3/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.15 (/home/username/anaconda3/lib/python3.7/site-packages/Cython)
numpy: 1.18.1 (/home/username/anaconda3/lib/python3.7/site-packages/numpy)
scipy: 1.4.1 (/home/username/anaconda3/lib/python3.7/site-packages/scipy)
pandas: 1.0.1 (/home/username/anaconda3/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.1 (/home/username/anaconda3/lib/python3.7/site-packages/dateutil)
patsy: 0.5.1 (/home/username/anaconda3/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.1.3 (/home/username/anaconda3/lib/python3.7/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.14.1 (/home/username/anaconda3/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.12.0 (/home/username/anaconda3/lib/python3.7/site-packages/IPython)
    jinja2: 2.11.1 (/home/username/anaconda3/lib/python3.7/site-packages/jinja2)
sphinx: 2.4.0 (/home/username/anaconda3/lib/python3.7/site-packages/sphinx)
    pygments: 2.5.2 (/home/username/anaconda3/lib/python3.7/site-packages/pygments)
pytest: 5.3.5 (/home/username/anaconda3/lib/python3.7/site-packages/pytest)
virtualenv: Not installed

</details>



Discussion:
Your initial x matrix for the calculation is singular since you have no variation in x1 for the first few observations.  You need to set skip to at least 8.

There needs to be a check that the initial xtx matrix has full rank which is required for LS to be well defined. 

That makes sens, thank you! 

If I'm not wrong, the skip parameter is not available when calling directly the Harvey Collier test, might be also useful to be able to specify it. it would be easier than changing the order of the residuals with order_by.

Thanks again!

Resolving patch:
diff --git a/statsmodels/stats/diagnostic.py b/statsmodels/stats/diagnostic.py
--- a/statsmodels/stats/diagnostic.py
+++ b/statsmodels/stats/diagnostic.py
@@ -1255,7 +1255,7 @@ def linear_reset(res, power=3, test_type="fitted", use_f=False,
     return res.wald_test(r_mat, use_f=use_f)
 
 
-def linear_harvey_collier(res, order_by=None):
+def linear_harvey_collier(res, order_by=None, skip=None):
     """
     Harvey Collier test for linearity
 
@@ -1269,6 +1269,9 @@ def linear_harvey_collier(res, order_by=None):
         Integer array specifying the order of the residuals. If not provided,
         the order of the residuals is not changed. If provided, must have
         the same number of observations as the endogenous variable.
+    skip : int, default None
+        The number of observations to use for initial OLS, if None then skip is
+        set equal to the number of regressors (columns in exog).
 
     Returns
     -------
@@ -1277,6 +1280,11 @@ def linear_harvey_collier(res, order_by=None):
     pvalue : float
         The pvalue of the test.
 
+    See Also
+    --------
+    statsmodels.stats.diadnostic.recursive_olsresiduals
+        Recursive OLS residual calculation used in the test.
+
     Notes
     -----
     This test is a t-test that the mean of the recursive ols residuals is zero.
@@ -1285,7 +1293,7 @@ def linear_harvey_collier(res, order_by=None):
     # I think this has different ddof than
     # B.H. Baltagi, Econometrics, 2011, chapter 8
     # but it matches Gretl and R:lmtest, pvalue at decimal=13
-    rr = recursive_olsresiduals(res, skip=3, alpha=0.95, order_by=order_by)
+    rr = recursive_olsresiduals(res, skip=skip, alpha=0.95, order_by=order_by)
 
     return stats.ttest_1samp(rr[3][3:], 0)
 
@@ -1611,8 +1619,14 @@ def recursive_olsresiduals(res, skip=None, lamda=0.0, alpha=0.95,
     rvarraw = np.nan * np.zeros(nobs)
 
     x0 = x[:skip]
+    if np.linalg.matrix_rank(x0) < x0.shape[1]:
+        err_msg = """\
+"The initial regressor matrix, x[:skip], issingular. You must use a value of
+skip large enough to ensure that the first OLS estimator is well-defined.
+"""
+        raise ValueError(err_msg)
     y0 = y[:skip]
-    # add Ridge to start (not in jplv
+    # add Ridge to start (not in jplv)
     xtxi = np.linalg.inv(np.dot(x0.T, x0) + lamda * np.eye(nvars))
     xty = np.dot(x0.T, y0)  # xi * y   #np.dot(xi, y)
     beta = np.dot(xtxi, xty)


Test patch:
diff --git a/statsmodels/stats/tests/test_diagnostic.py b/statsmodels/stats/tests/test_diagnostic.py
--- a/statsmodels/stats/tests/test_diagnostic.py
+++ b/statsmodels/stats/tests/test_diagnostic.py
@@ -423,6 +423,8 @@ def test_harvey_collier(self):
 
         hc = smsdia.linear_harvey_collier(self.res)
         compare_to_reference(hc, harvey_collier, decimal=(12, 12))
+        hc_skip = smsdia.linear_harvey_collier(self.res, skip=20)
+        assert not np.allclose(hc[0], hc_skip[0])
 
     def test_rainbow(self):
         # rainbow test
@@ -1235,6 +1237,14 @@ def test_rainbow_exception(reset_randomstate):
     with pytest.raises(TypeError, match="order_by must contain"):
         smsdia.linear_rainbow(res, order_by=("x0",))
 
+
+def test_small_skip(reset_randomstate):
+    y = np.random.standard_normal(10)
+    x = np.random.standard_normal((10, 3))
+    x[:3] = x[:1]
+    with pytest.raises(ValueError, match="The initial regressor matrix,"):
+        smsdia.recursive_olsresiduals(OLS(y, x).fit())
+
 # R code used in testing
 # J test
 #