Repository: statsmodels/statsmodels

Issue:
ENH: White spec test (issue1126)
White's two-moment specification test. Unit test compares to SAS 9.3.

#1126


Discussion:
This pull request **introduces 1 alert** when merging 271b8281c9a8d5561a4fc5faa5a407bc289b7525 into 8928f8de2eb092fa249db40a5e6e4d42ceb54e99 - [view on LGTM.com](https://lgtm.com/projects/g/statsmodels/statsmodels/rev/pr-3c986e35d094c58431b7dcff73ec17359e1ba4ea)

**new alerts:**

* 1 for Module\-level cyclic import

---

*Comment posted by [LGTM.com](https://lgtm.com)*
BTW: You can ignore the LGTM circular import alerts.
All the ones I checked before where false alarm.

[![Coverage Status](https://coveralls.io/builds/18598310/badge)](https://coveralls.io/builds/18598310)

Coverage increased (+0.4%) to 83.859% when pulling **c8fe207f4b608958f288f8d4a3d23de7721b54b9 on JimVaranelli:issue1126** into **8928f8de2eb092fa249db40a5e6e4d42ceb54e99 on statsmodels:master**.

This pull request **introduces 1 alert** when merging bb23f81b0e916164eeb285cbc05816548002c025 into 8928f8de2eb092fa249db40a5e6e4d42ceb54e99 - [view on LGTM.com](https://lgtm.com/projects/g/statsmodels/statsmodels/rev/pr-a434c529f89090a97c4100e2f721a78703037303)

**new alerts:**

* 1 for Module\-level cyclic import

---

*Comment posted by [LGTM.com](https://lgtm.com)*
This pull request **introduces 1 alert** when merging 5adc464409f9cd0c5e6f9063dfc943f2839d1e21 into 8928f8de2eb092fa249db40a5e6e4d42ceb54e99 - [view on LGTM.com](https://lgtm.com/projects/g/statsmodels/statsmodels/rev/pr-fa16656abdeb359bc5603efc3161cb77e473a98f)

**new alerts:**

* 1 for Module\-level cyclic import

---

*Comment posted by [LGTM.com](https://lgtm.com)*
This pull request **introduces 1 alert** when merging 256bf53d144d31f36cb91f72d564ec9692139dda into 8928f8de2eb092fa249db40a5e6e4d42ceb54e99 - [view on LGTM.com](https://lgtm.com/projects/g/statsmodels/statsmodels/rev/pr-ff2e55d4a73c6bf50e9f393a34fa037382e60f50)

**new alerts:**

* 1 for Module\-level cyclic import

---

*Comment posted by [LGTM.com](https://lgtm.com)*
# [Codecov](https://codecov.io/gh/statsmodels/statsmodels/pull/4721?src=pr&el=h1) Report
> Merging [#4721](https://codecov.io/gh/statsmodels/statsmodels/pull/4721?src=pr&el=desc) into [master](https://codecov.io/gh/statsmodels/statsmodels/commit/8928f8de2eb092fa249db40a5e6e4d42ceb54e99?src=pr&el=desc) will **increase** coverage by `0.49%`.
> The diff coverage is `95%`.

[![Impacted file tree graph](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/graphs/tree.svg?width=650&token=Og706qJV5S&height=150&src=pr)](https://codecov.io/gh/statsmodels/statsmodels/pull/4721?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4721      +/-   ##
==========================================
+ Coverage   80.88%   81.38%   +0.49%     
==========================================
  Files         577      581       +4     
  Lines       88572    90234    +1662     
  Branches     9885    10082     +197     
==========================================
+ Hits        71642    73435    +1793     
+ Misses      14672    14505     -167     
- Partials     2258     2294      +36
```


| [Impacted Files](https://codecov.io/gh/statsmodels/statsmodels/pull/4721?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [statsmodels/stats/diagnostic.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvc3RhdHMvZGlhZ25vc3RpYy5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [statsmodels/stats/tests/test\_diagnostic.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvc3RhdHMvdGVzdHMvdGVzdF9kaWFnbm9zdGljLnB5) | `92.19% <100%> (+0.42%)` | :arrow_up: |
| [statsmodels/sandbox/stats/diagnostic.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvc2FuZGJveC9zdGF0cy9kaWFnbm9zdGljLnB5) | `60.23% <90.9%> (+1.37%)` | :arrow_up: |
| [statsmodels/tsa/holtwinters.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvdHNhL2hvbHR3aW50ZXJzLnB5) | `90.19% <0%> (-4.9%)` | :arrow_down: |
| [statsmodels/tsa/statespace/kalman\_filter.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvdHNhL3N0YXRlc3BhY2Uva2FsbWFuX2ZpbHRlci5weQ==) | `84.16% <0%> (-3.54%)` | :arrow_down: |
| [statsmodels/regression/recursive\_ls.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvcmVncmVzc2lvbi9yZWN1cnNpdmVfbHMucHk=) | `93.77% <0%> (-3.09%)` | :arrow_down: |
| [statsmodels/stats/correlation\_tools.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvc3RhdHMvY29ycmVsYXRpb25fdG9vbHMucHk=) | `90.27% <0%> (-3.06%)` | :arrow_down: |
| [statsmodels/tsa/stattools.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvdHNhL3N0YXR0b29scy5weQ==) | `82.65% <0%> (-1.28%)` | :arrow_down: |
| [statsmodels/tools/\_testing.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvdG9vbHMvX3Rlc3RpbmcucHk=) | `59.09% <0%> (-0.62%)` | :arrow_down: |
| [...tatsmodels/sandbox/panel/correlation\_structures.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvc2FuZGJveC9wYW5lbC9jb3JyZWxhdGlvbl9zdHJ1Y3R1cmVzLnB5) | `32.78% <0%> (-0.55%)` | :arrow_down: |
| ... and [141 more](https://codecov.io/gh/statsmodels/statsmodels/pull/4721/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/statsmodels/statsmodels/pull/4721?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/statsmodels/statsmodels/pull/4721?src=pr&el=footer). Last update [8928f8d...c8fe207](https://codecov.io/gh/statsmodels/statsmodels/pull/4721?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@JimVaranelli  Thank you

LGTM, the linear algebra looks good, but I didn't try to understand the computation.

Can you squash this?
AFAICS, earlier commits don't provide additional information in this case. 
Cleared the Codacy unused variable check. Not sure what to do about the test_spec_white() method could be a function complaint?
>  Not sure what to do about the test_spec_white() method could be a function complaint?

You added the test as a method to a test class, but you are not actually using anything in self, e.g. any of the data or results attached to the class/instance.
In this case it is better to move the test into a separate function (not as method inside a class) next to the other test functions.
Moved the static function outside of the class.
@josef-pkt Looks ready.  

@JimVaranelli We can wait to move it out of the sandbox.  This is a nice function and should be in main sm.  There are a few other diagnostic functions in sandbox that can be moved at the same time in a later PR once this is merged. 
LGTM, the only issue is to clean up the commit history.
The best would be to open a new PR with rebase and completely squashed commits.
The current commit messages are mostly uninformative, but it might still be useful to leave this PR branch unchanged for the record.
@josef-pkt Sorry for the delay I've been swamped with deliverable deadlines these past few weeks. I should be able to get the new PR out this weekend or early next week. I will also finalize the code for the ZA test (https://github.com/statsmodels/statsmodels/issues/4746) and generate a PR. With @bashtage code optimization I think SM will have the best publically available version of that particular test.
@josef-pkt Sorry again for the delay here. Would you still like me to create a new clean PR for this? I expect to submit a new PR for #4746 in the next few days.
@JimVaranelli  Yes, please do.

I put on my personal todo list to rebase PRs that are essentially ready for merging but have fallen too far behind master.
(But currently my checkout is still dirty with uncommitted changes for my own PRs.)

@josef-pkt OK will do. I'll definitely get that done this weekend. I'll also try to push out the ZA test PR this weekend as well but if not this weekend then early next week.

Resolving patch:
diff --git a/statsmodels/sandbox/stats/diagnostic.py b/statsmodels/sandbox/stats/diagnostic.py
--- a/statsmodels/sandbox/stats/diagnostic.py
+++ b/statsmodels/sandbox/stats/diagnostic.py
@@ -991,6 +991,85 @@ def linear_lm(resid, exog, func=None):
     lm_pval = stats.chi2.sf(lm, k_vars - 1)
     return lm, lm_pval, ftest
 
+
+def spec_white(resid, exog):
+    '''
+    White's Two-Moment Specification Test
+
+    Parameters
+    ----------
+    resid : array_like
+        OLS residuals
+    exog : array_like
+        OLS design matrix
+
+    Returns
+    -------
+    stat : float
+        test statistic
+    pval : float
+        chi-square p-value for test statistic
+    dof : int
+        degrees of freedom
+
+    Notes
+    -----
+    Implements the two-moment specification test described by White's
+    Theorem 2 (1980, p. 823) which compares the standard OLS covariance
+    estimator with White's heteroscedasticity-consistent estimator. The
+    test statistic is shown to be chi-square distributed.
+
+    Null hypothesis is homoscedastic and correctly specified.
+
+    Assumes the OLS design matrix contains an intercept term and at least
+    one variable. The intercept is removed to calculate the test statistic.
+
+    Interaction terms (squares and crosses of OLS regressors) are added to
+    the design matrix to calculate the test statistic.
+
+    Degrees-of-freedom (full rank) = nvar + nvar * (nvar + 1) / 2
+
+    Linearly dependent columns are removed to avoid singular matrix error.
+
+    Reference
+    ---------
+    White, H. (1980). A heteroskedasticity-consistent covariance matrix
+    estimator and a direct test for heteroscedasticity. Econometrica,
+    48: 817-838.
+
+    See also het_white.
+    '''
+    x = np.asarray(exog)
+    e = np.asarray(resid)
+    if x.ndim == 1:
+        raise ValueError('X should have a constant and at least one variable')
+
+    # add interaction terms
+    i0, i1 = np.triu_indices(x.shape[1])
+    exog = np.delete(x[:,i0] * x[:,i1], 0, 1)
+
+    # collinearity check - see _fit_collinear
+    atol=1e-14; rtol=1e-13
+    tol = atol + rtol * exog.var(0)
+    r = np.linalg.qr(exog, mode='r')
+    mask = np.abs(r.diagonal()) < np.sqrt(tol)
+    exog = exog[:,np.where(~mask)[0]]
+
+    # calculate test statistic
+    sqe = e * e
+    sqmndevs = sqe - np.mean(sqe)
+    d = np.dot(exog.T, sqmndevs)
+    devx = exog - np.mean(exog, axis=0)
+    devx *= sqmndevs[:, None]
+    b = devx.T.dot(devx)
+    stat = d.dot(np.linalg.solve(b, d))
+
+    # chi-square test
+    dof = devx.shape[1]
+    pval = stats.chi2.sf(stat, dof)
+    return stat, pval, dof
+
+
 def _neweywestcov(resid, x):
     '''
     Did not run yet
diff --git a/statsmodels/stats/diagnostic.py b/statsmodels/stats/diagnostic.py
--- a/statsmodels/stats/diagnostic.py
+++ b/statsmodels/stats/diagnostic.py
@@ -5,8 +5,7 @@
     het_goldfeldquandt, het_arch,
     het_white, recursive_olsresiduals, acorr_breusch_godfrey,
     linear_harvey_collier, linear_rainbow, linear_lm,
-    unitroot_adf
-)
+    spec_white, unitroot_adf)
 
 from ._lilliefors import (kstest_fit, lilliefors, kstest_normal,
                           kstest_exponential)


Test patch:
diff --git a/statsmodels/stats/tests/results/wspec1.csv b/statsmodels/stats/tests/results/wspec1.csv
new file mode 100644
--- /dev/null
+++ b/statsmodels/stats/tests/results/wspec1.csv
@@ -0,0 +1,36 @@
+IV1,IV2,DV
+-0.02015,-8.12,-0.00262
+-0.09145,22.6,0.06002
+-0.23417,34.14,0.28942
+-0.11179,-24.21,-0.01991
+0.16144,-14.37,-0.11282
+0.15733,-10.98,-0.15465
+0.05293,-0.61,-0.09055
+0.05686,-3.38,-0.01311
+-0.11524,18.48,0.01793
+0.10973,-12.93,-0.02117
+0.11152,-9.32,-0.08318
+0.05917,5.86,-0.01982
+-0.00468,-6.67,0.03378
+-0.15653,25.27,0.10565
+0.115,-2.55,0.0483
+0.12352,-22.48,-0.07733
+-0.03604,3.69,0.07183
+0.05634,-6.19,-0.05512
+-0.00401,2.25,0.01318
+0.10524,-3.73,-0.00379
+0.02288,1.5,0.00922
+0.05646,-3.48,-0.02355
+0.09566,3.33,-0.06103
+0.01535,1.1,-0.0343
+0.0435,-4.41,-0.02346
+-0.00534,0,-0.05993
+0.04721,9.22,-0.00264
+0.01321,-3.86,-0.02038
+-0.00353,-3.54,-0.00693
+-0.07728,21.89,0.04091
+0.0572,-16.35,0.00791
+0.00372,3.75,0.10267
+0.02087,-2.38,-0.04426
+0.03918,-7.62,-0.05083
+0.03597,4.37,0.01428
diff --git a/statsmodels/stats/tests/results/wspec2.csv b/statsmodels/stats/tests/results/wspec2.csv
new file mode 100644
--- /dev/null
+++ b/statsmodels/stats/tests/results/wspec2.csv
@@ -0,0 +1,36 @@
+IV1,IV2,IV3,DV
+144.6,0.02846,2.28,-0.00262
+30,0.15082,2.768,0.06002
+-293.1,0.58902,3.323,0.28942
+-166,0.01642,-5.012,-0.01991
+-43.5,-0.23772,-1.471,-0.11282
+43.7,-0.42052,1.129,-0.15465
+182.4,-0.2129,1.837,-0.09055
+114.6,-0.22211,1.686,-0.01311
+207.5,0.02502,0.344,0.01793
+169.1,0.10521,-0.077,-0.02117
+172.5,-0.07982,0.637,-0.08318
+8.2,-0.08353,1.765,-0.01982
+222.5,-0.02544,2.345,0.03378
+126.2,0.28108,2.524,0.10565
+198.2,0.18466,1.465,0.0483
+188.6,-0.09714,1.014,-0.07733
+148,0.01393,1.279,0.07183
+106,-0.04318,0.482,-0.05512
+69.4,-0.14695,1.033,0.01318
+178.1,-0.07366,1.528,-0.00379
+66,0.02879,0.93,0.00922
+207.9,0.01782,-0.271,-0.02355
+250.6,-0.06627,1.253,-0.06103
+31.4,-0.114,0.906,-0.0343
+289.6,-0.08656,1.491,-0.02346
+301.4,0.01026,1.099,-0.05993
+113.6,0.12251,0.612,-0.00264
+138.8,0.05026,-0.418,-0.02038
+218.5,0,-1.507,-0.00693
+134.5,0.12344,1.373,0.04091
+59.5,0.05136,0.881,0.00791
+38,0.10883,0.212,0.10267
+212.8,-0.12725,0.067,-0.04426
+191.1,-0.09324,1.374,-0.05083
+176.4,-0.09707,1.06,0.01428
diff --git a/statsmodels/stats/tests/results/wspec3.csv b/statsmodels/stats/tests/results/wspec3.csv
new file mode 100644
--- /dev/null
+++ b/statsmodels/stats/tests/results/wspec3.csv
@@ -0,0 +1,36 @@
+IV1,IV2,DUM1,DUM2,DV
+-0.02015,7.6,0,0,0.01291
+-0.09145,44.1,0,0,-0.04267
+-0.23417,252.7,0,0,-0.25406
+-0.11179,9.4,1,0,0.34175
+0.16144,-122.1,0,0,0.10051
+0.15733,-156.2,0,0,0.13155
+0.05293,-57.3,0,0,0.09492
+0.05686,-48.1,0,0,0.03839
+-0.11524,4.9,0,0,0.0188
+0.10973,22,0,0,0.01934
+0.11152,-16.9,0,0,0.10318
+0.05917,-16.3,0,0,0.06143
+-0.00468,-4.7,0,0,0.00608
+-0.15653,59.2,0,0,-0.00281
+0.115,49,0,0,0.01905
+0.12352,-26.9,0,0,0.06962
+-0.03604,3.7,0,1,0.21069
+0.05634,-11.3,0,0,0.06319
+-0.00401,-35,0,0,0.03645
+0.10524,-15.7,0,0,0.0444
+0.02288,6,0,0,0.02033
+0.05646,3.8,0,0,0.06044
+0.09566,-13.8,0,0,0.06762
+0.01535,-21.7,0,0,0.03301
+0.0435,-14.9,0,0,0.01857
+-0.00534,1.7,0,0,0.06799
+0.04721,21.7,0,0,0.01813
+0.01321,9.7,0,0,0.02022
+-0.00353,0,0,0,0.01317
+-0.07728,26,0,0,-0.00628
+0.0572,11.8,0,0,0.02035
+0.00372,27.1,0,0,-0.04564
+0.02087,-31.4,0,0,0.04537
+0.03918,-20.6,0,0,0.07685
+0.03597,-19.5,0,0,0.03299
diff --git a/statsmodels/stats/tests/results/wspec4.csv b/statsmodels/stats/tests/results/wspec4.csv
new file mode 100644
--- /dev/null
+++ b/statsmodels/stats/tests/results/wspec4.csv
@@ -0,0 +1,36 @@
+IV1,IV2,IV3,DUM1,DUM2,DV
+144.6,-0.02036,0.03699,0,0,0.01291
+30,-0.09591,-0.25186,0,0,-0.04267
+-293.1,-0.2668,0.93698,0,0,-0.25406
+-166,-0.11855,0.73074,1,0,0.34175
+-43.5,0.14966,-0.29941,0,0,0.10051
+43.7,0.14612,-0.25366,0,0,0.13155
+182.4,0.05158,-0.2597,0,0,0.09492
+114.6,0.0553,-0.01949,0,0,0.03839
+207.5,-0.12244,-0.11013,0,0,0.0188
+169.1,0.10412,0.67668,0,0,0.01934
+172.5,0.10573,-0.28238,0,0,0.10318
+8.2,0.05749,-0.28363,0,0,0.06143
+222.5,-0.00469,0.24894,0,0,0.00608
+126.2,-0.17023,-0.22687,0,0,-0.00281
+198.2,0.10886,1.11175,0,0,0.01905
+188.6,0.11647,-0.05312,0,0,0.06962
+148,-0.03671,-0.49461,0,1,0.21069
+106,0.05481,0.16064,0,0,0.06319
+69.4,-0.00402,-0.23218,0,0,0.03645
+178.1,0.10007,0.10992,0,0,0.0444
+66,0.02262,-0.16417,0,0,0.02033
+207.9,0.05493,0.07899,0,0,0.06044
+250.6,0.09136,-0.16984,0,0,0.06762
+31.4,0.01524,0.19577,0,0,0.03301
+289.6,0.04258,0.05408,0,0,0.01857
+301.4,-0.00535,-0.20569,0,0,0.06799
+113.6,0.04613,0,0,0,0.01813
+138.8,0.01312,0.5414,0,0,0.02022
+218.5,-0.00354,-0.14705,0,0,0.01317
+134.5,-0.08043,-0.15811,0,0,-0.00628
+59.5,0.05562,1.16127,0,0,0.02035
+38,0.00371,-0.40133,0,0,-0.04564
+212.8,0.02066,0.15375,0,0,0.04537
+191.1,0.03843,-0.08458,0,0,0.07685
+176.4,0.03534,-0.29581,0,0,0.03299
diff --git a/statsmodels/stats/tests/test_diagnostic.py b/statsmodels/stats/tests/test_diagnostic.py
--- a/statsmodels/stats/tests/test_diagnostic.py
+++ b/statsmodels/stats/tests/test_diagnostic.py
@@ -721,6 +721,33 @@ def setup_class(cls):
         cls.exog = cls.res.model.exog
 
 
+def test_spec_white():
+    resdir = os.path.join(cur_dir, "results")
+    wsfiles = ['wspec1.csv', 'wspec2.csv', 'wspec3.csv', 'wspec4.csv']
+    for file in wsfiles:
+        mdlfile = os.path.join(resdir, file)
+        mdl = np.asarray(pd.read_csv(mdlfile))
+        # DV is in last column
+        lastcol = mdl.shape[1] - 1
+        dv = mdl[:,lastcol]
+        # create design matrix
+        design = np.concatenate((np.ones((mdl.shape[0], 1)), \
+            np.delete(mdl, lastcol, 1)), axis=1)
+        # perform OLS and generate residuals
+        resids = dv - np.dot(design, np.linalg.lstsq(design, dv, rcond=-1)[0])
+        # perform White spec test. wspec3/wspec4 contain dummies.
+        wsres = smsdia.spec_white(resids, design)
+        # compare results to SAS 9.3 output
+        if file == 'wspec1.csv':
+            assert_almost_equal(wsres, [3.251, 0.661, 5], decimal=3)
+        elif file == 'wspec2.csv':
+            assert_almost_equal(wsres, [6.070, 0.733, 9], decimal=3)
+        elif file == 'wspec3.csv':
+            assert_almost_equal(wsres, [6.767, 0.454, 7], decimal=3)
+        else:
+            assert_almost_equal(wsres, [8.462, 0.671, 11], decimal=3)
+
+
 def grangertest():
     #> gt = grangertest(ginv, ggdp, order=4)
     #> gt