Repository: statsmodels/statsmodels

Issue:
Off-by-one error / loose dtype checking in KPSS statistic
#### Describe the bug

Arguably the KPSS code has a couple of issues. Firstly, there's no check whether the input array has suitable dtype, which can result in some weird error messages.
Second, I _think_ there's an off-by-one error in the internal function `_sigma_est_kpss`, but it only actually causes an exception in a weird edge-case highlighted by the object dtype.

(If you look at the indexing here:
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/stattools.py#L1733
Then you'll see that in the edge case `nobs==lags`, and for the last iteration in the loop we are computing `np.dot` on two empty arrays. The return value in this case is dtype dependent, it defaults to 0 for anything numeric, but if it happens to be an object-array we get None and the line of maths below breaks.)

NB, I don't have access to Kwiatkowski et al. (1992), so can't verify the original equation for myself.

#### Code Sample, a copy-pastable example if possible


```python
import numpy as np
from statsmodels.tsa.stattools import kpss

# Confusing error one:
boring_data = np.full(20, 0).astype(object)
kpss(boring_data, lags="auto")

# Confusing error two, hits possible off-by-one error in _sigma_est_kpss:
weird_data = array([0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0,
       0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0,
       0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
       0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
       0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0,
       1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1,
       1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1,
       0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0,
       0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0,
       0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
       0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
       0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0,
       1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1,
       1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0])

kpss(weird_data.astype(object), lags="auto")

```

Both those kpss calls result in an exception.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``



<details>


INSTALLED VERSIONS
------------------
Python: 3.6.5.final.0
OS: Linux 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8

Statsmodels
===========

Installed: 0.10.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.14.3 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/numpy)
scipy: 1.1.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/scipy)
pandas: 0.21.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/pandas)
    dateutil: 2.7.2 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/dateutil)
patsy: 0.5.1 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 2.2.2 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: 6.3.1 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/IPython)
    jinja2: 2.10 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.2.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/pygments)
pytest: Not installed
virtualenv: 16.0.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages)




</details>

BUG: Limit lags in KPSS
Limit lags to nobs - 1 in KPSS

closes #5925

- [X] closes #5925
- [X] tests added / passed. 
- [X] code/documentation is well formatted.  
- [X] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html#writing-the-commit-message). 



Discussion:
> (If you look at the indexing here:
> https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/stattools.py#L1733
> Then you'll see that in the edge case `nobs==lags`, and for the last iteration in the loop we are computing `np.dot` on two empty arrays. The return value in this case is dtype dependent, it defaults to 0 for anything numeric, but if it happens to be an object-array we get None and the line of maths below breaks.)

The bug is the missing limit on `lags` that makes sure it is smaller than nobs.  The code is correct for a Newey-West estimator.

[![Coverage Status](https://coveralls.io/builds/24505432/badge)](https://coveralls.io/builds/24505432)

Coverage increased (+0.0005%) to 85.177% when pulling **4bc08199f3a5bc6a3676a6771900a6e4313205e4 on bashtage:fix-kpss-max-lag** into **72f315d10a5d8858a740b4df0f0b3bcf877c7d6e on statsmodels:master**.


Resolving patch:
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -1607,6 +1607,7 @@ def kpss(x, regression='c', lags=None, store=False):
     elif lags == 'auto':
         # autolag method of Hobijn et al. (1998)
         lags = _kpss_autolag(resids, nobs)
+        lags = min(lags, nobs - 1)
     else:
         lags = int(lags)
 
@@ -1672,5 +1673,5 @@ def _kpss_autolag(resids, nobs):
     s_hat = s1 / s0
     pwr = 1. / 3.
     gamma_hat = 1.1447 * np.power(s_hat * s_hat, pwr)
-    autolags = np.amin([nobs, int(gamma_hat * np.power(nobs, pwr))])
+    autolags = int(gamma_hat * np.power(nobs, pwr))
     return autolags


Test patch:
diff --git a/statsmodels/tsa/tests/test_stattools.py b/statsmodels/tsa/tests/test_stattools.py
--- a/statsmodels/tsa/tests/test_stattools.py
+++ b/statsmodels/tsa/tests/test_stattools.py
@@ -553,6 +553,27 @@ def test_kpss_fails_on_nobs_check(self):
         with pytest.raises(ValueError, match=msg):
             kpss(self.x, 'c', lags=nobs)
 
+    def test_kpss_autolags_does_not_assign_lags_equal_to_nobs(self):
+        # Test that if *autolags* exceeds number of observations, we set suitable lags
+        # GH5925
+        data_which_breaks_autolag = np.array(
+            [0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0,
+             0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0,
+             0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
+             0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
+             0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0,
+             1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1,
+             1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1,
+             0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0,
+             0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0,
+             0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
+             0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
+             0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0,
+             1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1,
+             1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0]).astype(float)
+
+        kpss(data_which_breaks_autolag, lags="auto")
+
     def test_legacy_lags(self):
         # Test legacy lags are the same
         with pytest.warns(InterpolationWarning):